Throughput

The time delay between an AI system receiving a request and producing a response or completing an operation.

A performance metric measuring the rate at which an inference engine generates output tokens per second.

The total duration required for an AI system or service to process an input query and return the completed output.

The duration of time it takes for a language model to produce its initial output token after receiving a request.

A metric that measures the average monetary cost incurred to process a single API call, inference, or user request.