Inference latency
The time between a request reaching a deployed model and its response being returned. It is shaped by model size, hardware, batching strategy, and precision, and it trades directly against accuracy: a smaller or quantized model typically answers faster but a little less well.
Why exams ask this
Tested as a tail-latency scenario: a use case with a strict per-request time budget needs latency measured at the tail, such as p95 or p99, not the average. A mean that looks comfortably inside budget can still hide the slower requests that blow it, and the distractor reports only that mean.
Relevant to
Related concepts
Resources
No resources linked to this concept yet.