Latency vs. Throughput: The Performance Tradeoff Every AI Deployment Faces
When an AI system moves from development to production, a new set of questions emerges that weren't relevant during training or evaluation.
How quickly does it respond to a single request? How many requests can it handle simultaneously? What happens when demand spikes? These aren't model quality questions. They're infrastructure questions, and the two metrics at their center are latency and throughput.
Understanding what each measures, and why optimizing for one tends to come at the expense of the other, is increasingly relevant for anyone involved in decisions about how AI gets deployed.
Latency is the time between a request being sent and a response being received. In a consumer AI application, it's the gap between pressing send and seeing the answer appear. Low latency means fast responses. High latency means waiting, which in user-facing applications erodes experience quickly. Research consistently shows that users abandon interactions that feel slow, and "slow" in the context of AI assistants is often measured in seconds rather than minutes.
Throughput is different. It measures how many requests a system can process in a given period of time, typically stated as requests per second or tokens per second. A system with high throughput can handle many simultaneous users without degrading. A system with low throughput becomes a bottleneck when demand increases, queuing requests and causing the latency experienced by each individual user to climb.
The tension between them emerges from how AI inference actually works. Language models generate output token by token, and each token generation requires a forward pass through the model. Running those forward passes in sequence for a single request, as fast as possible, minimizes latency for that request. But the hardware running the model, typically a GPU or specialized AI accelerator, has capacity that could be used to process multiple requests simultaneously through a technique called batching.
Batching groups multiple requests together and processes them in parallel, which dramatically improves throughput. The same hardware that might handle ten requests per second sequentially might handle fifty or more when batching is applied effectively. But batching introduces a tradeoff. A request that arrives when a batch is already being processed has to wait for the next batch. Individual response times increase. Throughput goes up; latency goes up with it.
The right balance depends entirely on the use case. A real-time customer-facing chat application has strict latency requirements. Users expect responses in under a second or two, and any batching strategy that compromises that will produce a poor experience regardless of how efficiently it uses the underlying hardware. An internal document processing pipeline that runs overnight has essentially no latency requirement. Processing a document in three seconds versus thirty seconds doesn't matter if the results will be ready by morning. In that context, aggressive batching to maximize throughput is exactly the right approach.
Several other factors shape where a system lands on the latency-throughput spectrum. Model size is a primary one. Larger models with more parameters produce higher quality outputs but require more compute per forward pass, which increases both latency and the cost of throughput. Quantization, a technique that reduces the numerical precision of model weights, can reduce compute requirements and improve both metrics at some cost to output quality. Caching, storing the computed representations of common inputs rather than recomputing them on every request, can dramatically reduce latency for repeated or similar queries.
Hardware choices matter too. Different accelerators have different characteristics in terms of memory bandwidth, compute density, and the efficiency with which they handle different batch sizes. Serving infrastructure, including load balancing, autoscaling, and the frameworks used to manage model serving, all affect how well a deployment handles variable demand without either wasting resources during quiet periods or degrading under peak load.
For non-technical stakeholders involved in AI deployment decisions, the latency-throughput tradeoff is a reason to be specific about requirements before choosing infrastructure. "We need it to be fast" and "we need it to handle a lot of users" are both reasonable requirements, but they can point in different directions, and the cost of optimizing for both simultaneously is higher than optimizing for either alone. Knowing which matters more for a given use case is the starting point for making sensible infrastructure decisions rather than expensive ones.