AI Systems Field Notes: The Trade-off Between Inference Latency and Throughput

Abstract

When deploying Large Language Models (LLMs) in production environments, engineers often face a trade-off between **inference latency** and **throughput**. This article explains the meaning of these two core metrics, the inherent conflict between them, and common engineering compromises.

Key Concepts

Latency

Latency refers to the time required from sending a request to receiving the complete response. For interactive applications (such as chatbots or code completion), low latency is critical. Users typically perceive delays exceeding 200–500ms.

Throughput

Throughput refers to the number of requests or tokens a system can process per unit of time. High throughput is more important for batch processing tasks, offline analysis, or high-concurrency scenarios.

The Core Conflict

There is an inherent tension between latency and throughput:

1. **Batching Effects**: Combining multiple requests into a single batch improves GPU utilization, thereby increasing throughput. However, each request must wait for the batch to fill up, which increases the latency for individual requests.

2. **Resource Contention**: Serving more requests simultaneously competes for memory bandwidth and computational resources, potentially lengthening the processing time for each request.

3. **Queuing Delay**: In high-throughput scenarios, requests may wait in queues, increasing end-to-end latency.

Common Compromises

1. Dynamic Batching

The system dynamically adjusts batch sizes based on current load:

• **Low Load**: Use small batches or process single requests to prioritize low latency.
• **High Load**: Increase batch sizes to prioritize throughput.

2. Continuous Batching (Iteration-level Batching)

Traditional batching requires all requests to start and finish simultaneously. Continuous batching allows new requests to join the batch immediately as old requests complete, significantly improving GPU utilization without sacrificing too much latency. This is a core optimization in modern inference engines (such as vLLM and TGI).

3. Speculative Decoding

This method uses a small, fast "draft" model to generate candidate tokens, which are then verified by the larger model. While this approach can reduce latency while maintaining output quality, it increases computational overhead, which may impact throughput.

4. KV Cache Optimization

In autoregressive generation, the Key-Value pairs (KV cache) of previous tokens can be reused. Optimizing KV cache management (e.g., using PagedAttention) reduces memory fragmentation, supporting larger batches and lower latency.

Practical Recommendations

| Scenario | Priority Metric | Recommended Strategy |

|------|----------|----------|

| Real-time Chat | Low Latency | Small batches, continuous batching, KV cache optimization |

| Batch Document Processing | High Throughput | Large batches, asynchronous processing |

| Mixed Workloads | Balanced | Dynamic batching, request classification routing |

Conclusion

There is no single "best" configuration. Engineering teams must find the appropriate balance between latency and throughput based on user expectations, hardware costs, and business goals specific to their application scenarios. Monitoring actual production data and making continuous adjustments is essential practice.

---

*This article avoids mentioning specific vendors or sensitive company names, focusing instead on general systems engineering principles.*