Slow Isn’t About the Model: Four Quantifiable Components Hidden in AI Service Latency

Last week, I helped a team troubleshoot a production issue: their customer service bot was responding too slowly, leading to user complaints. The boss’s immediate reaction was, “Switch to a faster model.” After making the switch, the P95 latency dropped by about 400 milliseconds, yet the volume of complaints remained unchanged. Where did things go wrong? They had only measured the “model inference” phase, but model inference accounts for only a small fraction of the wait time perceived by users.

If we break down a complete AI service call, latency typically consists of four parts: **network round-trip, queueing wait, prompt processing (prefill), and token-by-token generation (decode)**. These four phases have fundamentally different characteristics and require distinct optimization strategies. Lumping them together leads to vague conclusions like “just switch models.”

**Network round-trip** is straightforward to handle; simply measure `t_proxy`: the time from sending the request to receiving the first byte. With cross-region traffic叠加 TLS overhead, a single round-trip of 80 to 200 milliseconds is normal. Optimization here involves deploying inference services closer to users or using streaming responses to maintain persistent connections. The impact is limited, but the upper bound is clear.

Also, don’t forget the segment **between the client and the gateway**: if the user-side API timeout is set to just 10 seconds, your service might technically complete the response in 12 seconds, but to the user, it appears “down.” When coordinating across teams, aligning timeout settings on both sides yields faster results than tweaking any other parameters.

**Queueing wait** is the most easily overlooked phase and also the most prone to causing issues. Under continuous batching, your request might be queued behind dozens of others. For the same “7B model,” a first-token latency of 200 milliseconds during idle times isn’t unusual, nor is 3 seconds during peak hours. Since this latency is strongly correlated with traffic volume, you must monitor it using percentiles—averages are deceptive; P95 and P99 reflect the true user experience.

A common pitfall is treating “time to first token” as the overall SLA. The server might report a P95 first-token latency of 800 milliseconds, which feels responsive; however, if the average output is 500 tokens and the decode speed is 40 tokens per second, the user has to wait 13 seconds for the complete answer. A fast first token only affects whether anxiety sets in, while the total time to receive the full answer determines whether the user gives up waiting. Both metrics need monitoring.

**Prefill** is also related to concurrency, but it has a hard lower bound: processing a long prompt with tens of thousands of tokens will take several seconds for the first token, even if the GPU is completely idle. Therefore, services handling “long document Q&A” and those handling “short queries” have inherently different latency baselines and cannot be governed by a single SLA.

In practical tests, a frequent observation is: for the same model, a 10k-token prompt yields a first-token latency of about 1 second, whereas an 80k-token prompt jumps directly to 8–10 seconds. This isn’t because the model has slowed down; it’s being bogged down by the prompt length. Thus, approaches that “stuff the entire conversation history into the prompt” incur linearly growing latency costs. Users may not perceive the cause, but both bills and latency spike. Truncation, summarization, or retrieval-augmented injection of only relevant segments are all leverage points specifically targeting this phase.

**Decode** outputs tokens one by one, with speed primarily determined by VRAM bandwidth (since the KV cache must be read for every generated token). Upgrading to VRAM with higher bandwidth provides direct benefits. However, once users see the first character, their anxiety decreases. Therefore, a 50% slowdown in decode speed might not trigger complaints, whereas a 500-millisecond delay in the first token certainly will.

This explains why you might feel “the response isn’t fast, but no one is complaining,” while a competitor with “fast initial sentences but rough endings” actually delivers a better experience—the sense of progress is itself a product capability, making streaming output essentially a standard feature.

How to Implement

Three actions, achievable within two weeks:

1. **Break down timing**. Record four timestamps at the gateway layer: request arrival, first byte, an intermediate streaming point (e.g., the 10th token), and completion. Without this breakdown, all optimization is guesswork.

2. **Monitor using percentiles**. Report latency for all four phases separately using P50/P95/P99. Trigger alerts if any phase deteriorates by more than 30% in trend, rather than waiting for user complaints.

3. **Set scenario-based SLAs and fallbacks**. For short-query services, commit to a P95 first-token latency of < 1.5 seconds; relax this separately for long-document scenarios. When queue length exceeds a threshold, immediately fall back to a smaller model or return a “queuing” status. Don’t let requests expand indefinitely in the queue—100 requests stuck in a 30-second queue are worse than letting half of them proceed via a smaller model within 3 seconds.

“Switching to a faster model” is only cost-effective when the decode phase accounts for more than 50% of latency and the current model is indeed an order of magnitude slower. Before taking action, spend a day measuring the four latency components—most teams will discover after measuring that they spent ages waiting in queues, having upgraded their GPUs for nothing.