Inference Timeouts Are Not Black Magic: Set Three Numbers for LLM Calls

The dilemma over timeouts for LLM calls often plays out like this: 20 seconds is too short, occasionally killing long prompts; 120 seconds is too long, causing your queue to clog up before the upstream service even fails. Most teams end up picking a number out of thin air, only to rehash the same argument during every incident review.

In reality, you only need to finalize three numbers: **budget**, **hard timeout**, and **retry policy**. Once decided, write them into your configuration, and the rest is just execution.

Clarify Failure Modes First

Timeouts stem from different sources, and setting a single threshold for all of them will only cause conflicts:

- **Model Queuing**: Upstream concurrency is maxed out, so requests sit idle in the queue. Characteristics: Overall latency increases, and successful responses become slower.

- **Long Prompts / Long Outputs**: Prefill and decode steps are inherently slow. Characteristics: Latency is strongly correlated with token count.

- **Upstream Hangs**: The connection is established, but no response is ever returned. Characteristics: Individual requests hit the timeout limit exactly.

The solutions for these three scenarios are completely different: rate limiting for the first, adjusting the budget or switching models for the second, and timeout + retry for the third. Timeouts can only save the third scenario, but they apply to all three—which is why they serve as a safety net, yet are insufficient on their own.

Budget: Budget = p95 × 1.5

Don’t reason, “Average latency is 3 seconds, so I’ll set it to 10 seconds.” Instead, look at the p95 latency of all calls from the last 7 days (bucketing by prompt token count yields better accuracy) and multiply by 1.5 to establish the latency budget for normal calls.

Example: If an endpoint’s p95 is 8.2s (for the medium prompt bucket), the budget ≈ 12s. Occasional requests exceeding the budget are not “timeout failures” but long-tail cases. They should be counted separately in monitoring rather than mixed into the error rate. This budget figure serves as an anchor for future timeout adjustments, eliminating the need for repeated debates.

Hard Timeout: Budget + Buffer

Set the hard timeout to 1.5–2 times the budget, but you must account for decode characteristics: **the longer the output, the less appropriate a fixed cutoff becomes**. If the interface supports streaming (SSE), a better approach is to detect idleness only—apply a prefill timeout (e.g., 15s) before the first token, then reset the timer upon receiving each chunk. If the interval between two chunks exceeds 30s, declare the request dead. This prevents legitimate long outputs (e.g., 20k tokens) from being incorrectly terminated, while still cutting off truly hung requests.

For non-streaming interfaces, stick to a single hard timeout. Rule of thumb: `hard timeout` ≥ the maximum observed duration for a valid output on that interface. Otherwise, you will categorize “slow but successful” requests alongside “dead” ones.

Retries: Retry Only Once, Codify the Conditions

The pitfall of retries lies not in the count, but in the conditions:

- **Retry only on HTTP 5xx errors and connection drops**. Include 502/503/504. Treat client-side timeouts as a partial case (grant one retry opportunity).

- **Do not retry on 429; log as a failure due to rate limiting**. Retrying on 429 merely amplifies queue pressure; the rate limit won’t disappear. A 429 should trigger “throttling,” not “retrying.”

- **If a request times out after one retry → log as a failure**, letting the caller decide on degradation. Do not use exponential backoff: on online paths, a 30-second backoff harms user experience more than an immediate failure. Exponential backoff is only worthwhile for offline batches.

Additionally, retries must include idempotency checks. If the same user request reaches the model twice, the user might receive two different answers, or be billed twice. Use an idempotency key if possible; if not, at least link them via the same `request_id` in logs so you can clearly distinguish during reproduction whether it was “one failure” or “two partial successes.”

Encode the Three Numbers in Configuration

llm_call:
  budget_ms: 12000        # p95 x 1.5
  timeout:
    prefill_ms: 15000     # Streaming: before first token
    chunk_idle_ms: 30000  # Streaming: interval between chunks
    hard_ms: 30000        # Non-streaming: hard limit
  retry:
    max: 1
    on: [http_5xx, connection_error, hard_timeout]
    not_on: [http_429, http_4xx]

Align your alerts with this structure: count timeout rates in three layers—“long-tail within budget,” “hit hard timeout,” and “failed after retry”—with independent alerts for each. This way, incident reviews reveal not a vague “3% timeout rate,” but actionable insights like “increased long-tail → upstream needs scaling” or “more hard timeouts → something is broken.”

Conclusion

The essence of timeouts is not “how long to wait before giving up,” but distinguishing between two signals: “upstream is slow” and “upstream is dead.” By fixing the budget, hard timeout, and retry conditions, every timeout incident falls into one of the three categories above, leading to a unique remediation action. A timeout configuration that allows for this distinction is complete; the remaining task is simply to periodically calibrate the budget using the latest p95 data.