How to Implement Retry Logic for LLM API Calls: Which Errors to Retry and Which to Avoid

Anyone who has worked with LLM services has written this code: a call fails, so you wrap it in a `for` loop to retry three times. It seems robust, but half of production incidents are related to these three lines of looping code. The failure behavior of LLM APIs differs from traditional HTTP APIs, so retry strategies must be designed according to their specific characteristics.

First, Categorize the Errors

LLM API failures generally fall into five categories:

- **429 (Rate Limiting)**: Triggered when hitting RPM or token quota limits. The response headers usually include `Retry-After`. You should retry, waiting for the duration specified in the header.

- **5xx Errors and Network Timeouts**: Caused by instability in the upstream model service. You should retry, but these attempts must be counted toward circuit breaker statistics.

- **Streaming Hangs**: The connection is partially broken; the client doesn't throw an error but simply stops receiving new tokens for an extended period. You must set an idle timeout (e.g., disconnect if no data is received for 20 seconds) and treat timeouts as failures.

- **400 (Parameter Errors, Context Length Exceeded)**: Do not retry. If the input remains unchanged, the 101st attempt will fail just like the first. These errors indicate issues that need immediate fixing: input too long, malformed JSON, or incorrect model names.

- **Content Moderation Rejections**: Some services return specific error codes for this. Retrying will not bypass the filter; it will only waste money.

The rule of thumb is: **Only retry errors where the state might change.** 400 errors and moderation rejections are deterministic; retrying them will consistently result in failure.

How to Calculate Backoff

Do not use a fixed 1-second interval. If 20 requests hit the rate limit simultaneously, a fixed interval means all 20 will collide again at the same time. Use exponential backoff with jitter: `wait_time = 1s * 2^n * rand(0.5~1.5)`, capped at 30 seconds. If a 429 response includes a `Retry-After` header, prioritize the value provided by the server instead of calculating it yourself.

Cap retries at generally 3–4 attempts, and add a total timeout (e.g., wait no more than 60 seconds for the entire task). If the first three attempts fail, it is likely a server-side issue, making a fourth blind attempt meaningless. The total timeout serves as a contract with the upstream service: synchronous interfaces must return a 503 status by the deadline, or switch to asynchronous tasks for user polling, rather than hanging indefinitely.

The Hidden Cost of Retries: Duplicate Billing

This is a pitfall specific to LLMs: **If the request was actually executed but the response wasn't received, will you be charged twice if you retry?** A typical scenario: the request succeeds, the model finishes generation, but the network drops at the last moment, causing the client to experience a timeout. When you retry, the previous call has often already been billed.

There are two levels of countermeasures. For small requests (under a few hundred tokens), the loss is negligible and can be ignored. For large tasks (long-form generation, batch processing), use idempotency keys: include the same key in the request, and the server will return the existing result for duplicate keys instead of recalculating. Some APIs natively support this (e.g., via the `Idempotency-Key` header). If the service does not support it, store request fingerprints (input hash + parameter hash) in Redis with a TTL of 30 minutes, and check for existing results with the same fingerprint before retrying.

Circuit Breaking and Degradation

When calling multiple model services, implement a circuit breaker: if the error rate exceeds 50% within a sliding window (e.g., 1 minute), trip the circuit. New requests should immediately follow a degradation path, with a probe request sent every 30–60 seconds to check for recovery. You can use libraries like `pybreaker` or `Resilience4j`, or write your own implementation in about ten lines of code.

A practical degradation sequence for streaming interfaces is: Retry fails → Switch to a backup smaller model and label the result → If that still fails, return a cached conservative answer (labeling the cache source) → If that still fails, explicitly inform the user that the service is busy and return a 503. Do not let the caller wait indefinitely for a stream that will never arrive.

One final, often overlooked detail: Track "attempts" rather than just "requests." If an endpoint has a 90% success rate and all failures are silently rescued by retries, monitoring will show 100% success. When a real fault occurs, you won't be able to distinguish between normal fluctuations and a complete service outage. Keep the number of attempts in your logs, and track two metrics in your monitoring: "First-Attempt Success Rate" and "Final Success Rate." They may look similar under normal conditions, but during an incident, the divergence between these two lines is the alert signal you need most.