We Fell into the Circuit Breaker Trap Three Times

Last week, we delivered an internal Q&A system for a client. The most embarrassing part wasn’t the model giving wrong answers, but the circuit breaker taking down the entire service. Three times. Each time, it was due to the same configuration, and each time, I failed to anticipate it.

Here’s what happened: The client’s Q&A traffic went through our proxy layer, which connected to a large language model (LLM) API. We set a 30-second timeout and configured the circuit breaker to trip if the error rate exceeded 30%. Once tripped, it would immediately return a fallback message. Sounds stable, right?

**First failure:** The model once generated a response slowly, taking 35 seconds. The proxy logged this as a timeout. Several consecutive slow requests pushed the error rate past the threshold, triggering the circuit breaker. All subsequent questions returned "Service temporarily unavailable." In reality, the model was just slow, not down.

**Second failure:** This occurred after we fixed the timeout issue. We extended the timeout to 90 seconds. However, when the circuit breaker entered the half-open state, it only allowed one probe request through. This request happened to encounter another slow generation, failed again, and sent the circuit breaker back to the open state. The half-open state became an amplifier: it didn’t create errors, but it amplified a single occasional slow request into a global outage.

**Third failure:** This one was the most subtle. We changed the strategy to allow three probe requests in the half-open state. On launch day, traffic was low, and all three requests succeeded, so the circuit breaker "recovered." However, the underlying issue was intermittent. When traffic increased after recovery, the problem recurred, and the circuit breaker tripped again. From the client’s perspective, the service appeared to work normally and then glitched out unpredictably, which made them feel even less secure than if it had simply remained down.

The real fix didn’t come from tweaking circuit breaker parameters more cleverly, but from removing a pseudo-requirement: customer service scenarios don’t need hard circuit breaking. We implemented three changes:

1. **Decoupled timeout from circuit breaking.** Slow requests were no longer counted as errors. Instead, they were routed to a separate rate-limiting queue. Requests were only rejected with a "Please try again later" message if the queue was full, rather than returning "Service unavailable." Customers can distinguish between "queuing" and "outage," and these two concepts have vastly different impacts on trust.

2. **Restricted circuit breaking to genuine dependency failures.** Only connection refusals, 5xx errors, and gateway errors would trigger the circuit breaker. Slow generation was excluded. After this change, the circuit breaker went from triggering eight times a week to not triggering at all for two months.

3. **Added visibility into circuit breaker states.** The backend dashboard now shows whether the state is closed, open, or half-open, along with how many probe requests were allowed and how many succeeded during the half-open phase. With this visibility, we identified the "looks fine but isn’t" issue from the third failure on the same day, without waiting for customer complaints.

One easily overlooked point worth mentioning: the fallback message itself is part of the design. Our original fallback was "Service temporarily unavailable, please try again later." After the refactor, we split it into two scenarios: for queuing, we returned "High volume of inquiries; expect a response in about 1 minute," and for genuine failures, we returned "Service anomaly detected; we have been alerted." After launch, customer complaints containing questions like "Is your system down?" dropped significantly. Text is not just decoration; it is the last line of defense during failures. When users lose access to answers, it is this message that catches them, not the model.

Reflecting on this, my takeaway is: Circuit breaking is a good protection mechanism, but it protects dependencies, not business logic. If you leave it to the circuit breaker to adjudicate "a delay in business terms," it will shut down the entire entry point when you least expect it. To judge whether a protection mechanism is well-designed in a delivery context, look at just one thing: During a failure, do users see "I am queuing" or "You are down"? The former retains users; the latter does not.