Failover for Local Inference Routing: The 38 Minutes We Lost in a Drill

The customer demo was scheduled for 3 PM. During the morning routine inspection, I switched the primary backend from Pool A to Pool B, aiming to test a real failover while traffic was low. As a result, the switch script got stuck at the health check step for 38 minutes, nearly turning the demo into an incident. This article records the troubleshooting steps from that incident and the three rules we subsequently implemented—all lessons learned from our own mistakes.

The First Switch: How Health Checks "Waited" Themselves into a Stall

Our gateway probes liveness in sequence: it requests `/v1/models`, and only considers the backend healthy after receiving three consecutive 200 responses, with a 5-second timeout per attempt. The rule itself was fine; the problem lay in how the switch script was written. It treated a "single probe failure" as "backend unhealthy," while simultaneously treating "unhealthy" as "continue retrying." These two semantic layers overlapped, trapping the process in a 38-minute silent loop: the logs showed a timeout every 5 seconds, yet the script believed it was "waiting normally."

We eventually discovered that the image in Pool B had not finished syncing. Consequently, `/v1/models` returned a 503 error instead of a timeout. Since the health check only looked for "is it a 200?", it categorized all non-200 responses as "wait a bit longer."

**Rule 1: Clearly distinguish between "probe failure" and "confirmed backend unhealthiness," and set a hard cap on total wait time.** Our current failover circuit-breaker conditions are twofold: either five consecutive probe failures for a single backend, or a total wait time exceeding 5 minutes—whichever comes first triggers a rollback. Both thresholds are defined as command-line arguments in the switch script, leaving no room for "wait indefinitely by default."

The Second Switch: The Retry Budget Was Exhausted

A week later, a real fault occurred: the inference process on one node in Pool A crashed. The gateway automatically switched to Pool B cleanly, but customers reported that the latency for the first request spiked to 20 seconds. Investigation revealed this wasn't a network issue, but rather retry amplification: the client SDK defaulted to 3 retries, and the gateway layer added 2 more. Each retry re-entered the queue, meaning the request was actually queued more than 6 times.

**Rule 2: Retry budgets must be globally limited across the entire chain, rather than set independently at each layer.** We changed the client-side retry policy to "at most 2 retries per request lifecycle." Gateway-level retries are now enabled only for idempotent health probes; actual inference requests are never retried at the gateway level—timeouts trigger an immediate switch to the backup pool. After this change, the first-request latency in the same scenario stabilized under 4 seconds. A easily overlooked point here is that while LLM inference requests appear idempotent (submitting the same prompt yields the same result), "resubmission" in a queuing system consumes queue resources repeatedly, making it effectively non-idempotent.

The Third Pitfall: Clock Drift Made "Switch Time" Impossible to Align

During the post-mortem, we tried to correlate logs from both sides and discovered a 40-second clock drift between Pool A and Pool B, caused by NTP failing to sync after a machine reboot. This 40-second discrepancy made it extremely difficult to determine "which request failed on which side."

**Rule 3: Clock synchronization, like health checks, is an infrastructure assumption and must be treated as part of the service's self-probing routine.** The gateway now calibrates local clock deviation every 10 minutes, triggering an alert if the deviation exceeds 5 seconds. All timestamps in switch logs are强制 (mandatory) tagged with time zone information, prohibiting the use of untagged local wall-clock time.

After Integrating the Three Rules into CI

Over the past three months, we have run this drill once a month: switching primaries, cutting networks, and simulating partial image synchronization. We rolled back only once (because insufficient bandwidth in the image registry prevented Pool B from starting); all other drills completed as expected. Every triggered alert was logged, and there were zero instances of "silent failure."

The routing layer for local inference essentially does three things: probing, queuing, and switching. By clearly defining the failure modes for each of these tasks, failover drills become more than just "clicking a button." Rather than switching to a more advanced scheduling framework, clearly defining "behavior after failure" offers a much higher return on investment.