The P99 Latency Doubled, But It Wasn’t the Model’s Fault

Last month, I helped an internal team troubleshoot performance issues with their intent classification service. Regional monitoring showed that P99 latency had quietly crept up from 310ms to 740ms, and it took about three days before anyone triggered an alert. The model hadn’t changed, the GPUs were the same, and the prompt templates were untouched. Our first instinct was to involve the infrastructure team, but after spending an entire afternoon investigating, we couldn’t find any outliers. The root cause ultimately lay in the request gateway layer: a rate-limiting middleware, deployed without any monitoring coverage, had forced both batch preprocessing tasks from the chat backend and live online requests into the same connection pool.

The incident itself was minor, but it exposed three recurring pitfalls in AI delivery projects, each worth documenting separately.

First, Core Paths Were Missing “Middleware Layer” Metrics

Our original monitoring for the entire inference service focused solely on the application layer: token counts, time-to-first-byte (TTFB), and request body size. These metrics work well under normal conditions, but this issue originated at the gateway, where every application-layer metric appeared normal. In retrospect, we added three sets of metrics: gateway wait time, connection pool hit rate, and request queue depth per tenant.

There’s a trade-off here that many teams hesitate over: middleware metrics can be dense, making dashboards noisy. My subsequent approach was to route P99 metrics from the middleware layer to a dedicated alerting channel, keeping them off the main dashboard to prevent daily noise from drowning out genuine issues.

Second, Running Batch Jobs in the Same Thread Pool Means No Isolation

The backend batch generation tasks were running within the same FastAPI worker processes. Looking at the batch endpoint alone, there was no bug—it was simply slow—but it blocked the entire event loop, directly dragging up latency for online requests. Once we moved batch tasks to separate worker processes, similar issues never recurred.

The cost of splitting processes is low, but it requires someone to ask the right question. The lesson here is that the performance review checklist should include a section explicitly listing which endpoints share resources and which are isolated, rather than relying on post-mortem reflections every time.

Third, Alert Thresholds Were Based on Averages, Not Percentiles

The initial P99 alert threshold was set at 500ms, but the actual alert was triggered by request body timeouts, not the P99 metric itself. In other words, the monitoring system did detect the problem, but the attribution pointed to the wrong layer, sending us down the wrong troubleshooting path for half a day. We later restructured the thresholds using percentiles: an absolute value alert at 600ms, and a second alert if two consecutive windows exceeded the baseline by 30%. This approach is more sensitive than a single threshold without generating excessive false positives.

The Remediation Process Deserves Its Own Standardization

From detection to recovery, the actual downtime lasted 22 minutes. The process was: toggle the gateway rate-limiting switch → move batch workers to an independent thread pool → observe P99 → close the loop. None of these steps were improvised on the spot; they relied entirely on a pre-written runbook.

My habit is to maintain “recovery plans” and “diagnostic plans” separately: recovery plans are checklists to ensure actions stay on track under pressure, while diagnostic plans serve as a knowledge base for troubleshooting paths, accumulated after the fact. Mixing them usually results in neither being done well.

None of these pitfalls are difficult, but avoiding them requires someone to think ahead. There’s no passing grade for monitoring—only varying levels of granularity. Isolation isn’t a gatekeeper; it’s simply a matter of resource separation. If this performance troubleshooting case offers you any reference value, then these specific metric names and the rationale for splitting processes are the key takeaways I want to leave behind.