Post-Mortem: How a "Silent Pipeline" Let High-Value Detection Windows Slip Away for Two Weeks

Last month, we spent an entire week troubleshooting the issue of "model performance degradation on specific categories" within one of our identification pipelines. The root cause turned out to have little to do with the model scores themselves. Instead, it was an upstream pipeline event involving a shifting time window. This extended the availability timeline for detection tasks, and when combined with an off-peak scheduling change known only to a subset of teams, it caused the comparison window for high-value samples to shift backward overall.

Breaking down the process, four specific actions proved to be the effective ones.

The Symptoms We Observed

The business side reported for two consecutive days that "anomalous items in high-value categories were not detected until 8 hours after warehousing." There were no red flags on the monitoring dashboard: detection model scores were normal, throughput was stable, and there was no queue backlog. This combination is dangerous: all metrics appeared healthy, yet the delivery time had already been compromised.

Step 1: Check Delivery Timestamps First, Not Model Scores

We separately recorded the "model score delivery time" and the "data pipeline delivery time" for batches over the course of a week. The result showed that the model score delivery time remained virtually unchanged. However, due to an off-peak scheduling adjustment, the data pipeline delayed the start of detection for the daily 03:00 high-value batch until 04:40. The 8-hour delay perceived by the business side was calculated starting from 04:40.

The lesson here: When troubleshooting "degradation in a specific window," the first step should be to align the timelines, not to tweak the model. If the scores are stable but delivery has slipped, the issue is almost certainly within the pipeline.

Step 2: Add a "Next Batch Ready" Probe to the Pipeline

We added a lightweight probe. It does not detect business anomalies; it only confirms that "the previous batch of data completed disk writing and sharding within T+15 minutes." While this probe generates no scores, it outputs a timestamp that previously went unmonitored. After deploying it in the second week, troubleshooting similar issues required checking just one metric, rather than sifting through logs from three different systems.

Step 3: Document "Off-Peak Scheduling" in Change Requests, Not Chat Groups

The scheduling change that moved the 03:00 batch to 04:40 was part of the preparation for stress testing during a major promotion, aimed at off-peak operation. This change was only communicated in an operations chat group. Neither the detection team nor the model platform team was aware that their upstream timeline had changed. Two months later, we instituted a new rule: any scheduling changes that alter delivery time windows must be documented in the daily change request, listing the owners of affected downstream tasks. That very week, two batches nearly collided, but were identified and corrected in advance.

Step 4: Align Alert Thresholds with Delivery Time, Not Just Scores

The final mechanism for incident containment involved changing the alert targets. Previously, alerts monitored only "model scores below threshold." Now, they simultaneously monitor "delivery timestamps later than scheduled by 30 minutes." The score-based alert remains as a secondary layer. This is because the business side’s actual requirement is "when can I receive the judgment," not "what is the score of this judgment."

Post-Mortem Metrics

- Time spent troubleshooting: 1 week (6 of which were spent wrestling with "why didn't the scores change")

- Actual magnitude of delivery window shift: Approximately 100 minutes per day, lasting for 2 days undetected

- After deploying the delivery probe, similar "late detection" incidents dropped from twice a month to zero

- After switching alerts from score-only to a dual-track system (score + timestamp), one additional pipeline slip was captured in the first week

One-Liner for On-Call Duty

If you see "normal scores but reports of slowness/delays," check the data ingestion timestamps first, then look at the model. Most of the time, the answer lies in the former.