Health Checks All Green, Model Calls All Failing: The False Security of "Port Is Alive"

At 2 AM last Thursday, the daily update pipeline got stuck at the translation step; not a single line of the three-language publication went through. The on-duty script’s logs contained only one sentence: router connection timeout. Almost all health checks were green—the port was open, the process was running, and `/health` returned 200.

Scene Reconstruction

The inference service on that machine had switched model versions just the week before. The upstream team had renamed the model from the old version to the new one. Our validation logic failed to keep up:

- The health check script only ran `curl /health`; if the port was alive, it returned `healthy`.

- The actual call path was `/v1/chat/completions` with a specific model name, a path that no one had tested.

- The old model name had already been deprecated, causing all calls to return 404, with error messages stating `model_not_found`.

There was also an element of false confidence: the changelog for the version change had actually arrived in the group chat that morning. Two colleagues saw it but both assumed, "The routing layer should adapt automatically," so no one checked our own configuration list. In hindsight, this was a typical case of "interface contract change, each side only half-updated"—the upstream changed the name, we were still calling the old name, and the monitoring in between happened to focus only on "whether the process is running." All three layers failed.

To make matters worse, the fallback logic in the deployment script for translation failures was "retry three times, then write placeholder content and continue." This means that if no one had monitored the alerts, the day’s three-language articles would have gone live with placeholder text, even though unit tests for every stage were green. The placeholder fallback was originally intended for network jitter, but it was exploited by a deterministic error like "wrong model name."

Three Actions Decided During the Post-Mortem

**1. Change health checks to functional self-tests.** The new script no longer just checks the port. Instead, it follows the exact same path as production: the same endpoint, the same model name (read from configuration, not hardcoded), the same minimal prompt, and the same key-loading chain. It runs once per minute, with one complete request counting as one heartbeat. Any 4xx/5xx response at any step immediately turns the status red. We had already suffered from this issue when switching models, so model names are now unified in the configuration file. Both the check script and the business script read from the same source; no one is allowed to hardcode names in their own code.

**2. Reserve fallback retries for transient errors only.** Narrow down the "retry and continue" logic: perform limited retries only for 5xx errors and timeouts. For 4xx errors (parameter errors, model not found, authentication failures), fail immediately and mark as red—never write placeholders. The principle is simple: you can cover for network jitter, but not for configuration errors. Covering them up means paying for bad configuration with live content.

**3. Include success rates in daily reports, rather than checking only after incidents.** Aggregate the success rate, P95 latency, and 4xx details for each step daily, and trigger alerts if thresholds are breached. This time, the issue was discovered manually at 2 AM. If traffic had been higher or alert delays longer, the placeholder content would have been published. After the fact, there is no such thing as "good catch"; we can only say that monitoring failed to catch it. We cannot rely on luck as part of our design.

Implementation was faster than it sounds: the functional self-test script is just a single file. It reuses the deployment script’s own key-loading chain and configuration reading functions, avoiding reinventing the wheel and preventing discrepancies where the "check path" and "business path" use different keys or configurations. The daily report task is attached to the existing cron job, outputting a small three-column table: Step, Success Rate, and Slowest Tier Latency. Thresholds are initially set roughly (alert if success rate drops below 95%) and will be tightened after two weeks of operation. Avoid over-engineering with fine-tuned parameters right from the start.

For Those in the Same Lane

If you also run multi-node pipelines with model names scattered across various machine configurations, I suggest spending half an hour on two tasks: First, centralize all "model names" into a single configuration file shared by both check scripts and business logic. Second, upgrade your health check from `/health` to a real, minimal request. The first is a one-time effort; the second prevents an entire class of "port is alive, service is dead" issues. The original 404 error message is still preserved in the alert archives, complete with format, model name, and parameters—a reminder to ourselves: Green doesn’t necessarily mean good.