Running Evaluations in CI: A Model Upgrade Mishap and Its Fix

Last week, we did something quite routine: we swapped the main model behind our local inference gateway from a 32B parameter model to a 70B one, and also bumped the minor version of the inference framework. As per usual practice, we only ran "smoke tests"—three typical Q&A pairs. Since the answers looked correct, we merged the PR and promoted the gateway to production.

Forty minutes later, tickets started pouring in: batch translation tasks for three users had completely malformed output formats.

During the post-mortem, we dug up our validation records from that day and realized the "smoke tests" happened to cover the model's most stable domain: knowledge-based Q&A. The actual failure occurred in batch translation, which used a different prompt template and required strict JSON output. The larger 70B model turned out to be more "pedantic" about format constraints within long instructions—it started stuffing explanatory prefixes into JSON fields, causing the downstream parser to throw errors and fail the entire batch.

The problem wasn't that the model had become dumber; it was that our evaluation set completely lacked this type of case.

Since then, we’ve established four rules. They don’t cost much, but they block similar issues before they reach production:

**First, organize evaluation sets by "prompt template" rather than by "functional module."** Our live traffic actually uses 11 distinct templates: translation, extraction, summarization, rewriting, etc. Each template has its own dedicated set of evaluation cases, comprising at least 20 real, anonymized samples. When switching models or adjusting inference parameters, all 11 suites must run. If the pass rate for any suite drops by more than 2 percentage points, the release is blocked. This is written into CI as a hard gate, not just a "recommendation."

**Second, score format assertions and semantic assertions separately.** JSON outputs first undergo schema validation; only if they pass do they proceed to semantic scoring. The benefit here is rapid troubleshooting when things go wrong: if the schema check fails, it’s a format issue unrelated to the model’s intelligence, so you immediately check the prompts and parser. If the schema passes but the semantic score drops, only then do you suspect the model itself. The ambiguous zone we experienced last time—"answers look correct but batch tasks fail entirely"—will basically never happen again.

**Third, add a "read-only production traffic sampling" switch to the evaluation scripts.** We automatically save a weekly snapshot of anonymized input samples from production (without touching outputs, to avoid labeling costs) into the evaluation repository as sources for future cases. Production traffic knows best what inputs the model is sensitive to.

**Fourth, model switching is no longer an all-or-nothing cutover; we use shadow traffic.** The new model is first attached as a shadow node behind the gateway, receiving 5% of real traffic. It records responses but does not return them to users. Shadow outputs are compared item-by-item with the primary node: strict diffs for format fields, and similarity calculations for semantic fields. The shadow period lasts at least 48 hours to cover a complete business cycle of peaks and troughs. If degradation in any field category exceeds the threshold, the shadow node is automatically rolled back and an alert is triggered. This time, the 70B model went through this exact process—the JSON pollution in the translation template was caught on the second morning of the shadow period. It was the alert that saved us, not user tickets.

Some might think that running 11 templates × 20 cases × full suite is costly. In reality, running one full evaluation round on a local 70B model takes about 18 minutes. The hardware is already available, so there’s no extra expense. By comparison, investigating and rolling back a single production incident last time took three hours and cost us user trust.

The most painful lesson learned: we once assumed that "correct answers" equated to a "correct system." For LLM pipelines, the answer is just one of many output attributes. Format, length, language purity, and refusal behavior are each independent risk surfaces. You are only immune to the risks covered by your evaluation set.

The upgrade eventually went live—after fixing the format constraints in that translation template, all 11 evaluation suites passed. But from now on, the phrase "it looks correct" will no longer pass muster in review meetings.