Regression Testing for AI Systems: Engineering Practices for LLM Evaluation

In traditional software systems, you modify a function, run the unit tests, and if everything passes (all green), you commit. This approach fails in LLM applications: outputs are stochastic. For the same question, the model might return "Okay" this time and "Okay, correct" next time. If you tweak a prompt, how do you know you haven't broken something else? The answer lies in standard software engineering practices: build a regression test suite.

First, Compress "Output" into a "Score"

The key to evaluation is not comparing which output looks better, but compressing the output into a deterministic "judgment." Scoring methods fall into two categories:

- **Deterministic Rules**: Can the JSON be parsed? Are required fields non-empty? Does it include or exclude specific keywords? Is the SQL syntax valid? Does the answer match the reference answer? Anything that can be coded should be coded. This approach has zero cost and zero variance.

- **LLM as Judge**: For open-ended quality metrics (appropriate tone, missed potential risks, completeness of explanation), use another model to score on a 1–5 scale. Require the model to provide reasoning, and embed the scoring rubric item-by-item into the prompt.

A rule of thumb: If a rule can determine the outcome, do not invoke a judge model. Judges also exhibit variance; each additional model layer adds noise and incurs token costs.

Consider a concrete example: an insurance policy Q&A bot. The rule layer includes three checks: the output must contain the phrases "deductible," "waiting period," and "reimbursement ratio"; the total word count must not exceed 300; and it must be valid JSON with non-empty fields. The judge layer’s rubric includes only two items: whether incorrect clause numbers were cited (5-point scale) and whether any paragraph in the explanation exceeds three sentences (5-point scale). The combined output of these two layers is a pair of scores plus a detailed error report, not the response itself. Reviewers look at "which check failed and where," without needing to reread the model's output.

The Test Set Must Be Fixed and Version-Controlled

- **30–100 cases are sufficient**; beyond that, marginal returns diminish. Select cases from real production logs, ensuring they include: actual user queries, complaints received from users, edge cases, and the top 10 most frequent requests.

- **Store in Git**: Version control the questions, reference answers, and rubrics together. The test set is an asset, not a temporary file. When you switch models later, you rely on it for comparison.

- **Lock Variables**: Fix the sampling temperature, and lock down the model version and prompt version. Change only one variable at a time; otherwise, if the test set indicates a "change," you won’t know who is to blame.

Set Up CI Gates

Run the full test suite before making changes, recording the pass/fail status of each case as a baseline. Run it again after changes and compare. Recommended criteria for pass/fail (red/green):

1. The overall pass rate must not drop below the baseline.

2. No single case is allowed to change from "pass" to "fail" (new failures can be discussed but require justification).

3. Hard rule-based errors (JSON parsing failure, missing required fields) result in an immediate fail—zero tolerance.

Integrate this process into CI: for prompt changes, model upgrades, or dependency version updates, run the evaluation job first. Do not merge unless it passes. This is the "pre-commit check" for probabilistic systems.

Should you keep a report after each run? Yes. The report format should consistently include four sections: total score, pass rates by category, itemized reasons for newly failed cases, and a list of newly passed cases. The goal is to enable someone to answer "what did this change break?" within thirty seconds. If you cannot achieve this, the evaluation becomes a black-box score that no one will monitor long-term, rendering the CI gate ineffective.

Three Common Pitfalls

1. **Test Set Contamination**: Questions leak into the model's training data or appear in few-shot examples, leading to inflated scores that mask real changes. After switching to a new model, manually spot-check 10 cases before trusting aggregated metrics.

2. **Overfitting the Test Set**: Repeatedly tweaking the prompt against 50 cases until all pass, only for old issues to resurface in production. Regularly retire old cases and add new ones to keep the test set aligned with the production distribution.

3. **Judging by Single Cases**: One failure does not mean the change is bad, nor does one success mean it is good. Focus on aggregated metrics and the list of regressions; do not obsess over individual outputs.

What Problem Does Evaluation Actually Solve?

Evaluation does not seek high scores; it seeks "no regression." If you scored 85/100 before a change and drop to 72/100 after, you must identify which types of cases failed and why, fix them to reach at least 86, and then deploy. For AI systems, the value of regression testing is not guaranteeing model correctness, but ensuring that when you break something, your team knows about it immediately rather than waiting for user complaints.