Build an LLM Evaluation Suite in 15 Minutes: No MLflow, No Platform

Every time you switch models, tweak prompts, or adjust the temperature, you want to know if the results are getting better or worse. But often, you’re stuck reading through a few freshly generated outputs and relying on gut feeling to say, “It feels about the same.” This approach might survive a demo, but it won’t survive production: feelings are unstable, judgments vary between people, and in two weeks, you won’t even remember what the previous baseline looked like.

An evaluation suite is the antidote, but it doesn’t have to be heavy. The solution below requires just one JSON file and one Python script. You can get it running in 15 minutes, provided you have a list of real-world tasks on hand.

Step 1: Where Do the Questions Come From?

**Never hand-craft idealized questions.** Instead, dig into your production logs or beta user feedback. Filter for requests from the last 30 days where the model “answered incorrectly” or “the user had to ask a follow-up question.” Each of these cases becomes a future test case.

When filtering, keep three types:

- Cases with incorrect answers (clear right/wrong outcomes)

- Cases that were correct but overly verbose or absurd (quality issues)

- Edge cases (extremely long inputs, mixed Chinese/English text, ambiguous instructions)

10–20 cases are enough to start. Maintaining more than 50 becomes a burden; the first 20 provide the most value because they all stem from your actual memory of failures.

Step 2: What Does Each Question Look Like?

Use a JSON array with three fields per question:

{
  "id": "invoice-overflow-001",
  "prompt": "Here is a bill for $47.99, but the plan costs $45. Please handle the difference.",
  "expect": {
    "contains_any": ["$2.99", "2.99"],
    "max_tokens": 200
  }
}

The `expect` field is your assertion layer. Choose assertions based on the task type:

- **Factual tasks** (calculation, extraction, translation): Use `contains_any`, `not_contains`, or exact matching. Automate scoring wherever possible.

- **Formatting tasks** (requiring JSON or tables): Add `json_valid: true` or perform a schema check.

- **Stylistic tasks** (copywriting, rewriting): Avoid hard-coding keywords. Instead, leave a plain-language note in `expect_notes`. After the run, do a quick human review—spending 60 seconds per case is faster than maintaining fragile keyword checks.

Key principle: **Never use human judgment for what can be automatically asserted. Reserve human review only for truly subjective aspects.** If you find yourself writing and deleting half of your `expect` fields because you don’t know what to put, those questions aren’t suitable for the current version of your evaluation suite.

Step 3: Scoring and Comparison

The core of the script is just 20 lines: loop through each question, call the API once, collect the output and latency, run assertions against `expect`, and output a table:

| case | Pass | Latency | Notes |

|------|------|------|------|

| invoice-overflow-001 | ✅ | 2.3s | |

| mt-fallback-zh-003 | ❌ | 4.1s | Missed "keep half-width parentheses" |

The real value of comparison lies here: Before changing models or prompts, run a baseline and save the outputs to `runs/20260901-before/`. After making changes, run again and save to `runs/20260901-after/`. Then, diff the two folders. Focus on three types of changes:

1. **Previously correct, now incorrect** (Regression: most severe, fix first)

2. **Previously incorrect, now correct** (Improvement: verify why it’s now correct)

3. **Incorrect in both, but the error changed** (Neutral: make a note, observe trends)

Don’t aim to cover every dimension at once. The first version should answer only one question: **How many of our existing cases broke due to this change?**

Three Common Pitfalls

**Pitfall 1: Data Leakage.** If live users encounter the same prompt, and that prompt appears in public documentation, error messages, or logs, you’ve essentially opened a cheating channel. Keep question IDs and prompts confidential.

**Pitfall 2: The False Certainty of temperature=0.** Many models still exhibit slight output variability even at temperature 0. Different outputs across two runs don’t necessarily mean the model has gotten worse or better. To confirm a “regression,” run the test at least twice; record it as a failure only if it fails both times.

**Pitfall 3: The Suite Only Grows, Never Shrinks.** Move cases that have “passed correctly five consecutive times and never failed since” to an `archive/` subdirectory. Your main suite should always contain questions that differentiate model performance, not just a historical archive of every issue your company has ever faced.

Where to Start

Open today’s API logs, filter out 10 requests with “unsatisfactory answers,” write them into JSON, add assertions, run the script, and save the results. Congratulations, you now have Evaluation v0.1. Next time you tweak a prompt or switch models, run v0.1 first. You’ll get a number, not a feeling. That number is the contract between you and the next version.