Pinning Down the Seed: Minimal Fixes for LLM Reproducibility

Reproducibility is the cheapest quality signal in engineering: only when you run the same input and code twice on the same day and get consistent results can you determine whether a change actually improved the system or if you just got lucky.

In LLM systems, "getting different results on two runs" is so common that many mistakenly believe it’s an inherent property of the model. In reality, most inconsistencies are fixable. There are usually only four invisible variables: `model`, `seed`/decoding, timeouts/network, and data order. Pin these down, and most "sporadic" issues will disappear.

First, pin down the model

The same model name (e.g., `gpt-4o`, `qwen-plus`) may point to different versions behind the scenes. After a provider updates weights, API behavior changes, but your evaluation script remains unaware. A regression test case with 100% reproducibility last month might start failing sporadically the next week—90% of the time, it’s not because your code broke, but because `qwen-plus` silently upgraded to version `2026-06-17`.

The fix is simple: check the `model` field in the response metadata during calls and log it for every request. More importantly, write the exact version number into your evaluation configuration file instead of relying on "whatever is currently deployed." In CI, you can directly assert: `assert resp.model == EXPECTED_VERSION`, and fail if they don’t match. This ensures a one-to-one correspondence between evaluation results and model versions, so you’ll know exactly which version was tested even when reviewing reports three months later.

Next, pin down seed and decoding

Setting `temperature=0` does not guarantee determinism. Samplers and floating-point paths in kernel implementations can introduce jitter, making different punctuation or subsequent phrasing across two runs of the same sentence "valid" outputs. Since most commercial APIs do not expose a seed parameter, the correct approach is to set temperature to 0 (or the lowest setting) and `top_p` to 1, while documenting that "results may still drift by ±1 token." Evaluation assertions should not rely on exact string matches—either compare at the sentence level (embedding similarity ≥0.92) or check only key fields (such as the `status` field in JSON or tool call names).

For self-hosted setups (vLLM, TGI), the `seed` parameter can truly pin down outputs. This step is essential for batch evaluations and regression testing: the token stream from two runs of the same prompt set must be diffable. Note that vLLM’s `seed` parameter works for greedy decoding (`temperature=0`), but for sampling decoding, it depends on the kernel’s support for random number streams. Always verify by running a diff on two consecutive runs in a dev environment first.

Provide safeguards for timeouts and network

Some differences between two runs aren’t due to model changes but to silent degradation after request failures: the first attempt times out, and the retry goes to a different endpoint, which might host a model version differing by a minor release. Or, the first attempt hangs for 40 seconds while the second completes normally in 2 seconds, making the "latency" metric suggest model degradation when it was merely network jitter.

Minimal fix: Set a hard timeout of 45 seconds (30 seconds for online services), limit retries to 1, and only retry on HTTP 5xx errors or network disconnections. HTTP 429 indicates rate limiting; retrying will cause buildup, so you should mark it as a failure immediately and trigger an alert. Mark failures as failures—do not retry silently. Silent retries are the biggest killer of reproducibility because they mask environmental issues as model behavior.

Finally, pin down data order

A common accident in evaluation scripts is using `for item in dict.items()` or iterating over a set. While Python 3.7+ dictionaries maintain insertion order, sets never do. Lists maintain order, but if their source is any arbitrary order other than `sorted(key=str)`, two runs may yield different sequences. Even more subtly: if your evaluation data comes from JSON returned by `requests`, the parsing order of objects is determined by the JSON library. Most preserve order, but not all.

Minimal fix: Load evaluation data into a list once, add an explicit `assert len(set(id(x) for x in items)) == len(items)` to prevent duplicates, and always iterate by the same index. If you use `--shuffle`, pin its seed as well; otherwise, the distribution will differ each run, creating variance so large that you can’t distinguish whether issues stem from the model or from sampling.

A Minimal Template

import json, random

def run_eval(items, model_name, seed):
    random.seed(seed)
    out = []
    for i, item in enumerate(items):          # list, fixed order
        resp = call_llm(model_name, item, temperature=0, top_p=1)
        out.append({
            "idx": i,
            "input": item,
            "response_model": resp.model,     # exact version, not model_name
            "tokens": resp.usage.completion_tokens,
            "latency_ms": resp.latency_ms,
        })
    return out

Include `response_model` and `seed` in every evaluation report. Three months later, when rerunning the same set of prompts, these two lines are your only evidence to determine whether a "silent upgrade" occurred.

Conclusion

Models are upgrading, providers are iterating, and data is drifting. The only things you can control are these four configuration items: the effective model version, sampling parameters, timeout strategy, and data order. Pin them all down, and reproducibility transforms from wishful thinking into a standard process. "Sporadic issues" cease to be mysticism and become documented investigations.