Don’t Let “Hallucinations” Be a Fig Leaf for Delivery: Building a Deterministic Verification Loop in AI Labs

In many AI Lab delivery scenarios, the most anxiety-inducing moment isn’t when the model lacks intelligence, but when it “occasionally” makes mistakes. When clients ask, “Why was the result wrong this time?” many teams tend to mask the issue with a vague term: Hallucination.

However, from an engineering delivery perspective, the term “hallucination” is dangerous. It implies that errors are unpredictable, inevitable, and random. If a deliverable is defined as “occasionally prone to hallucinations,” it becomes commercially unusable.

True AI engineering should wrap “probabilistic generation” within “deterministic verification.”

From “Prompt Optimization” to “Verification Loop”

When facing errors, most teams follow this path: Discover error $\rightarrow$ Modify Prompt $\rightarrow$ Test $\rightarrow$ Discover new error $\rightarrow$ Modify Prompt again. This is a typical dead loop because you are trying to use a probabilistic tool (LLM) to fix another probabilistic outcome.

In actual AI Lab deliveries, we advocate for a Verification Loop. Its core logic is: Do not trust the LLM’s self-evaluation; only trust programmable assertions.

1. Define “Hard Constraint” Assertions

Instead of asking the model, “Is the answer you just generated correct?”, write code to check:
- Format Assertions: Does the output strictly conform to the JSON Schema?
- Citation Assertions: Can every factual point in the answer be traced back to the original text index in the retrieved Context? (By calculating token overlap or using small Cross-Encoder models)
- Logical Assertions: If the answer involves numerical calculations, can it be re-calculated via Python exec() or simple regex extraction?

2. Build a “Negative Sample” Regression Set

Many teams focus only on Positive Cases (did it run successfully?) while ignoring Negative Cases (what constitutes an error?).
A mature delivery project requires maintaining an $\text{Error-Bank}$: recording all historical hallucination cases and converting them into test cases. After every Prompt update, you must ensure that these historical errors do not recur, rather than just checking if a few new examples run smoothly.

Practical Case: Enhancing Determinism in Knowledge Base RAG

In a financial document analysis project, we encountered typical “detail hallucinations”—the model would attribute Company A’s revenue data to Company B.

We abandoned the approach of solving the problem by adding more instructions to the System Prompt (e.g., “Please carefully verify the company name”) and instead introduced a Dual-Path Verification Mechanism:
1. Extraction Path: While generating the answer, the LLM is forced to output a structured list in the format {"fact": "...", "source_chunk_id": "..."}.
2. Verification Path: A lightweight Python script iterates through all source_chunk_ids to check whether the Chunk actually contains the key entities and values mentioned in fact. If verification fails, it directly triggers a retry or marks the result as “Unconfirmed.”

This approach increased accuracy from 85% to 98%. More importantly, it allowed us to give clients a definitive explanation: “This was not closed because the verification path failed to find evidence in the original text,” rather than “The model hallucinated.”

Recommendations for AI Engineering Teams

Delivery in AI Labs is not about writing poetry; it is about building industrial assembly lines. Keep the following three points in mind:
- Beware of Prompt Dependency: When you find yourself tweaking the wording of the same Prompt for three consecutive days, you should consider adding a layer of code-level verification logic.
- Quantify Error Distribution: Don’t say “It feels much better”; say “In a regression set of 500 samples, the hallucination rate dropped from 12% to 3%.”
- Accept “I Don’t Know”: An Agent that dares to admit “Cannot draw a conclusion based on available materials” is far more valuable than one that confidently spouts nonsense.

True alchemy is not about turning lead into gold, but about removing all impurities through rigorous filtering and purification.