Structured Output in AI Systems: Why Your AI Often Outputs Malformed Formats? Analyzing the Engineering Principles of Constrained Decoding and JSON Mode

When you specify `response_format: { "type": "json_object" }` for every LLM API call, does your model truly guarantee valid JSON output? The answer is no—at least not without Constrained Decoding.

The Core Problem: Token Probabilities Don't Care About Format

At its core, an LLM is a next-token predictor. Each token it outputs is sampled based on probability, with no built-in mechanism to ensure that the number of closing braces equals the number of opening braces, nor any guarantee that quotes appear in pairs.

When you ask a model to "output JSON," you are essentially betting that its training data contains enough JSON examples to make it probabilistically inclined to generate valid JSON. But probability is not a guarantee. A typical failure case: when outputting deeply nested objects, the model might truncate midway, leaving unclosed JSON and causing downstream parsers to crash immediately.

This "probabilistic compliance" is disastrous in production environments. A malformed JSON means the entire processing pipeline breaks—log parsing fails, database writes throw exceptions, and front-end rendering results in blank screens.

Constrained Decoding: From Probability to Guarantee

The core idea behind Constrained Decoding is simple: **during the token generation phase, only allow the model to sample tokens that maintain the validity of the target format.**

The specific implementation involves two steps:

1. **Construct a Finite State Machine (FSM) for format constraints.** For JSON, this means maintaining a stack to track whether the current context is an object, array, string, or number, and calculating the count of unclosed brackets and quotes. At each step, the FSM determines "which tokens are currently valid."

2. **Filter logits during sampling.** After the model outputs the logits vector, set the probabilities of all tokens not in the valid set to negative infinity (or extremely low values), then sample from the filtered distribution.

Taking JSON as an example: if the current context has just output `{"name": "`, the FSM knows that only regular string characters or the closing quote `"` are acceptable next, while `{` or `[` would be disallowed. This filtering is executed for every generated token, ensuring that from the first token to the last, the output remains valid JSON.

Engineering Differences in JSON Mode Implementations

Different inference frameworks vary significantly in how deeply they implement "JSON Mode":

| Framework | Implementation Method | Guarantee Level |

|------|----------|----------|

| OpenAI API | Prompt-level soft constraint | Probabilistic compliance (no guarantee) |

| vLLM + Outlines | Regex/BNF hard constraint | Syntactic guarantee |

| Llama.cpp + Grammar | GBNF grammar constraint | Syntactic guarantee |

| Guidance | Interleaved templates + Token masking | Syntactic guarantee |

OpenAI's JSON Mode essentially just injects the `json_object` prompt into the system message and lowers the sampling probability for non-JSON tokens—it does not perform true constrained decoding. This is why, even with JSON Mode specified, you may still occasionally receive malformed responses.

vLLM integrates the Outlines library, which compiles JSON Schema into a finite state machine to achieve true hard constraints. Llama.cpp's GBNF (Grammar-Based Neural Formatting) adopts a similar approach but uses a custom grammar description language.

Performance Cost: How Slow Is Constrained Decoding?

Constrained decoding is not free. Running state machine transitions for every token generation introduces significant scheduling overhead during batch decoding.

Benchmark data (based on LLaMA-3-70B, A100 80GB):

- **Unconstrained**: Baseline, approx. 35 tokens/s

- **JSON Constraint (Outlines/vLLM)**: Approx. 28 tokens/s, ~20% decrease

- **Complex Nested JSON Constraint**: Approx. 22 tokens/s, ~37% decrease

The primary source of performance degradation is not the state machine itself (which is lightweight), but rather **logits filtering during the sampling phase**. When the set of valid tokens is very sparse (e.g., only a few tokens are valid in JSON), the sampler requires more time to resample from the filtered distribution.

Engineering Recommendations

1. **Use hard constrained decoding in production.** Do not rely on prompt-level JSON Mode; at minimum, use vLLM + Outlines or Llama.cpp + GBNF. For API calls, consider adding a layer of format validation and repair on the backend.

2. **Simpler schemas lead to faster constrained decoding.** Strive for flat JSON structures and avoid deep nesting. A JSON object with a depth of 1 is approximately 15% faster than one with a depth of 4.

3. **Applicable beyond JSON.** Constrained decoding is not limited to JSON—you can constrain models to output specific regular expressions, CSV lines, SQL statements, or custom DSLs. Any format describable by a formal grammar can be constrained.

4. **Balance constraints with generation quality.** Overly strict constraints may prevent the model from expressing itself in edge cases, resulting in truncated output or repeated tokens. It is advisable to include appropriate fallback branches in your constraint definitions (e.g., allowing `null` or empty strings as safeguards).

Structured output is not a problem that can be solved by simply "adding a parameter." Understanding the principles of constrained decoding is essential for making the right engineering trade-offs in production.