Constrained Decoding: Guaranteeing Valid JSON and Grammars

If you have ever parsed a model's JSON in production, you have hit it: 999 calls return perfect objects, and the thousandth returns a trailing comma, a stray sentence of explanation, or a quote in the wrong place — and your pipeline throws. Prompting for structure raises the odds of valid output, but it never guarantees them, and at scale the rare failure is a constant one.

Constrained decoding removes the gamble. Instead of asking the model to produce valid JSON and hoping, it makes invalid JSON impossible to produce in the first place — by controlling, token by token, which tokens the model is even allowed to pick. The output is guaranteed to match your schema or grammar, every single time.

Why prompting alone cannot guarantee structure

A model generates one token at a time by sampling from a probability distribution over the whole vocabulary. Nothing stops it from sampling a token that breaks your format — a closing brace too early, a word where a number belongs. Prompts and examples shift the probabilities toward valid output, but every token still has some chance of going off the rails.

• Sampling is probabilistic. As long as an invalid token has nonzero probability, it will eventually be chosen — and at scale, eventually is often.
• Errors compound. One wrong token early (an extra brace) can derail the entire rest of the structure.
• Retries are a band-aid. Re-prompting on a parse failure costs latency and money and still offers no guarantee on the next try.
• Schemas in the prompt are advice, not enforcement. The model can read your JSON schema and still ignore it under the wrong sample.

Masking the tokens that would break the rules

Constrained decoding works at the layer below prompting. At each step, before the model samples, it computes which tokens are allowed given what has been generated so far — and sets the probability of every disallowed token to zero. If the grammar says the next character must be a digit or a closing brace, every other token is masked out, and the model can only choose from the valid ones.

You describe the target with a JSON schema, a regular expression, or a formal grammar, and the decoder compiles it into these per-step rules. Libraries like Outlines, llguidance, and XGrammar do this efficiently enough to add almost no overhead, and most serving engines now expose it directly as a structured-output mode.

What it looks like

In practice you hand the decoder a schema and get back output that is guaranteed to fit it — no parsing defensively, no retries:

# Constrained decoding: only tokens that keep the output valid are ever sampled.
from outlines import models, generate
from pydantic import BaseModel

class Person(BaseModel):
    name: str
    age: int

model = models.transformers("meta-llama/Llama-3.1-8B")
generator = generate.json(model, Person)        # compile the schema into token rules
result = generator("Extract a person from: Dhileep is 30.")
# result is guaranteed to parse as Person — name is a string, age is an int — every time

The same idea extends past JSON: a regex forces a date or a phone number into shape, and a full grammar can constrain output to valid SQL, a specific DSL, or a fixed set of choices. Anything you can write as a grammar, the decoder can enforce.

Prompting asks the model to behave; constrained decoding removes its ability to misbehave. One is a request, the other is a guarantee — and only one survives the millionth call.

The trade-offs to know

• It guarantees structure, not correctness. The output will be valid JSON; whether the values are right is still on the model and your prompt.
• Over-constraining can hurt quality. Forcing a rigid format too early can box the model out of its best answer — leave room for reasoning before the structured part.
• Grammar compilation has a cost. Complex grammars take time to compile; cache them rather than rebuilding per request.
• You usually do not implement it. Reach for Outlines, XGrammar, or your serving engine's structured-output mode instead of writing token masks by hand.
• It needs logit access. Hosted APIs expose it only through their structured-output features; full grammar control is easiest when you run the model yourself.

The bottom line

If anything downstream parses your model's output, constrained decoding turns a reliability problem into a non-issue. Describe the shape you need as a schema or grammar, let the decoder mask away every token that would break it, and you get structurally valid output by construction — no defensive parsing, no retry loops, no 3 a. m. page over a stray brace.

Prompting for format is a probability play; constraining the decoder is a certainty. For the parts of your system that must not break, stop asking the model to stay in the lines and simply remove its ability to leave them.