Structured Output in LLMs: JSON Prompting and Schema Guide

Think of the railway reservation form at Indore Junction. It has fixed boxes: train number, date, from, to, passenger name, age, berth preference. The clerk can type it into the system in seconds because every form looks the same.

Now imagine passengers wrote letters instead: "Dear Sir, my mother and I wish to travel to Delhi sometime next Friday, preferably by the night train, lower berth if possible." Every letter is polite and informative, and every one needs a person to read it and type it in.

LLMs write letters by default. Structured output is how you make them fill in the form.

What is structured output in LLMs?

Structured output is a reply from a language model that follows a predefined, machine-readable format, most often a JSON object matching a JSON Schema, so that software can parse and use it directly.

In the analogy, the printed form is your schema, each box is a field with a type, the berth preference tick-boxes are an enum (a fixed list of allowed values), and the clerk's software is your application code that reads the result.

The prompt engineering guide showed how to ask for JSON and pull it out of a chatty reply. This tutorial covers what comes next: guaranteeing the shape, designing good schemas, and catching answers that are well formed but wrong. It also powers the reasoning checklists from prompting techniques for reasoning.

Three levels of structured output

Not every method of getting JSON is equally reliable. Think of it as a ladder:

Each rung removes a class of failure. None of them checks whether the values are true.

Schema-constrained output works through constrained decoding: at each step the server masks out tokens that would break the schema, so invalid output cannot be generated. The mechanism is explained in LLM decoding strategies. OpenAI calls this Structured Outputs with strict mode, and many open-model servers, including Ollama and vLLM, accept a JSON Schema too.

Designing a JSON schema an LLM can fill well

A schema is also a prompt. Field names, descriptions and order all influence what the model writes. Here is a schema for extracting a shop bill:

{
  "type": "object",
  "properties": {
    "shop_name":   {"type": "string"},
    "bill_date":   {"type": ["string", "null"], "description": "YYYY-MM-DD, or null if not printed"},
    "items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name":       {"type": "string"},
          "qty":        {"type": "integer"},
          "unit_price": {"type": "number"}
        },
        "required": ["name", "qty", "unit_price"],
        "additionalProperties": false
      }
    },
    "payment_mode": {"type": "string", "enum": ["cash", "upi", "card", "unknown"]},
    "total":        {"type": "number"}
  },
  "required": ["shop_name", "bill_date", "items", "payment_mode", "total"],
  "additionalProperties": false
}

Rules that make schemas work:

• Use enums for categories. "enum": ["cash", "upi", "card", "unknown"] stops the model inventing "PhonePe" or "online".
• Give "not found" a home. A nullable field or an "unknown" option prevents invented dates. Strict modes usually require every field to be listed as required, so optional means "nullable", not "missing".
• Write descriptions for anything ambiguous. Date formats, units and currency belong in the field description.
• Keep nesting shallow. Two levels are fine; five levels of nested objects invite mistakes.
• Put reasoning before the answer. Models write fields in order. If a decision needs thought, add a short "reasoning" field before "category", never after it.

Validating against the schema in code

Even with a strict provider, check the shape yourself. You may switch providers, the reply may be cut off by a token limit, or the model may refuse. Here is a small validator for the parts of JSON Schema used above, written in plain Python so you can see what a library like jsonschema does:

import json

SCHEMA = {"type": "object", "required": ["shop_name", "items", "payment_mode", "total"],
          "properties": {"shop_name": {"type": "string"},
                         "items": {"type": "array"},
                         "payment_mode": {"type": "string", "enum": ["cash", "upi", "card", "unknown"]},
                         "total": {"type": "number"}}}
PY_TYPES = {"string": str, "array": list, "number": (int, float), "object": dict}

def validate(data, schema):
    errors = [f"missing field '{k}'" for k in schema["required"] if k not in data]
    errors += [f"unexpected field '{k}'" for k in data if k not in schema["properties"]]
    for key, rule in schema["properties"].items():
        if key in data:
            if not isinstance(data[key], PY_TYPES[rule["type"]]):
                errors.append(f"'{key}' should be {rule['type']}")
            elif "enum" in rule and data[key] not in rule["enum"]:
                errors.append(f"'{key}' must be one of {rule['enum']}")
    return errors

replies = [
    '{"shop_name": "Sarafa Sweets", "items": [], "payment_mode": "upi", "total": 240}',
    '{"shop_name": "Sarafa Sweets", "items": [], "payment_mode": "PhonePe", "total": "240"}',
    '{"shop_name": "Sarafa Sweets", "payment_mode": "cash", "total": 240, "gstin": "23AB"}',
]
for r in replies:
    print(validate(json.loads(r), SCHEMA) or "valid")

In real projects use a tested library, such as Pydantic or jsonschema, but the logic is exactly this.

Structured output with the OpenAI SDK and Pydantic

With the official openai Python SDK you can describe the schema as a Pydantic model. The SDK converts it to a strict JSON Schema, sends it, and parses the reply back into a typed Python object. This uses the Responses API:

from typing import Literal, Optional
from openai import OpenAI
from pydantic import BaseModel

class Item(BaseModel):
    name: str
    qty: int
    unit_price: float

class Bill(BaseModel):
    shop_name: str
    bill_date: Optional[str]                     # nullable: None when not printed
    items: list[Item]
    payment_mode: Literal["cash", "upi", "card", "unknown"]
    total: float

client = OpenAI()                                # reads OPENAI_API_KEY from the environment
response = client.responses.parse(
    model="your-model-name",                     # a current model that supports Structured Outputs
    input=[
        {"role": "system", "content": "Extract the bill. Use null for anything not printed."},
        {"role": "user", "content": "SARAFA SWEETS  2 x Poha 30  1 x Jalebi 180  Paid by UPI  Total 240"},
    ],
    text_format=Bill,
)
bill = response.output_parsed                    # a Bill instance, or None if the model refused
print(bill)

Handle two edge cases. If the model refuses, the reply contains a refusal instead of parsed data. If the response hits the output token limit, the JSON can be incomplete. Check both before using the result.

Valid JSON is not correct JSON

Schema enforcement guarantees that total is a number. It cannot guarantee it is the right number. Models misread digits, double-count items and invent plausible values. A second layer of checks, called semantic or business-rule validation, catches these.

Two gates, one retry with feedback, then a safe fallback. Never loop forever.

The checks depend on your domain: line items should add up to the total, dates should not be in the future, a PIN code in Indore should start with 452. When a check fails, the most effective repair is to send the error back:

import json

model_replies = iter([   # what a model might return on attempt 1 and attempt 2
    '{"items": [{"name": "Poha", "qty": 2, "unit_price": 30}, {"name": "Jalebi", "qty": 1, "unit_price": 180}], "total": 210}',
    '{"items": [{"name": "Poha", "qty": 2, "unit_price": 30}, {"name": "Jalebi", "qty": 1, "unit_price": 180}], "total": 240}',
])

def business_errors(bill):
    errors = []
    computed = sum(i["qty"] * i["unit_price"] for i in bill["items"])
    if abs(computed - bill["total"]) > 1:
        errors.append(f"items add up to {computed} but total is {bill['total']}")
    if any(i["qty"] <= 0 for i in bill["items"]):
        errors.append("every qty must be positive")
    return errors

def extract_with_retry(max_attempts=2):
    feedback = ""
    for attempt in range(1, max_attempts + 1):
        prompt = "Extract the bill as JSON." + feedback      # would be sent to the model
        bill = json.loads(next(model_replies))
        errors = business_errors(bill)
        print(f"attempt {attempt}: {errors or 'passed'}")
        if not errors:
            return bill
        feedback = "\nYour previous answer was wrong: " + "; ".join(errors) + ". Re-read the bill."
    return None                                              # fallback: send to human review

print("final total:", extract_with_retry()["total"])

Specific feedback ("items add up to 240 but total is 210") fixes far more errors than a generic "try again". Cap retries at one or two, and log every failure, because they show you where your prompt or schema is weak. Deciding what text the extractor sees in the first place is the job of context engineering.

JSON prompts and tool calls

Structure helps on the input side as well. Writing a complex request as JSON, with keys like "task", "audience", "constraints" and "output_format", forces you to fill every slot and makes prompts easy to version and compare. It is optional, but useful when many people edit the same prompts.

Function calling, or tool calling, is structured output with a purpose: the model fills in a JSON object of arguments for a function your code will run. Everything in this tutorial, enums, nullable fields, validation and retries, applies there too. Tools and function calling for agents builds on it.

Common mistakes with structured output

• Trusting the schema for correctness. A valid total can still be wrong. Add business rules.
• No "unknown" option. Without a nullable field or an "unknown" value, the model must invent one.
• Reasoning after the answer. A "reasoning" field placed after "category" cannot influence it.
• Huge, deeply nested schemas. Split a 60-field extraction into two or three smaller calls.
• Infinite retry loops. One or two retries, then a fallback.
• Ignoring truncation. Long arrays can hit the token limit; set the limit generously and check for incomplete replies.

Key takeaways

• Structured output turns LLM replies into data your code can use without guessing.
• Prompt-only JSON, JSON mode and schema-constrained output are increasingly reliable rungs.
• Good schemas use enums, nullable fields, clear descriptions, shallow nesting and reasoning-first field order.
• Validate twice: the schema for shape, business rules for meaning.
• Retry once or twice with the specific error, then fall back safely.