Context Engineering Explained: What Goes Into an LLM's Context

At a big hospital in Indore, a night-duty doctor takes over 30 patients at 8 pm. Nobody hands her the full file of every patient, which would run to thousands of pages. The day doctor gives a handover sheet instead: for each patient, the diagnosis, today's key readings, allergies in red, medicines due tonight and what to watch for.

A good handover is short, current and ordered by importance. A bad one leaves out the penicillin allergy or buries it on page 14, and a skilled doctor makes a bad decision.

An LLM is that skilled doctor, and the context window is the handover sheet.

What is context engineering?

Context engineering is the practice of building the system that selects, formats and orders all the information an LLM receives on each call, so that the model has exactly what it needs for the current step and little else.

Mapping the hospital: the full patient file is all the data you could send, from documents and chat history to databases. The handover sheet is the context window. The allergy in red is a guardrail that must never be dropped. The day doctor writing the sheet is your context pipeline, the code that runs before every model call.

Prompt engineering asks "how should I word this?" Context engineering asks "what should the model see right now?" With retrieval, memory or tools, the second question matters more.

The prompt engineering guide covers the first column. This tutorial covers the second.

What goes into an LLM's context window?

A chat model remembers nothing between calls; memory is your code re-sending information each time. A typical agent call assembles six kinds of context:

1. Instructions: the system prompt with role, rules and output format.
2. Examples: a few demonstrations of good answers.
3. Knowledge: passages retrieved from documents or a database, as in RAG.
4. Memory: a summary of earlier turns and stored facts about the user.
5. Tools: names, descriptions and argument schemas of functions the model may call.
6. Tool results: what those functions returned on earlier steps.

The context window is a fixed-size box. The assembler decides what earns a place in it on each call.

All six compete for a fixed token budget. Even with windows of hundreds of thousands of tokens, more is not free: every token costs money and latency, and models tend to use information at the start and end of a long context better than information in the middle, an effect often called lost in the middle.

The four moves: write, select, compress, isolate

Almost every context technique is one of four moves.

Write and isolate keep things out of the window; select and compress decide what goes in.

Write. Save information outside the window: a to-do list, a notes file, user preferences in a database, or state such as {"order_id": 4417, "refund_approved": false}.

Select. Pull in only what the current step needs: the top 3 relevant passages rather than the whole PDF, the two tools that fit this request rather than all forty. Retrieval quality is the bottleneck here, which is why chunking strategies matter so much.

Compress. Shrink what you keep: summarise old turns, cut a 3,000-token API response down to the five fields that matter, remove duplicate passages.

Isolate. Split work so that each model call has a clean context. A research sub-agent reads twenty web pages and returns a one-paragraph finding; the main agent never sees the raw pages.

Packing a context under a token budget

Here is a small context assembler. Each piece has a priority, and the assembler packs the most important pieces first until the budget runs out. The system rules and the current question are marked as required, so they can never be dropped:

def tokens(text):
    return max(1, round(len(text.split()) * 1.3))     # rough rule of thumb for English

pieces = [
    {"name": "system rules",   "priority": 100, "required": True,
     "text": "You are the support bot for an Indore coaching institute. Never promise refunds. " * 3},
    {"name": "user question",  "priority": 100, "required": True,
     "text": "Can I shift from the weekday batch to the weekend batch next month?"},
    {"name": "policy: batch transfers", "priority": 80, "required": False,
     "text": "Batch transfers are allowed once, with 7 days notice, subject to seat availability. " * 4},
    {"name": "policy: fee refunds", "priority": 40, "required": False,
     "text": "Refund rules depend on the enrolment date and course stage. " * 6},
    {"name": "chat summary",   "priority": 60, "required": False,
     "text": "Student joined weekday batch in July, attends 3 days a week, asked about timings. " * 2},
    {"name": "old small talk", "priority": 5,  "required": False,
     "text": "Thanks! Have a nice day. Okay great. " * 10},
]

def assemble(pieces, budget):
    chosen, used = [], 0
    for p in sorted(pieces, key=lambda p: (not p["required"], -p["priority"])):
        cost = tokens(p["text"])
        if p["required"] or used + cost <= budget:
            chosen.append(p["name"]); used += cost
    return chosen, used

for budget in (120, 200, 320):
    names, used = assemble(pieces, budget)
    print(f"budget {budget}: {used} tokens ->", ", ".join(names))

At 120 tokens the transfer policy, the piece this question most needs, does not fit, so the smaller chat summary takes the space. That is a real weakness of greedy packing: trim an important piece rather than drop it.

The irrelevant refund policy only gets in when there is room to spare. Real systems count with the model's own tokenizer and set priorities from retrieval scores.

Compressing chat history and tool results

Long conversations are the most common way contexts overflow. A simple, robust pattern is a rolling summary: keep the last few turns word for word and fold older turns into a running note of facts.

history = [
    ("user", "Hi, I am Riya, final-year BCom from Indore."),
    ("assistant", "Welcome Riya! How can I help?"),
    ("user", "I prefer Hindi explanations and weekend classes."),
    ("assistant", "Noted. Weekend batches are available."),
    ("user", "My budget is limited, I need to know about EMI."),
    ("assistant", "I will share EMI details from the office."),
    ("user", "Also, do you teach SQL?"),
    ("assistant", "Yes, SQL is part of the data courses."),
    ("user", "Okay, and which batch starts first?"),
]
FACT_HINTS = ("i am", "i prefer", "my budget", "i need")

def compress(history, keep_last=3):
    old, recent = history[:-keep_last], history[-keep_last:]
    facts = [text for role, text in old
             if role == "user" and any(h in text.lower() for h in FACT_HINTS)]
    summary = "Known about user: " + " | ".join(facts)
    return [("system", summary)] + recent

def word_count(msgs):
    return sum(len(t.split()) for _, t in msgs)

compact = compress(history)
for role, text in compact:
    print(f"{role:>9}: {text}")
print(f"\nwords before: {word_count(history)}, after: {word_count(compact)}")

In production an LLM call writes the summary, and the saving grows with every turn. Stable facts survive; chatter disappears.

Tool results deserve the same treatment. An API that returns 80 fields about an order should reach the model as the 4 fields the task needs. A short, clear error such as "seat_unavailable: weekend batch full until 15 Dec" beats a 200-line stack trace every time. Asking tools and sub-agents to return structured JSON makes this trimming easy.

How context goes wrong

Four failure patterns cause most bad answers in context-heavy apps:

• Missing context. The fact the model needed was never retrieved. The model fills the gap with a confident guess.
• Distracting context. Twenty loosely related passages drown the two relevant ones. More retrieval is not better retrieval.
• Conflicting context. An old policy and a new policy both appear, and the model picks the wrong one. Attach dates and sources, and remove superseded documents.
• Poisoned context. An earlier hallucination, or text planted by an attacker, is saved to memory and re-sent on every turn. Planted instructions are the subject of prompt injection and how to defend against it.

Too many tools is a form of distraction: an agent offered 50 tool descriptions picks the wrong one more often. Avoiding tool overload covers loading tools on demand.

Debugging an answer: inspect the context first

When an answer is wrong, follow these steps before you touch the prompt wording:

1. Log the exact context sent on that call, not the template.
2. Check whether the needed fact is present at all.
3. If present, check where it sits and what surrounds it.
4. Look for contradictions or outdated text.
5. Only then change instructions or the model.

Common context engineering mistakes

• Sending everything because the window is big. Large windows raise cost and lower focus.
• Dropping guardrails during trimming. Mark system rules and safety notes as required.
• Raw tool output. Summarise or filter it before it re-enters the context.
• Memory without expiry. Old preferences and stale facts need dates and clean-up.

Key takeaways

• The model sees only what you send on each call; context engineering decides what that is.
• Six kinds of context compete for one token budget: instructions, examples, knowledge, memory, tools and tool results.
• Write, select, compress and isolate are the four core moves.
• Keep required rules pinned, put stable content first and trim tool output hard.
• When answers go wrong, inspect the assembled context before rewording the prompt.