Transformers and Attention Explained: How LLMs Read Context

Picture a pharmacist at a busy medical store in Palasia reading a handwritten prescription. Line four says "Take twice daily after food." Twice daily what? Her eyes jump back up the page to the medicine name on line two, and line four suddenly makes sense.

She asks one specific question ("which medicine?"), finds the line that answers it, and pulls in exactly that detail. That habit of looking back, choosing what is relevant and borrowing its meaning is the core idea of the transformer.

What is a transformer?

Mapping the pharmacist to the model: each token in the prompt is a line on the prescription. The question she asks is a query. The label each line offers ("I am a medicine name") is a key. The information she copies across is the value. Doing this for every line at once is self-attention.

A transformer is a neural network architecture that processes a sequence of tokens with stacked layers of self-attention, which lets each token gather information from the other tokens, followed by a feed-forward network that transforms each token's representation.

The design comes from the 2017 paper "Attention Is All You Need". The original had an encoder and a decoder for translation. Most chat LLMs today are decoder-only transformers: a single stack that reads the prompt and predicts the next token, as described in what is an LLM.

The stack's input is the token IDs from the tokenizer, turned into vectors by an embedding table.

How self-attention works: queries, keys and values

For every token, the model makes three vectors by multiplying the token's current vector with three learned weight matrices:

• Query (Q): what this token is looking for.
• Key (K): what this token offers to others.
• Value (V): the information it hands over if chosen.

Attention for one token: compare its query with every key, turn the scores into weights, then take a weighted average of the values.

The steps for one token are:

1. Take the dot product of its query with every key. A large dot product means "this key matches what I am looking for".
2. Divide by the square root of the key size, so scores do not grow too large as vectors get longer.
3. Apply softmax so the scores become weights that add up to 1.
4. Multiply each value by its weight and add them up. The result replaces the token's vector.

Here it is with tiny, hand-picked vectors so you can see the "it" token finding "bus":

import numpy as np

tokens = ["the", "bus", "was", "full", "so", "it"]
# toy 2-number keys: first number ~ "is a vehicle", second ~ "is a state or verb"
K = np.array([[0.0, 0.1], [1.0, 0.2], [0.1, 1.0], [0.3, 0.9], [0.0, 0.3], [0.5, 0.1]])
V = np.array([[0, 0], [9, 1], [1, 5], [2, 8], [0, 1], [3, 0]], dtype=float)
q_it = np.array([2.0, 0.4])            # "it" is looking for a noun to refer to

scores = K @ q_it / np.sqrt(2)         # dot product with every key, then scale
weights = np.exp(scores) / np.exp(scores).sum()   # softmax: weights add up to 1

for t, s, w in zip(tokens, scores, weights):
    print(f"{t:<5} score={s:5.2f}  weight={w:.2f}  " + "#" * int(w * 40))
print("new vector for 'it':", (weights @ V).round(2))

"bus" gets the biggest weight because its key lines up best with the query of "it". The new vector for "it" now carries much of the value of "bus". In a real model nobody hand-picks these numbers; the matrices that produce Q, K and V are learned in training.

Scaled dot-product attention and the causal mask

In practice the model does this for all tokens at once with matrix multiplication. The whole operation fits in one formula: softmax(Q·Kᵀ / √d_k) · V.

A decoder-only LLM adds one rule. When it is learning to predict token 5, it must not see token 5 or anything after it, or it could simply copy the answer. A causal mask sets every score above the diagonal to minus infinity, so those weights become exactly zero after softmax.

import numpy as np

rng = np.random.default_rng(0)
tokens = ["chai", "is", "too", "sweet", "today"]
n, d_model, d_k = len(tokens), 8, 4

X = rng.normal(0, 0.5, (n, d_model))                # one embedding row per token
Wq, Wk, Wv = (rng.normal(size=(d_model, d_k)) for _ in range(3))
Q, K, V = X @ Wq, X @ Wk, X @ Wv                   # learned projections

scores = Q @ K.T / np.sqrt(d_k)                    # (n, n) table of scores
mask = np.triu(np.ones((n, n), dtype=bool), k=1)   # True above the diagonal
scores[mask] = -np.inf                             # no peeking at future tokens

weights = np.exp(scores - scores.max(axis=1, keepdims=True))
weights /= weights.sum(axis=1, keepdims=True)
out = weights @ V

print("attention weights (row = token doing the looking):")
print("        " + "".join(f"{t:>7}" for t in tokens))
for t, row in zip(tokens, weights):
    print(f"{t:>7} " + "".join(f"{w:7.2f}" for w in row))
print("output shape:", out.shape, "| each row sums to", weights.sum(axis=1).round(2))

Read the table row by row: "chai" sees only itself, "is" sees "chai" and itself, and so on. Every zero above the diagonal is the mask at work.

Multi-head attention: several questions at once

One set of Q, K and V can only ask one kind of question. A real layer runs many attention heads in parallel, often 16 to 128, each with its own smaller weight matrices.

One head might track which noun a pronoun refers to, another might match an opening bracket with its closing one. The outputs of all heads are joined and mixed by one more weight matrix.

Inside one transformer block

Attention is only half of a block. The full decoder block, repeated dozens of times, looks like this:

A decoder-only LLM is the same block stacked N times, with an embedding layer at the bottom and a next-token output layer at the top.

• Layer normalisation keeps the numbers in a stable range before each sub-layer. Many recent models use a lighter variant called RMSNorm.
• Masked multi-head self-attention mixes information between tokens, as above.
• Residual connections add each sub-layer's input back to its output. This lets very deep stacks train without the signal fading.
• The feed-forward network (FFN) works on each token separately: two or three weight matrices with a non-linear activation in between. A large share of the model's parameters, and much of its stored knowledge, live here. The mixture of experts design replaces this FFN with several smaller "expert" networks.

How a transformer knows word order

Attention alone treats the input as a bag of tokens: "dog bites man" and "man bites dog" would look the same. So the model injects position information. The original paper added fixed sine and cosine patterns; many current LLMs use rotary position embeddings (RoPE), which rotate queries and keys by an angle that depends on position.

At the top of the stack, a final layer turns the last token's vector into one score per vocabulary entry. Softmax over those scores gives the next-token probabilities.

Why transformers replaced RNNs

Before transformers, language models were mostly recurrent neural networks such as LSTMs, covered in the RNN and LSTM tutorial. They read text one token at a time, passing a memory vector along.

Parallel training is the big win: it made training on trillions of tokens practical. The price is the last row of that table.

The cost of attention on long prompts

Every token is scored against every other token, so a prompt of n tokens produces an n × n table of scores, per head, per layer. Doubling the prompt quadruples that work.

heads, layers, bytes_fp16 = 32, 32, 2

print(f"{'tokens':>8} {'scores per head':>16} {'memory, one head':>18}")
for n in [1_000, 8_000, 32_000, 128_000]:
    entries = n * n                                  # every token vs every token
    mb = entries * bytes_fp16 / 1e6
    print(f"{n:>8,} {entries:>16,} {mb:>15,.0f} MB")

n = 32_000
total = n * n * heads * layers * bytes_fp16 / 1e12
print(f"\nAll {heads} heads x {layers} layers at {n:,} tokens: {total:.1f} TB if stored at once")
print("Doubling the prompt length multiplies this work by", (2 * n) ** 2 // n ** 2)

No real system stores all of that at once. Techniques such as FlashAttention compute attention in small tiles so the full table never sits in memory, and during generation the model keeps earlier keys and values in a KV cache instead of recomputing them.

That trick has its own article, KV cache explained. Long prompts still cost more time and money.

Common misunderstandings about transformers

• "Attention means the model understands." Attention is a weighted average driven by learned numbers. Useful behaviour emerges from training, not from the mechanism alone.
• "Attention weights explain the answer." They show where information flowed in one layer, not why the model decided something. Treat attention heatmaps as hints, not proof.
• "More layers is always better." A small model trained well often beats a large one trained badly.

Key takeaways

• A transformer is a stack of blocks, each with masked multi-head self-attention, a feed-forward network, layer norms and residual connections.
• Attention compares a query with keys, turns scores into softmax weights, and averages the values.
• The causal mask limits each token to earlier tokens, which makes next-token training and generation work.
• Position information, often RoPE today, tells the model word order.
• Attention cost grows with the square of prompt length, so long contexts are expensive.

Next in this series, see how LLMs are trained to learn where all those weight matrices get their values. To check your understanding of the maths, try a few timed quizzes on Debugshala Test.