KV Cache Explained: How LLMs Reuse Attention and Use GPU Memory

Picture a clerk at the Chappan Express office writing a reply letter one word at a time. Before every new word, a careless clerk rereads the entire letter from the top and re-summarises each sentence in his head. By the hundredth word, he spends almost all his time rereading.

A smart clerk keeps a notepad. After reading each word once, he jots a short note about it. For the next word, he only glances at his notes and adds one more line.

The notepad is the KV cache. It makes writing fast, but the notepad fills up, and with many letters in progress at once, the desk runs out of space.

What is the KV cache?

The KV cache is a store, kept in GPU memory during generation, of the key and value vectors that each attention layer has computed for every previous token, so they can be reused rather than recomputed when the model generates the next token.

Recall from transformers and attention that each token produces three vectors in every attention layer: a query, a key and a value. The new token's query is compared with the keys of all earlier tokens, and the result weights their values.

The keys and values of earlier tokens never change, because a decoder model's attention is causal: token 10 cannot look at token 11. So once computed, they can be kept. Only the newest token's query, key and value are fresh at each step.

This lesson continues the LLMOps series. Chappan Express now self-hosts the model behind Chappan Help on one GPU, so memory has become its problem.

Why do LLMs need a KV cache?

An LLM generates text one token at a time, feeding each output back as input, as covered in decoding strategies. Without a cache, producing token number t means running all t − 1 earlier tokens through every layer again.

Without a cache, step 5 redoes the work of steps 1 to 4. With a cache, every step does one token's worth of new work.

For a 500-token reply, recomputing means roughly 125,000 token-passes through the model instead of 500. That is the difference between a reply in seconds and a reply in minutes.

See the speed-up in numpy

The block below runs one toy attention layer two ways and checks that both give identical outputs.

import time
import numpy as np

rng = np.random.default_rng(0)
d, steps = 128, 300
Wq, Wk, Wv = (rng.normal(0, d ** -0.5, (d, d)) for _ in range(3))
X = rng.normal(0, 1, (steps, d))                 # embeddings of tokens as they arrive

def attend(q, K, V):
    w = np.exp(q @ K.T / np.sqrt(d)); w /= w.sum()
    return w @ V

t0 = time.perf_counter(); out_nocache = []
for t in range(1, steps + 1):                    # recompute K, V for all tokens so far
    K, V = X[:t] @ Wk, X[:t] @ Wv
    out_nocache.append(attend(X[t - 1] @ Wq, K, V))
t_nocache = time.perf_counter() - t0

t0 = time.perf_counter(); Kc, Vc, out_cache = [], [], []
for t in range(1, steps + 1):                    # compute only the newest token's K, V
    Kc.append(X[t - 1] @ Wk); Vc.append(X[t - 1] @ Wv)
    out_cache.append(attend(X[t - 1] @ Wq, np.array(Kc), np.array(Vc)))
t_cache = time.perf_counter() - t0

print("same outputs:", np.allclose(out_nocache, out_cache))
print(f"projection rows computed: {steps * (steps + 1) // 2:,} without cache vs {steps:,} with")
print(f"time: {t_nocache * 1000:.0f} ms without cache, {t_cache * 1000:.0f} ms with cache")

The outputs match exactly, so the cache is a pure optimisation, not an approximation. The count of projections falls from about 45,000 to 300. A real model repeats this in dozens of layers, so the saving is enormous.

The measured time gap in this toy is smaller, because the cached loop still converts the growing list to an array each step and numpy overheads dominate tiny matrices.

Prefill and decode: two phases of generation

With a cache, a request has two distinct phases.

1. Prefill. The whole prompt (system prompt, retrieved help articles, chat history and the new question) goes through the model in one parallel pass. Its keys and values fill the cache. This phase decides the time to first token.
2. Decode. The model generates one token per step, each time reading the entire cache and appending one new entry. This phase decides the time per output token.

Prefill is limited by raw compute, because thousands of tokens are processed at once. Decode is limited by memory bandwidth, because each step reads the full model weights and the full cache to produce a single token. That split drives almost every trick in LLM inference optimisation.

How much memory does the KV cache use?

Each layer stores one key and one value vector per KV head per token. Multiply it out:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value

Then multiply by the tokens in each conversation and the number of conversations running at once. Take a typical 8-billion-parameter open model shape: 32 layers, 8 KV heads, head dimension 128, stored in 16-bit precision (2 bytes).

GiB = 1024 ** 3

def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_value):
    return 2 * layers * kv_heads * head_dim * bytes_per_value

configs = {
    "multi-head (32 KV heads), fp16": (32, 32, 128, 2),
    "grouped-query (8 KV heads), fp16": (32, 8, 128, 2),
    "grouped-query (8 KV heads), fp8": (32, 8, 128, 1),
    "multi-query (1 KV head), fp16": (32, 1, 128, 2),
}
chat_tokens, gpu_gib = 4000, 80
weights_gib = 8e9 * 2 / GiB                       # 8B parameters in 16-bit
free_for_kv = gpu_gib * 0.90 - weights_gib - 4    # leave room for activations

for name, cfg in configs.items():
    per_tok = kv_bytes_per_token(*cfg)
    per_chat = per_tok * chat_tokens / GiB
    print(f"{name:<34} {per_tok / 1024:>5.0f} KiB/token  {per_chat:5.2f} GiB per 4k chat"
          f"  fits {int(free_for_kv // per_chat):>4} chats")

With full multi-head attention, one 4,000-token Chappan Help chat would hold 2 GiB of cache and the GPU would fit only about 27 at once. The grouped-query shape that most current open models use cuts that by four. Storing the cache in 8-bit halves it again.

Weights are a fixed cost. The KV cache budget is what decides how many users you can serve together.

Ways to shrink or reuse the KV cache

Prefix caching is especially useful for Chappan Help. Every conversation starts with the same 1,200-token system prompt and policy text. Cache it once and every new chat skips that part of prefill. The same idea, taken further, is cache-augmented generation, explained in RAG vs CAG.

Paging, which removes wasted gaps between conversations in memory, is the core idea behind vLLM and is covered in serving LLMs with vLLM.

Common misunderstandings about the KV cache

• "The KV cache changes the answer." It does not. Cached and uncached generation give the same outputs, as the numpy check shows.
• "It caches answers between users." No. It caches internal vectors for tokens. Only prefix caching shares them, and only for identical starting text.
• "Longer context is free if the model supports it." Every token of context costs cache memory for as long as the request runs.
• "Bigger batch is always faster." More conversations at once raise throughput until the cache fills, then requests queue or get evicted.
• "Quantising weights also shrinks the cache." Weight and cache precision are separate settings. See open-source LLM quantisation for weights.

Key takeaways

• The KV cache stores earlier tokens' keys and values so each generation step only computes the newest token.
• It gives identical outputs and turns quadratic recomputation into linear work.
• Memory per token is 2 × layers × KV heads × head dimension × bytes; multiply by tokens and active requests.
• Prefill is compute-bound and sets time to first token; decode is bandwidth-bound and sets time per token.
• GQA, 8-bit caches, paging and prefix caching are the main levers. Next: LLM inference optimisation.