Tokenization Explained: How LLMs Split Text into Tokens

Picture an old letterpress shop near Rajwada that prints wedding cards. The typesetter keeps single letter blocks for spelling anything, but for words he sets every day, like "shubh", "vivah" and "Indore", he has cast whole-word blocks. One block instead of six saves him time on every card.

For a rare surname he falls back to individual letters, or to a few pieces he has cast for common endings like "-wal" and "-kar". He never gets stuck, because single letters can spell anything.

An LLM's tokenizer works the same way. It has a fixed tray of pieces, it uses the biggest ones it can, and it falls back to smaller ones for anything unusual.

What is tokenization in an LLM?

Here is the analogy mapped to the real terms:

Tokenization is the process of splitting text into tokens from a fixed vocabulary and converting each token into an integer ID, so that a language model can process the text as numbers.

The what is an LLM article showed that a model only sees token IDs and predicts the next one. Here you will see how those tokens are chosen in the first place.

Word, character and subword tokenization compared

There are three broad ways to cut text into pieces.

Word-level tokenizers break the moment someone types "Indoreans" or a typo. Character-level ones never break, but a 1,000-word answer becomes 5,000-plus steps, each a full pass through the network.

Subword tokenization is the compromise almost every LLM uses. Frequent words stay as one token, while rare words are built from frequent pieces.

How byte-pair encoding (BPE) works, step by step

Byte-pair encoding is a training algorithm for tokenizers. It starts with single characters and repeatedly glues together the pair of neighbouring symbols that appears most often in a training corpus.

1. Split every word in the corpus into single characters.
2. Count every pair of adjacent symbols across the corpus.
3. Merge the most frequent pair into one new symbol and add it to the vocabulary.
4. Repeat steps 2 and 3 until the vocabulary reaches the size you want.

Each BPE merge turns the most common adjacent pair into one new token, so frequent words end up as a single piece.

The list of merges, in the order they were learned, is the tokenizer. Here is a complete BPE trainer, run on a tiny corpus of chai-stall words.

from collections import Counter

corpus = ("chai chai chai chaiwala chaiwala chaat chaat "
          "poha poha poha pohawala samosa").split()
words = Counter(tuple(w) for w in corpus)        # each word as a tuple of characters

def merge(words, pair):
    out = Counter()
    for sym, freq in words.items():
        s, i = [], 0
        while i < len(sym):
            if sym[i:i + 2] == pair:
                s.append(sym[i] + sym[i + 1]); i += 2
            else:
                s.append(sym[i]); i += 1
        out[tuple(s)] += freq
    return out

for step in range(7):
    pairs = Counter()
    for sym, freq in words.items():
        for a, b in zip(sym, sym[1:]):
            pairs[(a, b)] += freq
    best, count = pairs.most_common(1)[0]
    words = merge(words, best)
    print(f"merge {step + 1}: {best[0]!r} + {best[1]!r}  (seen {count} times)")

print("\nFinal segmentation:")
for sym, freq in words.items():
    print(f"  {' '.join(sym):<16} x{freq}")

Notice what it learns first: h + a, because "ha" appears in chai, chaat and poha alike. Within seven merges, "chai" and "poha" are single tokens, "wal" has become a reusable piece, and the rare word "samosa" is still spelled letter by letter. Real tokenizers run tens of thousands of merges over billions of words.

Encoding new text with the learned merges

To tokenize a new word, start from characters and replay the merges in the order they were learned. The word need not have appeared in training.

merges = [("h", "a"), ("c", "ha"), ("cha", "i"), ("p", "o"),
          ("po", "ha"), ("w", "a"), ("wa", "l")]      # learned in this order

def encode(word):
    symbols = list(word)
    for a, b in merges:                    # replay every merge, in order
        i = 0
        while i < len(symbols) - 1:
            if symbols[i] == a and symbols[i + 1] == b:
                symbols[i:i + 2] = [a + b]
            else:
                i += 1
    return symbols

vocab = {}
for w in ["chai", "chaipoha", "pohawala", "halwa", "xyz"]:
    toks = encode(w)
    ids = [vocab.setdefault(t, len(vocab)) for t in toks]
    print(f"{w:<10} -> {str(toks):<28} ids={ids}")

"chaipoha" and "pohawala" reuse learned pieces, "halwa" gets partial help, and "xyz" falls back to single characters. That fallback is why subword tokenizers never hit an unknown word.

The full tokenization pipeline inside an LLM

BPE merges are only one stage. A production tokenizer runs text through a short pipeline before the model sees anything.

Text becomes IDs in four stages; the embedding lookup that follows is part of the model, not the tokenizer.

• Normalisation cleans the text, for example Unicode normalisation. Some tokenizers lowercase; most LLM tokenizers keep case.
• Pre-tokenisation splits on spaces and punctuation so merges never cross word boundaries. Many tokenizers attach the leading space to the word, so " chai" and "chai" are different tokens.
• The subword model (BPE, WordPiece or Unigram) cuts each piece into tokens.
• ID lookup turns each token into its row number in the vocabulary.

After that, the model looks up a vector for each ID. That step is covered in Embeddings Explained.

Bytes, Hindi and why token counts vary by language

Many modern LLM tokenizers are byte-level: they run BPE on the UTF-8 bytes of the text rather than on characters. With only 256 base symbols, any string in any script, including emoji, can be encoded.

The catch is that non-Latin scripts take more bytes. An English letter is 1 byte in UTF-8, while each Devanagari character is 3 bytes. If the training data had less Hindi, fewer Hindi merges were learned, and Hindi text breaks into more tokens.

samples = {
    "English": "Indore is clean",
    "Hindi":   "इंदौर साफ़ है",
    "Code":    "if x>=10: print(x)",
}
for name, text in samples.items():
    chars = len(text)
    utf8 = len(text.encode("utf-8"))
    print(f"{name:<8} chars={chars:<3} utf-8 bytes={utf8:<3} bytes per char={utf8 / chars:.1f}")

The Hindi line has fewer characters but more bytes. In practice, the same meaning in Hindi can cost two or more times as many tokens as in English, depending on the tokenizer. Newer tokenizers with larger vocabularies have narrowed this gap, so always measure with the model you will actually use.

Special tokens and chat templates

Every vocabulary also reserves special tokens that never come from ordinary text: a beginning-of-sequence marker, an end-of-sequence marker that tells generation to stop, padding, and role markers that separate system, user and assistant turns.

A chat template is the rule that wraps your messages with these markers before tokenizing. If you build the prompt by hand and get the markers wrong, an open model can ramble, ignore the system prompt or never stop.

Using a real tokenizer in Python

The pure-Python versions above teach the idea. For real work, use the tokenizer that ships with your model. The Hugging Face tutorial explains installing transformers; this code needs a network connection to download the tokenizer files once.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")   # any model id on the Hub works

text = "Tokenization decides what an LLM sees."
print(tok.tokenize(text))           # the subword pieces
ids = tok.encode(text)
print(ids, len(ids))                # token IDs and the count you pay for
print(tok.decode(ids))              # back to the original text

# for chat models, let the tokenizer add the special tokens for you
# chat_tok = AutoTokenizer.from_pretrained("<an instruct model id>")
# prompt = chat_tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

In the GPT-2 vocabulary, a Ġ at the start of a token means "this token begins with a space". It is the byte-level way of storing that leading space.

Common tokenization mistakes

• Estimating cost in words. Budget in tokens, and measure non-English text separately.
• Mixing tokenizers and models. Token ID 4,000 means something different in every vocabulary. Always load the tokenizer from the same model id.
• Hand-writing chat markers. Use apply_chat_template instead of guessing the special tokens.
• Ignoring leading spaces. "Yes" and " Yes" are usually different tokens, which matters when you check model output or set stop sequences.
• Expecting digit-perfect arithmetic. Numbers are often split into odd chunks such as "20" and "26", one reason LLMs slip on long sums.

Key takeaways

• A tokenizer turns text into token IDs from a fixed vocabulary; the model only ever sees those IDs.
• BPE learns its vocabulary by merging frequent character pairs; the ordered merge list is the tokenizer.
• Byte-level BPE can encode any script, but scripts with fewer learned merges, such as Hindi, use more tokens.
• Special tokens and chat templates mark roles and the end of a reply; let the library add them.
• Always count tokens with the exact tokenizer of the model you deploy.

Next in this series, Transformers and Attention Explained shows what the model does with those IDs. If loops and dictionaries in the code felt new, revise them with the Python strings tutorial and then test yourself with quick quizzes on Debugshala Test.