Build a Small LLM from Scratch: The Complete 2026 Roadmap

A polytechnic student in Indore can buy a scooter, or she can rebuild a 100cc engine on the workshop bench. The rebuilt engine will never beat a showroom bike. But after one rebuild she knows exactly what the piston, carburettor and clutch do, and she can diagnose any engine for the rest of her career.

Training a tiny language model yourself is that workshop engine. You will not compete with the frontier labs. You will understand, line by line, what happens between a pile of raw text and a model that answers "What is the capital of Madhya Pradesh?"

What does it mean to build an LLM from scratch?

Building an LLM from scratch means you design the tokenizer, write the transformer code, and train all of the model's weights yourself, starting from random numbers, instead of downloading a pretrained checkpoint and fine-tuning it.

In the engine analogy, the tokenizer is cutting raw metal into standard parts, pretraining is running the engine in until it moves smoothly, and supervised fine-tuning (SFT) is tuning it for a specific rider. Alignment (DPO or reinforcement learning) is the final road test where you adjust behaviour based on feedback.

If the words "token", "attention" or "pretraining" are new, read how tokenization works, transformers and attention and how LLMs are trained first. This series assumes that background and stays hands-on.

What are the stages of building an LLM?

Every serious LLM, from a 30M toy to a frontier model, goes through the same broad stages. The difference is scale, not shape.

Each stage takes the previous stage's output as input. A weak tokenizer or dirty data hurts every stage after it.

Here is the series as a map. Each row is one tutorial, in reading order.

Stages 2 to 6 are the core. Stages 7 to 9 are optional polish. Stages 10 and 11 turn an experiment into something you can show.

Which stages are optional?

You can stop after stage 6 and still have a working chat model. Distillation helps most when you can borrow outputs from a stronger model. DPO and RL help most when you can define "better" with pairs or an automatic checker. Skip them on your first build and come back later.

How big should your first model be?

The right size depends on three numbers: parameters, tokens and GPU hours. They are tied together by a well-known rule of thumb: training compute is roughly 6 × parameters × tokens floating-point operations.

A second rule, from DeepMind's Chinchilla paper, says a compute-efficient model sees roughly 20 training tokens per parameter. Small hobby models often go well beyond that, because extra tokens keep improving a small model and data is cheaper than a bigger GPU.

This series uses one running example, plus a smaller variant for quick experiments. Both share the same tokenizer and chat format:

You can check the compute and time yourself. The block below uses an assumed sustained throughput of 25 TFLOPS. That is a rough planning figure for a recent consumer GPU running bf16 at realistic utilisation, not a measurement, so change it to match your card.

# Rough compute and time plan for a small LLM (6*N*D rule of thumb)
models = {"small": 27e6, "running": 64e6}
tokens_per_param = 20
sustained_tflops = 25          # assumption: tune for your GPU
rupees_per_gpu_hour = 60       # assumption: check current cloud prices

for name, n_params in models.items():
    tokens = n_params * tokens_per_param
    flops = 6 * n_params * tokens
    hours = flops / (sustained_tflops * 1e12) / 3600
    print(f"{name:8s} params={n_params/1e6:5.0f}M tokens={tokens/1e9:4.2f}B "
          f"compute={flops:.1e} FLOPs  time~{hours:4.1f} h  cost~Rs {hours*rupees_per_gpu_hour:,.0f}")

The small variant needs about an hour of pure pretraining compute; the 64M running example needs five to six hours. Real runs take longer because of data loading, evaluation and checkpoints, so budget roughly 1.5 to 2 times the estimate.

What GPU do you need?

Memory matters more than speed for a first build. With mixed precision and the AdamW optimiser, training costs roughly 16 bytes per parameter for weights, gradients and optimiser state, plus activations that grow with batch size and sequence length.

# Training memory estimate: fp32 master weights + grads + 2 Adam moments
bytes_per_param = 4 + 4 + 8
for name, n in {"27M": 27e6, "64M": 64e6, "350M": 350e6}.items():
    static_gb = n * bytes_per_param / 1e9
    print(f"{name:5s} static training state ~{static_gb:4.2f} GB (+ activations)")

The 64M model's fixed state is about 1 GB, so an 8 GB to 12 GB card works if you keep the micro-batch small and use gradient accumulation. A 24 GB card lets you use longer sequences and bigger batches. No GPU at all? A rented cloud GPU for a day is enough for the running example.

What can a 30M to 100M parameter model actually do?

Be honest with yourself before you start. A model this size can:

• Write grammatical, on-topic sentences in the language and style of its training data.
• Follow a chat template, answer short factual questions it saw often in training, and stop at the right place.
• Learn narrow, checkable skills after fine-tuning, such as a fixed output format or simple tool-call JSON.

It cannot reliably do multi-step reasoning, long documents, accurate rare facts or code beyond short snippets. It will hallucinate confidently. Expect benchmark scores close to random guessing on hard multiple-choice tests; the evaluation tutorial shows how to measure this fairly.

That is fine. The goal is understanding. Once your 64M model works, the same code scales to larger models on rented hardware, and you will read research papers with far more confidence.

How to follow this series step by step

1. Set up Python and PyTorch. You need Python 3.10+, PyTorch 2.x with CUDA, and the Hugging Face tokenizers, datasets and transformers libraries. New to Python? Start with getting started and for loops.
2. Pick your data early. English, Hindi or both. Your tokenizer and data choices lock in which language the model can handle.
3. Build the tokenizer and data pipeline, then freeze them. Changing the tokenizer later forces you to retrain from zero.
4. Write the model and run a tiny overfit test. Train on 100 sentences until the loss is near zero. If it can't memorise 100 sentences, it has a bug.
5. Pretrain, then SFT. Save a checkpoint every few hundred steps.
6. Evaluate, then add distillation, DPO or GRPO only if they solve a problem you can see.
7. Export and serve. Put your model behind an API, the same way you would run an open model locally.

A suggested project layout keeps each stage separate:

my-small-llm/
  tokenizer/        tokenizer.json, tokenizer_config.json (chat template)
  data/             raw/, clean/, pretrain_train.bin, pretrain_val.bin, sft.jsonl
  model/            config.py, model.py
  train/            pretrain.py, sft.py, dpo.py, grpo.py
  checkpoints/      pretrain_last.pt, sft_epoch1.pt
  eval/             prompts.txt, results/
  export/           hf/, gguf/

Common mistakes on a first LLM build

• Starting too big. A 300M model that takes a week teaches you less than ten 30M runs that each take an hour.
• Skipping the overfit test. Most "my model outputs garbage" problems are bugs, such as labels not shifted or a broken causal mask.
• Changing the tokenizer midway. Every checkpoint is tied to its tokenizer.
• Judging the model by one prompt. Use a fixed prompt set and a held-out loss so you can compare runs.
• Expecting ChatGPT. Compare your model with your previous run, not with frontier models.

If you prefer learning with mentors in a classroom, Debugshala's Generative AI course, taught in English and Hindi, runs offline classes in Old Palasia, Indore.

Key takeaways

• An LLM build has 11 stages; stages 2 to 6 (tokenizer, data, architecture, pretraining, SFT) are the core.
• Plan with two rules of thumb: compute is about 6 × parameters × tokens, and about 20 tokens per parameter is compute-efficient.
• A 27M to 64M model fits one consumer GPU and trains in hours, not weeks.
• Run an overfit test before any long run, and freeze the tokenizer early.
• A small model is for learning how LLMs work, not for replacing hosted assistants.