Free course · by Debugshala, Indore
Build an LLM from Scratch Tutorial
11 free lessons, in the order we teach them in class: start at lesson 1 and work down. Every lesson has an analogy, a diagram, code you can run or copy, a quick quiz and an FAQ.
- 11Lessons
- 3Modules
- ~2 hReading time
- FreeAlways
01Build an LLM: Plan and Data
- Build a Small LLM from Scratch: The Complete 2026 RoadmapA step-by-step roadmap to build a 25M–100M parameter LLM on one GPU: tokenizer, data, architecture, pretraining, SFT, alignment, evaluation and deployment.
- Train a BPE Tokenizer for Your LLM: Vocab Size and Chat TokensTrain a byte-level BPE tokenizer for a small LLM: the merge loop in plain Python, choosing vocab size, chat special tokens and Hugging Face tokenizers code.
- LLM Training Data Preparation: Pretraining and Chat DatasetsPrepare LLM training data step by step: token budgets, cleaning and dedup, packing into blocks, uint16 token files, and chat, DPO and RL formats.
- Small LLM Architecture in Code: RMSNorm, RoPE, GQA and SwiGLUBuild a small decoder-only LLM in PyTorch: count parameters, then code RMSNorm, RoPE, grouped-query attention and SwiGLU with runnable numpy demos.
02Build an LLM: Training
- Pretrain an LLM from Scratch: Loss, LR Schedule and CheckpointsPretrain a small LLM in PyTorch: next-token loss, warmup and cosine LR, bf16, gradient accumulation, checkpoints, resuming and reading the loss curve.
- Supervised Fine-Tuning Tutorial: From Base LLM to Chat ModelSupervised fine-tuning in PyTorch: render chats with a ChatML template, mask the loss to assistant tokens, pick SFT hyperparameters and test your chat model.
- LLM Knowledge Distillation: Teacher Data and Reasoning TracesDistil a small LLM from a bigger teacher: black-box SFT on teacher text, white-box KL with temperature, and clean reasoning data with think tags.
- DPO Training Tutorial: Align a Small LLM with Preference PairsTrain a small LLM with DPO: preference pair format, the loss from log-probs, choosing beta, a PyTorch loop and the reward curves that warn of trouble.
- PPO and GRPO for LLMs: Hands-On RL, CISPO and Tool CallingHands-on PPO and GRPO for small LLMs: rewards, group advantages, GAE, the CISPO loss tweak and agentic RL for tool calling, with runnable numpy code.
03Build an LLM: Evaluate and Ship
- How to Evaluate a Small LLM: Loss, Benchmarks and RoPE TestsEvaluate the small LLM you trained: loss and perplexity checks, a fixed prompt set, lm-evaluation-harness benchmarks and RoPE length extrapolation.
- Deploy a Small LLM: Transformers, GGUF, Ollama and OpenAI APIExport your own small LLM to Transformers format, convert it to GGUF, quantise it, run it in Ollama or vLLM and serve an OpenAI-compatible API.