How LLMs Are Trained: Pretraining, SFT and RLHF Explained

Think about how a new cook joins a busy food stall at 56 Dukan. For months he simply watches and tastes thousands of orders, learning how flavours combine and which orders usually follow which.

Then the head cook trains him on the stall's own menu. "When someone asks for a garlic dosa, this is exactly how we make it." He copies worked examples until he serves dishes the house way.

Finally customers start rating his plates. Nobody writes him a recipe for "better"; he simply learns, from which plate gets the thumbs up, what people prefer.

What does training an LLM mean?

The cook's three phases map neatly onto the three classic LLM training stages:

Training an LLM means adjusting its billions of parameters, step by step, so that it assigns higher probability to good text: first to real text from the web and books, then to example answers, and finally to the answers people rank higher.

All three stages use the same tool, gradient descent, which nudges each weight in the direction that lowers a loss. What changes from stage to stage is the data and the loss. If gradient descent is new to you, read gradient descent and optimizers first.

Each stage starts from the previous stage's model and uses smaller but more carefully chosen data.

Stage 1: pretraining on next-token prediction

Pretraining is where almost all the compute goes. The model reads a vast mix of web pages, books, code, papers and other text, and at every position it tries to predict the next token. Recent open models report pretraining on well over ten trillion tokens.

The loss is cross-entropy: for each position, take the probability the model gave to the true next token and compute minus its logarithm. A confident correct guess costs almost nothing; giving the right token a tiny probability costs a lot.

import numpy as np

# probability each model gave to the TRUE next token at each position of
# "Indore is famous for poha and jalebi"
targets = ["is", "famous", "for", "poha", "and", "jalebi"]
vocab_size = 50_000
untrained = np.full(len(targets), 1 / vocab_size)    # random guessing
trained = np.array([0.30, 0.45, 0.90, 0.40, 0.55, 0.60])

def cross_entropy(p):
    return -np.log(p).mean()            # average surprise per token

for name, p in [("untrained", untrained), ("trained", trained)]:
    loss = cross_entropy(p)
    print(f"{name:<10} loss={loss:6.2f}  perplexity={np.exp(loss):9.1f}")

worst = targets[int(np.argmin(trained))]
print("hardest token for the trained model:", worst)

An untrained model is guessing among 50,000 tokens, so its perplexity, the exponential of the loss, is 50,000. After training it behaves as if choosing between about two plausible tokens.

Pretraining is mostly a data engineering problem. Teams filter out spam and low-quality pages, remove duplicates, strip personal data where possible, and balance languages and domains. The data mix shapes what the model knows.

The result is a base model. Ask it "What is the capital of Madhya Pradesh?" and it might reply with three more quiz questions, because on the web a question is often followed by more questions. It knows a lot but does not yet behave like an assistant.

Stage 2: supervised fine-tuning (SFT)

Supervised fine-tuning, also called instruction tuning, continues training on curated conversations: a user request paired with an ideal assistant answer. Typical SFT sets range from tens of thousands to a few million examples, written by people, generated by stronger models and then filtered, or both.

The loss is the same next-token cross-entropy, with one important change. The model should learn to write answers, not to predict users' questions, so the prompt tokens are masked out of the loss.

# one SFT example after the chat template, as (token, role) pairs
tokens = [("<user>", "user"), ("Best", "user"), ("snack", "user"), ("in", "user"),
          ("Indore", "user"), ("?", "user"), ("<assistant>", "user"),
          ("Poha", "assistant"), ("at", "assistant"), ("Sarafa", "assistant"),
          (".", "assistant"), ("<end>", "assistant")]

IGNORE = -100                                   # PyTorch's ignore index for loss
labels = [i if role == "assistant" else IGNORE for i, (_, role) in enumerate(tokens)]

# made-up per-token losses from the model, just to show the masking
token_loss = [2.1, 3.0, 2.7, 1.9, 2.4, 0.8, 0.5, 1.2, 0.9, 1.6, 0.3, 0.2]
counted = [l for l, lab in zip(token_loss, labels) if lab != IGNORE]

print("skipped:", " ".join(t for (t, _), lab in zip(tokens, labels) if lab == IGNORE))
print("learned:", " ".join(t for (t, _), lab in zip(tokens, labels) if lab != IGNORE))
print(f"\nloss over all tokens      : {sum(token_loss) / len(token_loss):.2f}")
print(f"loss over assistant tokens: {sum(counted) / len(counted):.2f}  <- what SFT trains on")

The label value -100 is the convention PyTorch's cross-entropy uses for "ignore this position". Libraries such as Hugging Face TRL can apply this masking for you.

SFT is where the model picks up the chat format, an assistant's tone, refusal behaviour and skills such as writing JSON. Quality beats count: a few thousand excellent examples beat a million sloppy ones.

Building such a set is the topic of creating an instruction fine-tuning dataset, and the cheap way to run SFT yourself is covered in fine-tuning LLMs with LoRA.

Stage 3: preference tuning with RLHF and DPO

Many questions have no single correct answer. Which of two explanations of GST is clearer? Is a reply too long? SFT cannot teach this well, because you would need a perfect example for every case. It is far easier for people to compare two answers than to write the best one.

Reinforcement learning from human feedback (RLHF), popularised by OpenAI's InstructGPT work in 2022, turns those comparisons into training signal in three steps:

1. Collect preferences. For each prompt, the model writes two or more answers and human raters pick the better one.
2. Train a reward model. A separate model learns to give a higher score to the chosen answer than to the rejected one.
3. Optimise the LLM against the reward. An RL algorithm, classically PPO, updates the LLM to produce answers the reward model scores highly, with a penalty for drifting too far from the SFT model.

RLHF in one loop: answers, human comparisons, a learned reward, then a policy update kept on a leash by the KL penalty.

The reward model is trained with a simple pairwise loss: the probability that the chosen answer wins is the sigmoid of the score difference.

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# reward-model scores for pairs of answers to the same prompt: (chosen, rejected)
pairs = {
    "clear, correct answer vs vague one":   (2.4, 0.3),
    "polite refusal vs harmful how-to":     (1.8, -1.5),
    "two nearly equal answers":             (0.9, 0.8),
    "reward model got it backwards":        (0.2, 1.6),
}

losses = []
for name, (r_chosen, r_rejected) in pairs.items():
    p = sigmoid(r_chosen - r_rejected)      # P(humans prefer the chosen answer)
    loss = -np.log(p)                       # small when the model agrees with humans
    losses.append(loss)
    print(f"{name:<38} P(chosen)={p:.2f}  loss={loss:.2f}")

print(f"\naverage pairwise loss: {np.mean(losses):.2f}")

The fourth pair is the dangerous one. When the reward model disagrees with people, the LLM will happily learn to exploit that error, a failure called reward hacking. The penalty for drifting from the SFT model, measured with KL divergence, limits how far it can go chasing a flawed reward.

DPO: preference tuning without a separate reward model

Direct Preference Optimization (DPO), published in 2023, skips the reward model and the RL loop. It trains the LLM directly on chosen and rejected pairs with a loss that raises the probability of the chosen answer relative to the rejected one, compared against a frozen reference copy.

It is simpler and more stable to run, which is why it is popular for open models.

Stage 4: reasoning with verifiable rewards

For maths, coding and logic puzzles there is a correct answer, and a program can check it: run the unit tests, compare the final number. That removes the need for human raters.

In this stage the model attempts each problem several times, a checker scores each attempt, and RL pushes up the attempts that scored well. GRPO, introduced in DeepSeek's 2024 maths research, compares each attempt with the average of its group instead of training a separate value model.

Models trained this way learn to write longer step-by-step reasoning before answering. The trade-offs between these methods get their own comparison in SFT vs RLHF vs GRPO.

Common misconceptions about LLM training

• "The model learns from my chats." Inference never changes weights. A provider may use logged conversations in a future training run if its policy allows, which is a separate decision.
• "Fine-tuning adds new knowledge reliably." SFT mostly teaches behaviour and format. For fresh facts, retrieval is usually more dependable.
• "RLHF makes the model truthful." It makes answers preferred. People sometimes prefer confident or flattering answers, so preference tuning can encourage that too.

Key takeaways

• LLM training runs in stages: pretraining, SFT, preference tuning and, increasingly, RL with verifiable rewards.
• Pretraining uses next-token cross-entropy over trillions of tokens and consumes most of the compute.
• SFT uses the same loss on curated conversations, with prompt tokens masked out.
• RLHF trains a reward model from human comparisons and optimises against it with a KL penalty; DPO gets a similar effect without the reward model.
• Each stage changes behaviour more than knowledge, and each can fail if its data or reward is flawed.

Before this article you saw the architecture in transformers and attention explained. Next, mixture of experts shows how large models grow capacity without growing the cost of every token.