SFT vs RLHF vs GRPO: Choosing an LLM Post-training Method

At a cricket academy in Indore, a young fast bowler learns in three phases. First, the coach bowls and the student copies the action, delivery by delivery.

Second, the coach watches pairs of the student's deliveries and simply says which one was better, because he cannot always explain why. Third, the student bowls overs at a single stump, a machine counts the hits, and each ball is judged against the average of that over.

Copying gets the basics right fast. Comparing teaches subtle things a demonstration cannot. Scoring against a target lets the student discover techniques the coach never showed.

Modern LLM post-training follows the same three phases.

What are SFT, RLHF and GRPO?

SFT (supervised fine-tuning), RLHF (reinforcement learning from human feedback) and GRPO (group relative policy optimization) are post-training methods that improve a pretrained LLM using, respectively, example answers to imitate, human preferences between answers, and automatically checked rewards compared within a group of the model's own attempts.

Mapping the academy: the coach's demonstrations are SFT data, the "this one was better" calls are preference pairs for RLHF or DPO, and the stump-hit machine is a reward function for GRPO. Judging each ball against that over's average is the "group relative" part of GRPO's name.

How LLMs are trained introduced each stage. This tutorial compares them side by side: what data each needs, how much compute, what can go wrong, and how to run them with Hugging Face TRL.

Three signals: copy this answer, prefer this answer, beat your own average.

SFT vs RLHF vs GRPO compared

SFT is cheap and stable but can only imitate. Preference methods capture taste. GRPO lets the model find solutions better than any example you could write, as long as a program can tell right from wrong.

SFT: learning by imitation

In SFT the model reads a prompt and is trained to predict your reference answer token by token, with prompt tokens usually left out of the loss. It needs no reward model and no sampling, so it is the cheapest and most predictable method.

Its limit is that the model only sees perfect answers, never its own mistakes. If your references are mediocre, the model learns to be mediocre. Building good references is covered in creating an instruction fine-tuning dataset, and the low-memory way to run SFT is in fine-tuning LLMs with LoRA.

RLHF and DPO: learning from preferences

Classic RLHF trains a separate reward model on human comparisons, then uses the PPO algorithm to push the LLM toward high-reward answers. PPO needs the policy being trained, a frozen reference copy, the reward model and a value model, often all in GPU memory at once. It is powerful and notoriously fiddly.

DPO (Direct Preference Optimization) reaches a similar goal with one simple loss on the preference pairs. It compares how much the policy has raised the probability of the chosen answer, relative to a frozen reference, against how much it has raised the rejected one:

import math

def dpo_loss(pol_chosen, ref_chosen, pol_rejected, ref_rejected, beta=0.1):
    """Inputs are total log-probabilities of each answer under the policy and the frozen reference."""
    chosen_gain = pol_chosen - ref_chosen          # how much more likely the good answer became
    rejected_gain = pol_rejected - ref_rejected    # how much more likely the bad answer became
    margin = beta * (chosen_gain - rejected_gain)
    return -math.log(1 / (1 + math.exp(-margin))), margin

cases = {
    "start of training (policy = reference)": (-42.0, -42.0, -40.0, -40.0),
    "policy now prefers the chosen answer":    (-38.0, -42.0, -46.0, -40.0),
    "policy drifted toward the rejected one":  (-44.0, -42.0, -37.0, -40.0),
}
for name, logps in cases.items():
    for beta in (0.1, 0.5):
        loss, margin = dpo_loss(*logps, beta=beta)
        print(f"{name:<40} beta={beta}: margin={margin:+.2f}  loss={loss:.3f}")

At the start the loss is log 2, about 0.693, because the policy equals the reference. Training lowers it by widening the gap between chosen and rejected. The beta setting controls how sharply the loss reacts and, in practice, how far the model may drift from the reference. TRL's default is 0.1.

DPO needs only two models and no sampling, which made it the standard preference method for open models. Its weaknesses: it can favour longer answers if your chosen answers tend to be longer, and it learns only from fixed pairs.

GRPO: learning from verifiable rewards

GRPO, introduced in DeepSeek's 2024 maths research and used widely for reasoning models since, drops the reward model and the value model.

For each prompt it samples a group of answers, scores each with a reward function, and computes each answer's advantage as its score minus the group mean, divided by the group's standard deviation. Above-average answers are reinforced; below-average ones are discouraged.

The reward function is ordinary code. Here is one for maths problems that rewards a correct final answer and a little for the right format:

import re

def reward(completion, answer):
    score = 0.0
    if re.search(r"<answer>.*?</answer>", completion, re.S):
        score += 0.2                                   # followed the output format
    found = re.search(r"<answer>\s*([-\d.]+)\s*</answer>", completion)
    if found and float(found.group(1)) == answer:
        score += 1.0                                   # correct final answer
    return score

group = [   # 6 sampled completions for "A bus covers 120 km in 2.5 hours. Speed in km/h?"
    "120 / 2.5 = 48 <answer>48</answer>",
    "Speed is 50 <answer>50</answer>",
    "120 / 2.5 = 48. The answer is 48.",
    "<answer>48</answer>",
    "2.5 * 120 = 300 <answer>300</answer>",
    "Distance over time gives 48 km/h <answer> 48 </answer>",
]
for c in group:
    print(f"{reward(c, 48):.1f}  {c}")

Turning those scores into advantages is a few lines of numpy. The second group shows GRPO's blind spot:

import numpy as np

def advantages(rewards, eps=1e-4):
    r = np.array(rewards, dtype=float)
    return (r - r.mean()) / (r.std() + eps)

groups = {
    "mixed results":       [1.2, 0.2, 0.0, 1.2, 0.2, 1.2],
    "all wrong (too hard)": [0.2, 0.2, 0.2, 0.2, 0.2, 0.2],
    "all right (too easy)": [1.2, 1.2, 1.2, 1.2, 1.2, 1.2],
}
for name, rewards in groups.items():
    adv = advantages(rewards)
    print(f"{name:<22} advantages={np.round(adv, 2)}  learning signal={np.abs(adv).sum():.2f}")

When every answer in a group gets the same score, all advantages are zero and that prompt teaches nothing. So GRPO training sets are chosen to be just hard enough: the model solves them sometimes. The group also replaces the value model that PPO needs, which is GRPO's main memory saving.

Choosing between SFT, RLHF and GRPO

Most real pipelines pass through more than one box, in this order.

Follow these steps for a project:

1. Start with SFT whenever you can write or collect good answers. It teaches format and basic behaviour, and later methods work much better from an SFT starting point.
2. Add DPO if reviewers keep preferring some answers over others for reasons that are hard to write down, such as tone or clarity.
3. Add GRPO if the task has checkable answers and you want the model to get better than your examples, for example at multi-step calculations or code that passes tests.
4. Evaluate after each stage on a held-out set, including general questions, to catch lost skills. Putting each checkpoint behind a Gradio demo on Spaces lets reviewers compare them by hand.

Running SFT, DPO and GRPO with Hugging Face TRL

TRL provides one trainer per method with the same overall shape. These snippets follow the current TRL documentation; they need a GPU and network access:

import re
from datasets import load_dataset
from trl import DPOConfig, DPOTrainer, GRPOConfig, GRPOTrainer

# DPO: rows with "prompt", "chosen" and "rejected"
prefs = load_dataset("json", data_files="preferences.jsonl", split="train")
dpo = DPOTrainer(model="your-username/admissions-sft",           # start from your SFT model
                 args=DPOConfig(output_dir="admissions-dpo", beta=0.1),
                 train_dataset=prefs)
dpo.train()

# GRPO: rows with "prompt" and an extra "answer" column, passed to the reward function
def correct_answer(completions, answer, **kwargs):
    rewards = []
    for text, gold in zip(completions, answer):
        m = re.search(r"<answer>\s*([-\d.]+)\s*</answer>", text)
        rewards.append(1.0 if m and float(m.group(1)) == float(gold) else 0.0)
    return rewards

problems = load_dataset("json", data_files="fee_problems.jsonl", split="train")
grpo = GRPOTrainer(model="your-username/admissions-sft",
                   reward_funcs=correct_answer,
                   args=GRPOConfig(output_dir="admissions-grpo", num_generations=8),
                   train_dataset=problems)
grpo.train()

Two details worth knowing. The reward function receives every dataset column except the prompt as a keyword argument, which is how answer reaches it.

And recent TRL versions set the GRPO KL penalty beta to 0.0 by default, so no reference model is loaded; set it above zero if you want to keep the model closer to its starting point. Defaults change between releases, so check your installed version.

Common mistakes with post-training methods

• Skipping SFT. Preference and RL methods on a raw base model are unstable and slow to learn format.
• Reward hacking. A reward that only checks for <answer> tags will be satisfied with tags and nonsense. Read samples every few hundred steps.
• Unbalanced preference pairs. If chosen answers are always longer, DPO learns "longer is better".
• Prompts that are too easy or too hard for GRPO. Zero-variance groups give zero signal.
• No general evaluation. Narrow training can erode broad skills; test a few general questions too.
• Trusting a rising reward curve. It proves the model found what the reward pays for, not that it improved.

Key takeaways

• SFT imitates answers, RLHF and DPO learn from preferences, and GRPO learns from verifiable rewards.
• SFT is cheapest and comes first; DPO captures taste with two models; GRPO can exceed its training examples on checkable tasks.
• GRPO's advantage is reward minus group mean over group spread; identical rewards teach nothing.
• Every reward can be gamed, so read samples and evaluate broadly after each stage.
• Hugging Face TRL offers SFTTrainer, DPOTrainer and GRPOTrainer with a shared interface.