PPO and GRPO for LLMs: Hands-On RL, CISPO and Tool Calling

Imagine you cook poha for your family every morning. One week you try four versions: more lemon, less salt, extra peanuts, and the usual. Everyone rates each plate. You don't need to know the "perfect" poha. You only need to know which versions beat that morning's average.

Next week you lean towards whatever scored above average. Repeat for a month, and the poha keeps improving.

That loop of trying, scoring and shifting is reinforcement learning, and the "compare with the average of this batch" trick is exactly GRPO.

What are PPO and GRPO for LLMs?

PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization) are policy-gradient reinforcement learning algorithms that fine-tune an LLM on its own sampled answers, increasing the probability of answers with higher reward while limiting how far each update moves the model.

Mapping the kitchen: each version of poha is a sampled answer (a rollout). The family's rating is the reward. The morning's average is the baseline. Leaning towards better versions is the policy update. Not changing the whole recipe overnight is the clipping and KL penalty that keep updates small.

Unlike DPO, which learns from a fixed set of pairs, PPO and GRPO are on-policy. The model must keep generating fresh answers, because it learns from its own current behaviour. For the conceptual comparison, see SFT vs RLHF vs GRPO.

PPO vs GRPO: what runs on your GPU

Both start from your SFT checkpoint. The big difference is how they estimate whether an answer was better than expected.

The critic is a second trainable network. Removing it is GRPO's main practical win.

Reward design for a small model

Before choosing an algorithm, decide where rewards come from. There are three common sources:

1. Rule-based rewards: is the maths answer correct, does the JSON parse, are think tags closed, is the length sensible.
2. A reward model: a separate model, often 1B to 2B parameters, that scores an answer to a prompt with a continuous number.
3. Environment feedback: did the tool call succeed, did the code pass its tests, did the task finish.

Small models hit a hard limit called reward sparsity. Give a 64M model competition maths with a right-or-wrong reward and almost every answer scores 0, leaving nothing to learn. The fix is dense, continuous rewards: combine a reward model's score with small format bonuses, and pick tasks the model can sometimes solve.

How GRPO computes advantages

GRPO samples G answers per prompt, scores them, and normalises each reward against its own group: advantage = (reward - group mean) / group std.

import numpy as np

# 3 prompts, 4 sampled answers each (G = 4). Rewards from a scoring function.
rewards = np.array([
    [1.0, 0.0, 1.0, 0.0],    # mixed: a useful learning signal
    [0.9, 0.7, 0.8, 0.2],    # continuous scores
    [0.0, 0.0, 0.0, 0.0],    # every answer wrong: a degenerate group
])

mean = rewards.mean(axis=1, keepdims=True)
std = rewards.std(axis=1, keepdims=True)
adv = (rewards - mean) / (std + 1e-4)

np.set_printoptions(precision=2, suppress=True)
print("advantages:\n", adv)
dead = (std.ravel() < 1e-6)
print("degenerate groups (no gradient):", np.where(dead)[0].tolist())
print(f"share of prompts that teach nothing: {dead.mean():.0%}")

The third prompt is a degenerate group. All four answers scored 0, so every advantage is 0 and that prompt contributes nothing except KL. If a third of your batch looks like this, a third of your compute is wasted. Filter prompts that are too hard or too easy.

Every token of an answer gets that answer's advantage. The loss is the PPO-style clipped objective with a per-token KL penalty to the frozen reference model:

import torch

def grpo_loss(logp, old_logp, ref_logp, adv, mask, eps=0.2, beta=0.04):
    # logp, old_logp, ref_logp, mask: [B*G, T] per answer token; adv: [B*G]
    ratio = torch.exp(logp - old_logp)
    a = adv.unsqueeze(1)
    surrogate = torch.min(ratio * a, torch.clamp(ratio, 1 - eps, 1 + eps) * a)
    # Low-variance KL estimate: exp(ref - logp) - (ref - logp) - 1, always >= 0
    d = ref_logp - logp
    kl = torch.exp(d) - d - 1
    per_token = -(surrogate - beta * kl)
    return ((per_token * mask).sum(1) / mask.sum(1).clamp(min=1)).mean()

# One training step, in outline:
# 1. sample G answers per prompt (the chat() loop from the SFT tutorial, temperature 0.9)
# 2. rewards = score(prompts, answers)  -> advantages per group
# 3. old_logp = logp under the sampling policy (no grad); ref_logp from frozen ref
# 4. loss = grpo_loss(...); loss.backward(); clip grads to 1.0; optimizer.step()

A starting point for a small model: G of 4 to 8, a learning rate around 1e-7 to 1e-6, beta between 0.01 and 0.1, and answers capped at a few hundred tokens. The TRL GRPOTrainer docs list the same knobs if you prefer a library.

How PPO uses a critic and GAE

PPO gives each token its own advantage. The reward model scores the full answer once at the end. A small KL penalty is subtracted at every token. The critic guesses the expected future reward at each position, and Generalised Advantage Estimation (GAE) blends those guesses into advantages.

import numpy as np

# 5 generated tokens. Reward model score arrives only at the end; KL penalty per token.
values = np.array([0.20, 0.25, 0.30, 0.40, 0.55])   # critic's guesses
kl_pen = np.array([0.01, 0.02, 0.01, 0.03, 0.02])
rewards = -kl_pen.copy()
rewards[-1] += 1.0                                   # final score from the reward model

gamma, lam = 1.0, 0.95
adv = np.zeros_like(values)
last = 0.0
for t in reversed(range(len(values))):
    next_v = values[t + 1] if t + 1 < len(values) else 0.0
    delta = rewards[t] + gamma * next_v - values[t]
    last = delta + gamma * lam * last
    adv[t] = last
returns = adv + values
print("advantages:", np.round(adv, 3))
print("critic targets (returns):", np.round(returns, 3))

The returns are what the critic should have predicted. Until the critic learns, the actor follows a noisy signal, which is why PPO rewards often climb slowly at first.

PPO reuses each batch of rollouts for 2 to 4 update passes, stopping early if the KL between new and old policy passes a threshold. The critic is about as big as the actor, so expect roughly 1.5 to 2 times GRPO's memory.

What is CISPO?

Look closely at PPO and GRPO clipping. When a token's probability ratio moves outside the range 1 - eps to 1 + eps, the clipped branch wins and that token's gradient becomes exactly zero. For rare but important tokens, such as a "wait, let me recheck" in a reasoning trace, that can silence the very tokens you want to learn.

CISPO (Clipped Importance Sampling Policy Optimization), proposed in the MiniMax-M1 report (2025), clips the importance weight instead of the update. The ratio is capped and detached, then multiplied by the advantage and the token's log-probability. Every token keeps a gradient; only its size is limited.

import numpy as np

# One token, advantage A, ratio r = pi_new / pi_old. Compare per-token gradient weight.
eps, eps_high = 0.2, 5.0
for A in (1.0, -1.0):
    print(f"advantage {A:+.0f}")
    for r in (0.7, 1.0, 1.3, 2.0, 6.0):
        # PPO/GRPO: d/dlogp of min(r*A, clip(r)*A). Zero when the clipped branch wins.
        clipped = np.clip(r, 1 - eps, 1 + eps)
        ppo_w = r * A if r * A <= clipped * A else 0.0
        # CISPO: weight = stopgrad(min(r, eps_high)) * A, always multiplies grad of log p
        cispo_w = min(r, eps_high) * A
        print(f"  ratio={r:<4} PPO grad weight={ppo_w:+.2f}   CISPO grad weight={cispo_w:+.2f}")

With a positive advantage, PPO stops learning from a token once its ratio passes 1.2. CISPO keeps pushing, with the weight capped at 5. In code, CISPO is a one-line change to the GRPO loss above:

w = torch.clamp(ratio, max=eps_high).detach()                 # e.g. eps_high = 5.0
per_token = -(w * adv.unsqueeze(1) * logp - beta * kl)       # same groups, same rewards

Agentic RL: training tool calling with rewards

After SFT on tool-call examples, your model can emit a call like <tool_call>{"name": "calc", "arguments": {...}}</tool_call>. Agentic RL improves that skill by letting the model actually use tools during rollouts.

The reward arrives once per trajectory, after the last turn, not after each token.

Each rollout is a short episode:

1. Give the model a prompt plus the tool list, using the same chat template as SFT.
2. Generate one assistant turn. If it contains a tool call, parse the JSON and run the tool.
3. Append the tool's result as a tool message and generate again, up to a turn limit such as 3.
4. Score the whole trajectory: correct final answer, valid calls, closed tags, and a penalty if the turn limit was hit.
5. Use GRPO or CISPO advantages across G trajectories for the same prompt.

The critical detail is the loss mask. Tool results were written by the tool, not the model, so their tokens must be masked out. Otherwise the model is trained to "predict" calculator output instead of calling the calculator.

import json, re

def trajectory_reward(turns, expected):
    """Score a whole multi-turn tool-use episode once, at the end."""
    reward = 0.0
    for t in turns:
        if t.count("<tool_call>") != t.count("</tool_call>"):
            reward -= 0.5                       # broken tags
        for raw in re.findall(r"<tool_call>(.*?)</tool_call>", t, re.S):
            try:
                call = json.loads(raw)
                reward += 0.25 if "name" in call and "arguments" in call else -0.5
            except json.JSONDecodeError:
                reward -= 0.5                   # invalid JSON
    final = turns[-1]
    if "<tool_call>" in final:
        reward -= 0.5                           # ran out of turns mid-task
    elif expected in final:
        reward += 2.0                           # verified final answer
    return max(-3.0, min(3.0, reward))

good = ['<tool_call>{"name": "calc", "arguments": {"expr": "256*37"}}</tool_call>',
        "256 x 37 = 9472."]
bad_json = ['<tool_call>{"name": "calc", arguments: 256*37}</tool_call>', "It is 9000."]
no_tool = ["256 x 37 is about 9500."]
for name, ep in (("good", good), ("bad json", bad_json), ("no tool", no_tool)):
    print(f"{name:<9} reward = {trajectory_reward(ep, '9472'):+.2f}")

This scoring function has a hole. It pays 0.25 for every valid call, so a model can farm points by calling tools it doesn't need. That is reward hacking. Cap the tool bonus per episode, or pay it only when the final answer is correct.

Keep the tool set small and deterministic, such as a calculator and a clock.

Common RL mistakes on small LLMs

• Binary rewards on hard tasks. All-zero groups mean zero gradient. Use easier prompts or continuous scores.
• Watching the policy loss. Track mean reward, reward spread per group, KL to the reference and average answer length.
• No KL or too little. The model drifts into repetitive text that exploits the reward model.
• Unmasked tool outputs. The model learns to hallucinate tool results.

RL is only worth keeping if a fair comparison shows it helped, so the next step is evaluating your small LLM. If you want classroom support while you learn generative AI, Debugshala's GenAI course for non-IT graduates and freshers starts with a free demo class.

Key takeaways

• PPO uses a critic and GAE; GRPO uses group-relative advantages and trains one model.
• Small models need dense rewards and achievable tasks; watch the reward spread inside each group.
• The policy loss is not a progress metric; mean reward and real generations are.
• CISPO is a drop-in loss change for GRPO; agentic RL adds multi-turn rollouts, tool execution and masked tool outputs.