How to Evaluate a Small LLM: Loss, Benchmarks and RoPE Tests

Think about how a new driver in Indore gets better. She glances at the dashboard every few minutes: fuel, temperature, warning lights. Every Sunday she drives the same route to Rajwada and notices whether her turns feel smoother. Once, she takes the official driving test. Later she tries a long highway trip to Ujjain, longer than anything she practised.

Each check answers a different question, and none replaces the others.

What does evaluating a small LLM involve?

Evaluating a small LLM means measuring, at each training stage, whether the model predicts text better, answers your prompts better, scores above chance on standard benchmarks and still works on inputs longer than it was trained on.

Mapping the driver: the dashboard is your training and validation loss. The Sunday route is a fixed prompt set. The driving test is a benchmark suite such as lm-evaluation-harness. The highway trip is a length extrapolation test for RoPE.

This tutorial is specific to a model you trained yourself in this series, from pretraining through DPO and PPO or GRPO. For evaluating LLM applications in general, with golden sets and metrics, see LLM evaluation methods.

The cheaper the check, the more often you run it.

Check training loss and perplexity

Loss is the first thing to read, and it catches most bugs early.

A freshly initialised model guesses roughly uniformly over the vocabulary, so its loss should start near ln(vocab size). With a 6,400-token vocabulary that is about 8.76. If your step-0 loss is 30 or 0.5, something is wrong with initialisation, labels or the shift between inputs and targets.

Perplexity is exp(loss). You can read it as "the model is as unsure as if it were choosing among this many tokens".

import math

vocab_size = 6400
print(f"loss of a random model ~ ln({vocab_size}) = {math.log(vocab_size):.2f}")

# Mean validation cross-entropy (nats per token) logged at a few checkpoints
log = {"step 0": 8.80, "step 2k": 4.90, "step 10k": 3.40, "step 30k": 2.95, "step 60k": 2.80}
for name, loss in log.items():
    print(f"{name:<9} loss={loss:.2f}  perplexity={math.exp(loss):8.1f}")

# Perplexity must come from the token-weighted mean loss, not a mean of batch perplexities
batches = [(2.5, 1000), (3.5, 100)]           # (mean loss, tokens) per batch
tok_mean = sum(l * n for l, n in batches) / sum(n for _, n in batches)
naive = sum(math.exp(l) for l, _ in batches) / len(batches)
print(f"correct ppl={math.exp(tok_mean):.1f}  averaged-batch ppl={naive:.1f}")

The numbers above are illustrative, not measurements. Your own curve depends on data, model size and tokenizer. The last two lines show a real trap: average the loss over all tokens first, then take exp. Averaging per-batch perplexities overweights the bad batches.

What to look for in the curves

Two rules keep perplexity honest. Compare only models that share a tokenizer, because a bigger vocabulary changes what "one token" means. And keep a held-out validation file that never enters training.

Build a fixed prompt set for before/after comparisons

Loss tells you the model predicts text better. It does not tell you whether answers got better. For that, use the same prompts at every checkpoint.

Follow these steps:

1. Write 20 to 50 prompts across categories: simple facts, instructions with a format ("list three…"), short arithmetic, a little code, a request the model should refuse, and a few in Hindi if your data includes it.
2. Add cheap checks where possible: a keyword that must appear, a number, a required format.
3. Fix decoding. Use greedy decoding, or a fixed temperature and seed, so differences come from the weights, not chance. See LLM decoding strategies.
4. Save outputs per checkpoint as jsonl, named after the stage: pretrain, sft, dpo, grpo.
5. Read them side by side, then score with a rubric or with an LLM as a judge.

The helper below works on a checkpoint exported to Transformers format, as shown in the deployment tutorial. For a raw .pt checkpoint, the chat() function from the SFT tutorial with a fixed seed does the same job.

import json, os, torch

@torch.no_grad()
def run_prompt_set(model, tokenizer, prompts, tag, max_new_tokens=200):
    model.eval()
    os.makedirs("evals", exist_ok=True)
    with open(f"evals/{tag}.jsonl", "w") as f:
        for p in prompts:
            text = tokenizer.apply_chat_template(
                [{"role": "user", "content": p["prompt"]}],
                tokenize=False, add_generation_prompt=True)
            ids = tokenizer(text, return_tensors="pt").input_ids.to(model.device)
            out = model.generate(ids, max_new_tokens=max_new_tokens, do_sample=False,
                                 eos_token_id=tokenizer.eos_token_id)
            reply = tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
            f.write(json.dumps({"id": p["id"], "reply": reply}, ensure_ascii=False) + "\n")

Then run automatic checks over the saved files. This version tests required keywords and a simple repetition score, distinct-2, the share of unique word pairs:

# Cheap automatic checks on a fixed prompt set, run for two checkpoints.
prompts = [
    {"id": "fact-1", "must": ["bhopal"]},
    {"id": "math-1", "must": ["105"]},
    {"id": "fmt-1",  "must": ["1.", "2.", "3."]},
]
outputs = {
    "sft": {"fact-1": "The capital of Madhya Pradesh is Bhopal.",
            "math-1": "Seven pens cost Rs 100. Seven pens cost Rs 100. Seven pens cost",
            "fmt-1":  "1. Wake up 2. Brush 3. Eat breakfast"},
    "dpo": {"fact-1": "Bhopal is the capital of Madhya Pradesh.",
            "math-1": "One pen costs Rs 15, so seven pens cost Rs 105.",
            "fmt-1":  "Wake up, brush and then eat breakfast."},
}

def distinct2(text):
    w = [t.strip(".,!?") for t in text.lower().split()]
    pairs = list(zip(w, w[1:]))
    return len(set(pairs)) / max(1, len(pairs))

for ckpt, outs in outputs.items():
    hits = sum(all(m in outs[p["id"]].lower() for m in p["must"]) for p in prompts)
    rep = sum(distinct2(outs[p["id"]]) for p in prompts) / len(prompts)
    print(f"{ckpt}: passed {hits}/{len(prompts)} checks, distinct-2 = {rep:.2f}")
for p in prompts:
    a, b = (all(m in outputs[c][p["id"]].lower() for m in p["must"]) for c in ("sft", "dpo"))
    if a != b:
        print(f"  {p['id']}: {'fixed' if b else 'REGRESSED'} after DPO")

Both checkpoints pass two of three checks, but they are different two. DPO fixed the maths answer and stopped repeating itself, yet broke the numbered-list format. Totals hide this; per-prompt diffs reveal it.

Run benchmarks with lm-evaluation-harness

lm-evaluation-harness from EleutherAI is the standard open tool for running hundreds of benchmarks the same way. It loads models in Hugging Face Transformers format, so export first as shown in exporting and serving your LLM.

pip install "lm_eval[hf]"
lm_eval --model hf \
  --model_args pretrained=./export/my-llm,dtype=float16 \
  --tasks arc_easy,hellaswag,piqa,openbookqa \
  --device cuda:0 --batch_size 16 \
  --output_path results/sft
# For a chat-tuned checkpoint, add: --apply_chat_template

How multiple-choice scoring works

Most of these tasks never ask the model to write an answer. The harness feeds the question plus each candidate answer and sums the log-probabilities of the answer's tokens. The highest score wins.

No text generation happens, so even a weak model gets a fair, deterministic score.

Longer options collect more negative log-probs, so the harness also reports acc_norm, which divides by the option's length in characters. The two metrics can disagree:

import numpy as np

# One multiple-choice item, scored the way benchmark harnesses do it:
# sum the log-probs of each option's tokens given the question, pick the highest.
options = {
    "boil the water":                 [-1.2, -0.8, -0.5],
    "put the water in the freezer":   [-1.0, -0.9, -0.7, -0.6, -0.9, -0.4],
    "pour the water on the floor":    [-2.5, -0.9, -0.7, -0.6, -2.2],
}
answer = "boil the water"
n_chars = {o: len(o) for o in options}

total = {o: sum(lp) for o, lp in options.items()}
norm = {o: total[o] / n_chars[o] for o in options}     # length-normalised score
pick_acc = max(total, key=total.get)
pick_norm = max(norm, key=norm.get)
for o in options:
    print(f"{o:<30} sum={total[o]:6.2f}  per-char={norm[o]:6.3f}")
print("acc picks:", pick_acc, "->", pick_acc == answer)
print("acc_norm picks:", pick_norm, "->", pick_norm == answer)

Report both, and say which one you quote.

Compare with chance and with the error bar

Be ready for humbling numbers. For reference, GPT-2 small (124M parameters) is commonly reported at roughly 30% on HellaSwag, only a few points above chance. A 25M to 100M model trained on a few billion tokens will usually sit close to chance on several of these tasks. That is normal for a learning project.

The harness prints a standard error for every score. Use it before you celebrate:

import math

def acc_with_stderr(correct, n):
    p = correct / n
    return p, math.sqrt(p * (1 - p) / n)

chance = {"4-option task": 0.25, "2-option task": 0.50}
runs = {"SFT model": (1310, 5000), "DPO model": (1352, 5000)}   # (correct, items)

res = {k: acc_with_stderr(*v) for k, v in runs.items()}
for k, (p, se) in res.items():
    print(f"{k}: {p:.1%} +/- {se:.1%} (chance 25%)")

(p1, s1), (p2, s2) = res["SFT model"], res["DPO model"]
diff, se_diff = p2 - p1, math.sqrt(s1**2 + s2**2)
print(f"difference {diff:+.1%}, needs > {2 * se_diff:.1%} to be meaningful (about 2 standard errors)")
print("real improvement?", abs(diff) > 2 * se_diff)

A gap of 0.8 points on 5,000 items is noise. Note also that a small model's score can jump just by aligning it to the question-and-options format, without it learning anything new. And never train on a benchmark's test split; contaminated scores are meaningless.

Test RoPE length extrapolation

Your model was trained with a fixed context length, perhaps 512 or 1,024 tokens. Rotary position embeddings, built in the architecture tutorial, encode position as rotation angles. Beyond the trained length, the model sees angle combinations it never learned, and quality collapses.

Scaling methods such as position interpolation and YaRN (Peng et al., 2023) shrink positions so long inputs fall back into the familiar angle range.

import numpy as np

head_dim, base, train_len = 96, 10000.0, 512
inv_freq = base ** (-np.arange(0, head_dim, 2) / head_dim)
slowest = inv_freq[-1]

print(f"slowest pair rotates {slowest * train_len:.3f} rad over the {train_len} trained positions")
for length in (512, 1024, 2048):
    plain = slowest * (length - 1)
    scaled = slowest * (length - 1) / (length / train_len)   # linear position interpolation
    print(f"len {length:5}: max angle plain={plain:.3f}  interpolated={scaled:.3f}")

# Simulated per-token losses on one long document, bucketed by position
rng = np.random.default_rng(1)
pos = np.arange(2048)
loss = 3.0 + rng.normal(0, 0.3, 2048) + np.where(pos >= train_len, 0.002 * (pos - train_len), 0)
for lo, hi in ((0, 512), (512, 1024), (1024, 2048)):
    print(f"positions {lo:4}-{hi:4}: ppl={np.exp(loss[lo:hi].mean()):6.1f}")

The second half uses simulated losses to show what a real test reveals. Perplexity is fine inside the trained 512 positions and climbs steeply beyond them.

To run it on your model:

1. Take 20 to 50 long documents your model never trained on, each longer than 4 times the training length.
2. Run one forward pass per document at the full length and keep the per-token loss.
3. Report perplexity per position bucket (0–512, 512–1,024, 1,024–2,048, and so on).
4. Repeat with RoPE scaling turned on and compare.

In Transformers format, scaling is a config change. Transformers 5 reads a rope_parameters block in config.json, while 4.x versions use the older key rope_scaling with the same fields (without rope_theta). Check the Transformers RoPE docs for your version.

"rope_parameters": {"rope_type": "yarn", "rope_theta": 10000.0, "factor": 4.0,
                    "original_max_position_embeddings": 1024}

Scaling fixes the position problem only. It cannot give a tiny model the skill of using long context well; it just stops perplexity from exploding.

Common evaluation mistakes

• Changing decoding settings between checkpoints, then crediting the difference to training.
• Comparing perplexity across tokenizers.
• Quoting one benchmark number without chance level or standard error.
• Only reading the average of a prompt set; regressions hide inside totals.
• Testing long context without a baseline, so you cannot tell whether scaling helped.

You can practise Python and logic questions on Debugshala Test to sharpen the scripting these checks need. If you would like to learn generative AI with an instructor, there is a Generative AI course near Old Palasia.

Key takeaways

• Start with loss: step 0 near ln(vocab), smooth decline, validation tracking training.
• A fixed prompt set with fixed decoding is the clearest before/after comparison.
• lm-evaluation-harness gives standard scores; read them against chance and standard error.
• Measure perplexity by position to test RoPE extrapolation, with and without scaling.