Instruction Dataset Creation for LLM Fine-tuning: Practical Guide

A new restaurant on 56 Dukaan is training five cooks. The head chef writes a house recipe book. If she writes 40 versions of dal and no desserts, the cooks will be lost the first time someone orders gulab jamun. If every recipe uses different units, one in grams and one in "a fistful", the cooks will copy the inconsistency.

So she plans first: every dish on the menu, every common variation, written in the same format. She cooks each recipe before adding it to the book. And she keeps a few dishes out of the book entirely, for the final test, to see whether the cooks learned to cook or only memorised pages.

That is exactly how a good instruction dataset is built.

What is an instruction fine-tuning dataset?

An instruction fine-tuning dataset is a collection of examples, each pairing an input request (optionally with a system prompt and context) with the ideal model response, used to train an LLM with supervised fine-tuning so that it reproduces that behaviour.

Mapping the kitchen: the menu is your coverage plan, each recipe is one example, the fixed recipe format is your output style guide, cooking before adding is quality filtering, and the dishes kept out are the held-out test set.

The model learns everything in the responses, including their length, tone, structure and mistakes. How LLMs are trained explains why: supervised fine-tuning (SFT) is next-token prediction on your answers. The LoRA tutorial showed a basic record format and a quick validator. This guide covers building the dataset itself, once prompting vs RAG vs fine-tuning analysis says training is worth it.

Dataset formats: messages vs prompt-completion

Most training libraries accept two shapes. Hugging Face TRL, for example, takes either a messages list or a prompt and completion pair:

{"messages": [
  {"role": "system", "content": "You are the admissions assistant. Reply in under 60 words."},
  {"role": "user", "content": "Weekend batch hai kya Data Analytics ka?"},
  {"role": "assistant", "content": "Haan, weekend batch available hai. Timings aur start date ke liye office se confirm karein: +91-8982385539."}
]}
{"prompt": [{"role": "user", "content": "Classify: 'Fee paid but portal shows pending'"}],
 "completion": [{"role": "assistant", "content": "{\"category\": \"billing\"}"}]}

Use the messages shape for multi-turn chat and the prompt-completion shape for single-turn tasks. With prompt-completion data, TRL computes the loss only on the completion by default, so the model learns to answer rather than to imitate users.

Two consistency rules matter more than the format:

• Use the same system prompt in training and in production. A model trained with one system prompt and served with another behaves unpredictably.
• Write every answer in one house style. If half your answers start with "Sure!" and half do not, the model will flip between them at random.

Where do the examples come from?

The usual recipe is a mix: real inputs where you have them, expert-written answers for a core set, and synthetic examples to fill coverage gaps, all filtered the same way. If you generate data with a commercial model, read its terms first; some restrict using outputs to train other models.

A practical synthetic loop has four steps: write 20 to 50 seed examples by hand, ask a strong model for new inputs that vary the seeds, have it (or a human) draft answers in your style, then score every draft against a rubric and keep only the passes. Using a model as the scorer is covered in LLM-as-a-judge.

Every stage removes a specific kind of problem. Skipping one usually shows up later as a strange model habit.

Plan coverage before you write

Start with a grid. For a support assistant, the rows might be intents (fees, timings, admission, placement, out of scope) and the columns might be language and difficulty. Count what you have in each cell:

import pandas as pd

rows = (
    [("fees", "English", "easy")] * 40 + [("fees", "Hinglish", "easy")] * 25 +
    [("timings", "English", "easy")] * 35 + [("timings", "Hinglish", "hard")] * 3 +
    [("admission", "English", "hard")] * 12 + [("admission", "Hinglish", "easy")] * 18 +
    [("placement", "English", "easy")] * 6 + [("out_of_scope", "English", "easy")] * 2
)
df = pd.DataFrame(rows, columns=["intent", "language", "difficulty"])

grid = pd.crosstab(df["intent"], [df["language"], df["difficulty"]])
print(grid, "\n")

TARGET = 15
gaps = grid.stack([0, 1]).loc[lambda s: s < TARGET]
print(f"{len(gaps)} cells below {TARGET} examples:")
print(gaps.sort_values().to_string())

The grid shows what intuition hides: almost no hard Hinglish questions, and only 2 out-of-scope examples. Out-of-scope examples are the ones that teach the model to say "I can't help with that", so a dataset without them produces a model that confidently answers everything.

Filter for quality, then remove near-duplicates

Filtering happens in two passes. Rule checks are cheap and catch mechanical problems: empty answers, answers over the length limit, broken JSON, phone numbers or email addresses that should not be there. Rubric checks, by a person or a judge model, catch the rest: is the answer correct, in house style, and grounded in the facts provided?

Then remove near-duplicates. Synthetic generation loves to produce the same question with one word changed. Exact matching misses these; comparing overlapping character chunks, called shingles, catches them:

def shingles(text, k=4):
    t = " ".join(text.lower().split())
    return {t[i:i + k] for i in range(len(t) - k + 1)}

def jaccard(a, b):
    return len(a & b) / len(a | b)

questions = [
    "What is the fee for the Data Analytics course?",
    "What is the fees for the data analytics course?",
    "Data Analytics course ki fees kitni hai?",
    "Is there a weekend batch for Data Analytics?",
    "What's the fee for the Data Analytics course",
    "Do you help with placements after the course?",
]

kept, sets = [], []
for q in questions:
    s = shingles(q)
    best = max((jaccard(s, other) for other in sets), default=0)
    if best < 0.7:
        kept.append(q); sets.append(s)
    else:
        print(f"drop ({best:.2f}): {q}")
print("\nkept:")
for q in kept:
    print(" ", q)

The Hinglish version survives, which is what you want: it is a genuinely different input. For datasets of hundreds of thousands of rows, the same idea is run with MinHash to avoid comparing every pair.

Split by group so the test set stays honest

A random split is not enough. If you generated five paraphrases of one seed question, a random split puts some in training and some in testing, and your test score measures memory, not skill. Split by group instead: all variants of one seed go to the same side.

Same data, two splits. Only the group split tells you how the model handles questions it has never seen.

scikit-learn does this in a few lines:

from sklearn.model_selection import GroupShuffleSplit

examples = [f"seed{s}-variant{v}" for s in range(10) for v in range(4)]
groups = [e.split("-")[0] for e in examples]

random_test = set(examples[i] for i in range(0, 40, 5))     # every 5th row, ignoring groups
leaky = {e.split("-")[0] for e in random_test} & {g for e, g in zip(examples, groups) if e not in random_test}
print("random split: seeds appearing on both sides =", len(leaky))

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(examples, groups=groups))
train_seeds = {groups[i] for i in train_idx}
test_seeds = {groups[i] for i in test_idx}
print("group split: train", len(train_idx), "rows, test", len(test_idx), "rows")
print("seeds on both sides =", len(train_seeds & test_seeds))

Also decontaminate: make sure none of your test questions, or public benchmark questions you plan to report, appear in the training data.

Saving and training with Hugging Face datasets and TRL

Once the data is clean, load it with the datasets library, optionally push it to a private Hub repository, and hand it to TRL's SFTTrainer. This needs network access and, for training, a GPU:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})
data.push_to_hub("your-username/admissions-sft", private=True)     # optional, needs `hf auth login`

trainer = SFTTrainer(
    model="HuggingFaceTB/SmolLM2-360M-Instruct",
    train_dataset=data["train"],
    eval_dataset=data["test"],
    args=SFTConfig(output_dir="admissions-sft", num_train_epochs=2, learning_rate=1e-4,
                   eval_strategy="epoch", max_length=1024),   # prompt-completion: loss on completion only
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"),
)
trainer.train()

For messages data, set assistant_only_loss=True in SFTConfig to skip the user turns, which requires a chat template that marks assistant spans. Check the TRL SFT documentation for your version.

Write a short dataset card as you publish: where examples came from, how they were filtered, known gaps, and the licence. If you later want to go beyond imitation, with preference pairs or reward functions, see SFT vs RLHF vs GRPO.

Common mistakes in instruction dataset creation

• No out-of-scope examples. The model never learns to refuse or redirect.
• Inconsistent answer style. The model copies the inconsistency.
• Facts that will change. Fees and dates go stale in weights; keep them in RAG and teach the model to use provided context.
• Random splits over paraphrases. Test scores look great and mean nothing.
• Personal data left in logs. Remove names, phone numbers and IDs before training.
• Too long for the context. Check token lengths; examples cut off by max_length teach half-answers. Tokenization explained shows how to count.

Key takeaways

• An instruction dataset teaches behaviour by example; the model copies every habit in your answers.
• Plan a coverage grid of intents, languages and difficulty, and fill the gaps on purpose.
• Mix real, expert and synthetic examples, and filter all of them with rules and a rubric.
• Remove near-duplicates, re-check balance and split by group to keep the test honest.
• Keep the system prompt identical in training and production, and document the dataset.