Prompting vs RAG vs Fine-tuning: How to Choose for an LLM App

A car comes into a garage near Siyaganj making a grinding noise. An inexperienced mechanic says, "The gearbox needs rebuilding," and quotes Rs 40,000. The senior mechanic first asks three questions. Is the driver riding the clutch? Is the gear oil simply low? Or is a gear actually worn?

If it is the driver, a two-minute explanation fixes it for free. If the oil is low, a top-up fixes it, though it needs checking every service. Only a worn gear justifies the expensive rebuild, and even then the oil still needs topping up afterwards.

Choosing between prompting, RAG and fine-tuning works the same way. Diagnose first, then pay for the cheapest repair that fixes the real problem.

What is the difference between prompting, RAG and fine-tuning?

Prompting, retrieval-augmented generation (RAG) and fine-tuning are three ways to adapt an LLM to a task: prompting changes the instructions you send, RAG adds relevant external information to each request, and fine-tuning further trains the model so the new behaviour is stored in its weights.

Mapping the garage: explaining the clutch to the driver is prompt engineering, cheap and instant. Topping up the oil is RAG, a supply that must be refreshed as it runs out or goes stale. Rebuilding the gearbox is fine-tuning, expensive and slow, but it changes the machine itself. The senior mechanic's questions are error analysis.

The LoRA fine-tuning guide gave the short rule: prompt first, add RAG for facts, fine-tune for behaviour. This tutorial shows how to apply that rule to a real project, with failure diagnosis and cost numbers.

Where each approach acts in an LLM app

All three change a different part of the same request pipeline:

Prompting and RAG both change the input. Only fine-tuning changes the model.

That placement explains most of their strengths and weaknesses:

Two rows deserve emphasis. Updating knowledge: a fact baked into weights goes stale the day it changes, while a re-indexed document is current within minutes. Per-request cost: fine-tuning is the only option that can make each request cheaper, because a small tuned model with a short prompt can replace a large model with a long one.

Diagnose failures before choosing

Suppose you are building an admissions assistant for an Indore institute. A base model with a basic prompt gets 62 percent of your test questions right. Do not guess what to do next. Take the wrong answers, read each one, and label why it failed:

• Knowledge gap: the answer depends on information the model never saw, such as this month's batch dates. Fix with RAG.
• Instruction gap: the model could do it but misunderstood what you wanted, such as the wrong length or a missing field. Fix with prompting.
• Behaviour gap: you explained clearly, with examples, and the model still drifts from your format, tone or labelling scheme on a good share of inputs. Fix with fine-tuning.
• Capability gap: the task is too hard for this model even with perfect context, such as multi-step fee calculations with exceptions. Fix with a stronger model or reasoning techniques.
• Retrieval gap: RAG is in place but fetched the wrong passage. Fix the retriever, not the model.

Then count. The biggest bucket tells you where to spend next:

from collections import Counter

failures = [  # (question, why it failed) from reading 20 wrong answers
    ("Next weekend batch date?", "knowledge"), ("Fee for the online track?", "knowledge"),
    ("Reply was 300 words, asked for 60", "instruction"), ("Missing 'next step' line", "instruction"),
    ("Tone too casual for parents again", "behaviour"), ("Labelled 'fees' ticket as 'other'", "behaviour"),
    ("Labelled 'timings' ticket as 'other'", "behaviour"), ("Wrong EMI arithmetic", "capability"),
    ("Retrieved last year's brochure", "retrieval"), ("Is there a Hindi batch?", "knowledge"),
    ("Old address in the answer", "knowledge"), ("Asked for bullets, got a paragraph", "instruction"),
    ("Labelled 'placement' ticket as 'fees'", "behaviour"), ("Demo class timing?", "knowledge"),
    ("Retrieved wrong course page", "retrieval"), ("Hostel facilities?", "knowledge"),
    ("Mixed up two batch codes", "behaviour"), ("Exam date for module 3?", "knowledge"),
    ("No greeting in Hindi reply", "instruction"), ("Seats left in Data Analytics?", "knowledge"),
]
FIX = {"knowledge": "add RAG over current documents", "instruction": "rewrite the prompt, add examples",
       "behaviour": "consider fine-tuning after a prompt retry", "capability": "try a stronger model",
       "retrieval": "improve chunking, search and re-ranking"}

counts = Counter(reason for _, reason in failures)
for reason, n in counts.most_common():
    print(f"{reason:<12} {n:>2} ({n / len(failures):.0%})  -> {FIX[reason]}")

Here knowledge gaps dominate, so RAG comes first. Behaviour gaps are the second-largest group, but you should retry them with a better prompt and examples before you train. Most diagnoses end up as "RAG plus a better prompt", and fine-tuning becomes a later step for whatever is left.

One loop: label failures, fix the biggest bucket with the cheapest tool, re-measure.

The cost question: per request vs per month

Fine-tuning has a reputation for being expensive, and its upfront cost is high. But a long few-shot prompt is paid for on every request, forever. At high volume, a small fine-tuned model can be the cheaper option overall.

This calculator uses illustrative prices in rupees per million tokens. Replace them with your provider's real rates:

requests_per_month = 300_000
options = {
    # name: (price per 1M input tokens, per 1M output, input tokens per request, output tokens, one-off cost)
    "large model + long few-shot prompt": (250, 1000, 2500, 150, 0),
    "large model + RAG (short prompt)":   (250, 1000, 1800, 150, 40_000),
    "small fine-tuned model + RAG":       (25,  100,  1000, 150, 120_000),
}

def monthly_cost(price_in, price_out, tok_in, tok_out):
    per_request = (tok_in * price_in + tok_out * price_out) / 1_000_000
    return per_request * requests_per_month

baseline = None
for name, (p_in, p_out, t_in, t_out, setup) in options.items():
    month = monthly_cost(p_in, p_out, t_in, t_out)
    baseline = baseline or month
    saving = baseline - month
    payback = f"{setup / saving:.1f} months" if saving > 0 else "n/a"
    print(f"{name:<36} Rs {month:>9,.0f}/month  setup Rs {setup:>7,}  payback {payback}")

At 300,000 requests a month, the fine-tuned option repays its setup cost within weeks. Drop requests_per_month to 10,000 and payback stretches past a year, which may be longer than your base model stays current. Volume decides. Remember to include the hidden costs: someone must maintain the training data, re-run evaluation and retrain when the base model is upgraded.

Combining prompting, RAG and fine-tuning

The choice is rarely either-or. Common combinations:

• Prompting + RAG: the default for question answering over company documents. Most projects stop here.
• Fine-tuning + RAG: a small model fine-tuned to answer from retrieved passages in your format, with RAG supplying fresh facts. This gives low cost per request and current knowledge.
• Fine-tuned embeddings: sometimes the retriever is the weak link, and fine-tuning the embedding model on your queries helps more than tuning the LLM. Embeddings explained covers what that model does.
• Long context instead of RAG: when your whole knowledge base is small and stable, loading it into the context and caching it can replace retrieval. RAG vs CAG compares the two.

A practical order of work

Follow these steps on a new project:

1. Write a test set of 50 to 100 real questions with expected answers.
2. Measure a strong model with a careful prompt. This is your baseline.
3. Label the failures and fix the largest bucket with the cheapest suitable tool.
4. Add RAG if knowledge gaps appear, and evaluate retrieval separately.
5. Consider fine-tuning only for behaviour gaps that survive steps 3 and 4, or when the cost calculator says a smaller model would pay off. Your first job then is creating an instruction fine-tuning dataset.
6. Re-run the test set after every change and keep the results.

The test set is what makes every decision on this page possible. LLM evaluation methods shows how to score answers once you have one.

Common mistakes when choosing an approach

• Fine-tuning to add facts. Weights are a poor database; facts go stale and still get mixed up.
• Skipping the baseline. Without a measured prompt-only score, you cannot prove that RAG or fine-tuning helped.
• Blaming the model for retrieval errors. Check whether the right passage was retrieved before changing anything else.
• Comparing on price per token alone. Compare monthly cost including prompt length, volume and maintenance.
• Fine-tuning before the prompt is good. You will bake a weak instruction into weights.

Key takeaways

• Prompting changes instructions, RAG changes information, fine-tuning changes weights.
• Label real failures before choosing: knowledge, instruction, behaviour, capability or retrieval.
• Fix the biggest bucket with the cheapest tool, then re-measure.
• Compare monthly cost, not price per token; volume decides when fine-tuning pays off.
• Combinations, especially RAG with a good prompt, are the norm in production.