When to fine-tune an LLM instead of prompting or RAG

Fine-tuning is the answer people reach for when a model isn't behaving, and often the wrong first move. Prompting and retrieval are cheaper, faster to iterate, and don't lock you into a training pipeline. Before we recommend a fine-tune, we walk through a specific decision. Sometimes the honest answer is: don't.

Try the cheap levers first

If the task is essentially knowledge lookup, retrieval-augmented generation usually gets you there without touching the model's weights. If it's about behaviour or format, better prompting and few-shot examples often close most of the gap. These are hours of work, not weeks, and they're reversible. We exhaust them before proposing anything heavier.

When fine-tuning genuinely pays off

Fine-tuning earns its keep when the need is structural, not informational:

• Consistent format: you need the same rigid output shape every time, and prompting keeps drifting.
• Domain language: the model has to speak your field's vocabulary and conventions fluently.
• Latency and cost at scale: a smaller tuned model can match a big general one on your narrow task, far cheaper per call.
• Privacy and control: you want a self-hosted model on your own infrastructure, not an API.

The prerequisite people skip: data and evals

Fine-tuning is only as good as the dataset behind it, and only trustworthy if you can measure the result. Before tuning we assess whether you have enough representative examples, and we build an evaluation set so we can prove the tuned model actually beats the baseline. No eval, no fine-tune: otherwise you're shipping a model on faith.

The short version

Prompt first, retrieve second, fine-tune when the need is format, domain, latency, cost, or privacy, and only with data and evals to back it. We walk clients through exactly this at ChitraYantra, and we'll tell you when tuning isn't worth it.