What fine-tuning is — and what prompting and RAG handle instead
Fine-tuning takes a model that has already been trained and trains it further on a smaller, purpose-specific dataset, changing the weights themselves. Two cheaper options are routinely confused with it: prompt engineering, which changes only the instruction, and RAG, which leaves the model alone and feeds it the documents it needs when answering. The dividing rule is simple — use RAG for what the model does not know, fine-tuning for how it behaves, and prompting for everything else. Methods like LoRA train only a small added set of parameters
The three lines
- Definition — continued training of an already-trained model on task data, changing its weights
- Division — RAG for missing knowledge, fine-tuning for tone, format and behaviour, prompting for the rest
- Order — cost runs prompting < RAG < fine-tuning. Working down from the top is the sound approach
Key questions
- What does fine-tuning actually do?
- It **takes a model whose training is already complete and trains it further on a smaller dataset**. The essential point is that **the model's weights actually change**. However long a prompt you write, the model stays the same; fine-tune it and **you have a different model**. The scale differs from training from scratch by orders of magnitude: a base model consumes hundreds of billions to trillions of tokens on a large GPU cluster, while fine-tuning can produce meaningful change from **hundreds to tens of thousands of examples**. Current practice mostly avoids retraining everything — **LoRA and QLoRA freeze nearly all existing parameters and train only a small added set**, because retraining every parameter of a large model has become impractical.
- Should I use RAG or fine-tuning?
- It depends on whether you are changing **what the model knows** or **how it behaves**. If the answer needs information the model does not have — internal policy, today's price list, recent news — the answer is **RAG**. Documents are retrieved and inserted at answer time, so when the material changes you **swap the documents and leave the model alone**. If the problem is not missing information but **tone, output format or judgement criteria that do not match yours**, that is fine-tuning. The most common mistake is **fine-tuning to inject current information**: it works, but every update to the source data requires another training run. In practice the two are often combined — fine-tuning fixes the shape of the output while RAG supplies the content.
- Does fine-tuning make answers more accurate?
- **Not reliably, where facts are concerned.** What fine-tuning improves clearly is **format, style and task behaviour**; factual accuracy is a weaker case. The reason lies in the training itself — fine-tuning teaches the pattern "for input like this, produce output like that," not a stored set of facts. So when the model meets a question outside its fine-tuning data, **the risk of a confident, well-formatted invention can actually increase**, and good formatting makes people more likely to believe it. For factual accuracy, **RAG is structurally better placed**: the supporting document travels with the answer, so sources can be cited and a wrong answer can be traced to the document that caused it. This brief covers the root of that risk in "What AI hallucination is — why models are trained not to say they don't know."
Fine-tuning trains an already-trained model further.
It is not the same as changing the prompt. The model itself changes.
And in practice, it is the right answer less often than people expect.
1. Three options
The same problem — the model does not give the answer you want — has three standard remedies.
| Method | What changes | Model weights |
|---|---|---|
| Prompt engineering | The instruction | Unchanged |
| RAG (retrieval-augmented generation) | The material supplied at answer time | Unchanged |
| Fine-tuning | The model itself | Changed |
Only the third touches the model. The first two work outside it.
The ordering matters because cost and reversibility both worsen as you go down. A prompt takes minutes to change. RAG means swapping a document. Fine-tuning means running training again.
2. How to choose
One question decides it: are you changing what the model knows or how it behaves?
| Problem | Right method | Why |
|---|---|---|
| Doesn't know internal policy | RAG | Hand it the document |
| Doesn't know today's prices | RAG | The data changes daily |
| Tone isn't ours | Fine-tuning | Style is fixed by training |
| Output format varies | Fine-tuning | Lock the pattern in |
| Misreads instructions | Prompting | Rewrite the instruction |
| Does well when shown examples | Prompting | Put examples in the prompt |
The commonest error is fine-tuning in order to inject current information. It is technically possible. The problem is that every change to the underlying data requires another training run. Fine-tune a weekly price list and you pay for training weekly. Do the same job with RAG and it is one document replaced.
3. What fine-tuning looks like in practice
| Base model training | Fine-tuning | |
|---|---|---|
| Data | Hundreds of billions to trillions of tokens | Hundreds to tens of thousands of examples |
| Compute | Large GPU cluster | Far smaller |
| Result | A new model | A variant of an existing model |
Modern practice mostly does not retrain everything. LoRA and QLoRA freeze nearly all existing parameters and train only a small added set — a compromise that appeared because retraining every parameter of a large model became impractical.
A small training target means two things. Cost falls, and the original model's abilities are damaged less. Full fine-tuning is known to improve the target task at the expense of things the model previously did well.
4. What fine-tuning is bad at
This is the most misunderstood point. Fine-tuning is not a procedure for inserting knowledge.
| Improves | Does not reliably improve |
|---|---|
| Output format | Factual accuracy |
| Tone and style | Current information |
| Task behaviour | Source attribution |
| Domain conventions of expression | Traceability when wrong |
The reason sits in the nature of the training. Fine-tuning teaches the pattern "for input like this, produce output like that." It does not store facts the way a dictionary does.
Which produces a specific side effect: a fine-tuned model asked something outside its training data still answers confidently, in the learned voice. The format is right and the content is wrong — and because the format is right, people believe it more readily.
If factual accuracy is the goal, RAG is structurally better suited. The supporting document accompanies the answer, so sources can be cited and a wrong answer can be traced to the document that caused it. This brief covers the underlying risk in "What AI hallucination is — why models are trained not to say they don't know."
5. In practice they are layered
Treating this as a choice among three tends not to resolve. Real systems stack them.
| Layer | Responsibility |
|---|---|
| Fine-tuning | Fixes output format, style and domain conventions |
| RAG | Supplies the facts needed for this answer |
| Prompting | Carries the specific instruction for this request |
The recommended order of adoption runs bottom-up, not top-down. Try prompting. If that fails, add RAG. If what remains is a problem of format or style, then consider fine-tuning.
Starting with fine-tuning means applying the most expensive remedy before diagnosing the problem. Once a training run has been done, it is no longer possible to tell whether a better prompt or a missing document would have solved it.
The same ordering applies to building agents. This brief covers tool and permission design in "What an AI agent is — what changes when you hand a model tools," and the recurring finding there is that agents usually fail on tool descriptions and permission scope rather than model capability — another problem fine-tuning does not solve.
6. What is not settled
- Data volume required — "hundreds to a few thousand" comes from introductory material and varies widely.
- Factual accuracy gains — no quantitative comparison established; the description is directional.
- Cost ordering — prompting < RAG < fine-tuning is conventional, and can reverse at very high call volumes.
- LoRA performance loss — reports vary by task and were not checked against the original papers.
- Layering — common practice, not a standardised methodology.
- Related briefs — "AI training vs inference — why the same GPU gets used twice" covers the compute structure; "What AI token pricing is" covers per-call cost.
Sources
- Rsupport — RAG vs fine-tuning vs prompting: which is right for your company's AI
- A-Cloud Blog — Comparing prompt engineering, RAG and fine-tuning
- Vizensoft — Fine-tuning vs RAG vs prompt engineering compared
- e4ds News — Raising AI to the level you want: fine-tuning and RAG
- TreeSoop — LLM fine-tuning vs RAG guide 2026: decision matrix, cost and accuracy