Fine-tuning vs RAG: a decision procedure, not a debate

The framing is wrong. These solve different problems and compose well.

Diagnose first#

Ask what the model is getting wrong.

It does not know something. That is a knowledge gap — retrieval. Facts change; weights are expensive to change with them.

It knows, but responds badly. Wrong format, wrong tone, ignoring your schema, too verbose. That is behaviour — fine-tuning, and often a few hundred examples is enough.

It is correct but too slow or expensive. Distil a smaller model on the larger one's outputs.

The combination most production systems land on#

Retrieval for knowledge, a small fine-tune for format and refusal behaviour, and a distilled model for the high-volume path. Each addresses a different failure.

What fine-tuning will not fix#

It will not add facts reliably. Training on documents teaches style and pattern far more effectively than it teaches recall, and what it does teach it teaches without a citation you can verify.