At least once a month, a prospective client tells us they need to fine-tune a model on their company data. When we ask what problem they're solving, the answer is almost always some version of: 'We want the AI to know our documentation, our policies, our products.' That's a retrieval problem, not a fine-tuning problem — and the difference is worth tens of thousands of dollars and months of schedule.
What each one actually does
Retrieval-augmented generation (RAG) gives a model access to your knowledge at answer time. Your documents are indexed in a vector database; when a question arrives, the most relevant passages are retrieved and handed to the model as context. The model 'knows' your content the way a sharp consultant knows the binder you just handed them — and it can cite exactly where each answer came from.
Fine-tuning, by contrast, changes the model's weights. It's for teaching behavior, not facts: a specific output format, a house style, a specialized task the base model performs poorly. Fine-tuning on your knowledge base is a poor way to make a model factual about your business — the knowledge blurs together, goes stale the day your docs change, and can't cite sources.
The decision test we use
Ask one question: when your content changes, should the system's answers change too? If yes — policies, product docs, prices, procedures — you want RAG, because retrieval reflects updates the moment they're indexed. A fine-tuned model would need retraining every time your knowledge moves.
Reach for fine-tuning only when you can name a behavior the base model gets wrong after serious prompt engineering: a strict output schema it keeps breaking, domain shorthand it misreads, a tone it can't hold. Even then, we usually run RAG underneath and fine-tune on top.
What this means for your budget
A production RAG pilot — one process, evaluation harness, human-in-the-loop controls — typically ships in four to eight weeks. Fine-tuning adds data preparation, training runs, evaluation, and a retraining pipeline you'll maintain forever. That's justified when behavior is genuinely the problem. It's an expensive detour when what you actually needed was a good retrieval index.
The honest summary: start with RAG, measure, and let the gaps — not the hype — tell you whether fine-tuning earns its cost.