Stop Fine-Tuning Your LLM: When to Use RAG, Context, and Last-Resort Optimisation

A custom-trained language model can look like the serious option: collect your company data, fine-tune a model, deploy it and gain a permanent competitive advantage. In practice, that plan can leave a business with a large bill, a long implementation cycle and a model that is overtaken by the next generation of general-purpose systems before the work is finished.

Fine-tuning still has a place. It can help with strict latency requirements, model distillation and some reinforcement-learning use cases. But it is no longer the default answer when a business wants an AI system to understand its documents or follow a particular workflow.

For most teams, the better starting point is simpler: retrieve the right information at query time, provide the model with well-structured context, and give it explicit skills or tools for the work it needs to perform.

The custom-model assumption is becoming harder to defend

Fine-tuning changes a model's weights by continuing training on a focused dataset. That can improve performance on a particular task, style or format. The trade-off is that the resulting model becomes a moving target.

A model trained today is competing with frontier models that may improve during the months required to prepare data, run training, evaluate results and integrate the system into production. The more specialised the custom model, the more carefully the business must ask whether the advantage will survive the next general-purpose model release.

The source highlights two useful examples. In 2023, a legal AI company reportedly fine-tuned a model that attorneys preferred to GPT-4 in blind tests 97% of the time. That was a meaningful result for the task and time period. But the source also notes that later benchmarking found several general-purpose models outperforming that custom model. The lesson is not that fine-tuning cannot work. It is that a lead over a frontier model may be temporary.

BloombergGPT illustrates the same risk at a larger scale. Bloomberg trained a model from scratch using financial data and expertise, but later evaluations found GPT-4 and ChatGPT performing better on many financial benchmarks. Large training programmes do not automatically create durable superiority.

This matters commercially. A business considering a custom model should include the cost of obsolescence in its decision. That includes engineering time, evaluation work, hosting, monitoring, retraining and the opportunity cost of delaying a useful workflow while the team builds infrastructure.

Fine-tuning, RAG and agent skills solve different problems

The most useful question is not “Which technique is most advanced?” It is “What exactly needs to change?”

Use RAG when the model needs to know changing information

Retrieval-Augmented Generation, or RAG, leaves the model's weights unchanged. When a user asks a question, the system searches an approved collection of documents, inserts relevant passages into the prompt and asks the model to answer using that context.

This is usually the right fit for information that changes regularly, such as:

• Product specifications and pricing rules
• Internal policies and operating procedures
• Client contracts and account records
• Support documentation
• Compliance guidance
• Sales enablement material

For example, suppose a sales team wants an assistant that answers questions about service packages. Fine-tuning could teach the model patterns from old sales documents, but it would not provide a clean way to update a price or remove an obsolete offer. A RAG workflow can retrieve the current pricing sheet and show the answer's supporting source.

A practical RAG workflow looks like this:

1. Define which documents are authoritative.
2. Remove duplicates, outdated versions and access-sensitive material.
3. Split documents into meaningful sections rather than arbitrary blocks.
4. Add metadata such as product, region, date, owner and permission level.
5. Retrieve a small set of relevant passages for each question.
6. Instruct the model to answer from those passages and state when the evidence is insufficient.
7. Test the system against real questions, including questions it should refuse to answer.

The growing context windows of modern models also reduce the pressure to fine-tune simply to make more information available. The source contrasts GPT-3's roughly 2,000-token context window with current models offering more than one million tokens. Larger context does not remove the need for retrieval or careful document selection, but it makes direct context strategies more practical than they once were.

Use context engineering when the model needs better instructions and structure

Context engineering is the deliberate design of everything the model sees at the moment it works: instructions, retrieved material, user history, examples, constraints, tool results and output formats.

Consider an operations assistant that reviews a new customer request. Its performance may improve more from a clear workflow than from training on thousands of examples. The context could tell it to:

• Identify the customer's objective
• Check eligibility against the current policy
• Ask for missing information
• Calculate the relevant fields using an approved tool
• Produce a draft response in a defined structure
• Escalate exceptions rather than guessing

This changes how the model behaves without changing its underlying weights. It also makes the workflow easier to inspect and revise. If the policy changes, the team can update the retrieved source or instruction instead of launching another training run.

Context has costs and limits. Long prompts can increase latency and usage costs, and irrelevant material can make an answer worse. More context is not automatically better context. Teams should measure retrieval quality, prompt length, response time and answer accuracy together.

Use agent skills when the model needs to act differently

An agent skill is a reusable capability that defines how a system performs a task. It may include instructions, decision rules, tools, validation steps and an expected output format.

For a finance operations team, a “supplier invoice review” skill might:

1. Read the invoice fields.
2. Match the supplier against the approved vendor list.
3. Check the purchase order.
4. Flag discrepancies above a defined threshold.
5. Draft an approval note.
6. Send exceptions to a human reviewer.

Fine-tuning is not necessary merely because the business wants this sequence to happen consistently. The sequence can be expressed as workflow logic, tool access and structured instructions. Fine-tuning becomes more relevant only if repeated evaluation shows that the model still fails in a way that other controls cannot fix.

Use LoRA when fine-tuning is justified but full training is excessive

LoRA, or Low-Rank Adaptation, is a parameter-efficient fine-tuning method. Instead of updating all the weights in a base model, it trains smaller adapter layers while leaving the base model largely frozen.

That can reduce training and storage requirements and make it easier to maintain multiple specialised adapters. It does not remove the central risks, however. The team still needs high-quality training data, a reliable evaluation set, deployment controls and a plan for what happens when the base model changes.

LoRA is an implementation option within the fine-tuning category, not evidence that fine-tuning is automatically the right strategy.

A decision framework for business teams

Start with the desired change, not the available technique.

1. Does the system need current business knowledge?

If yes, begin with RAG or another controlled retrieval approach. Examples include current product data, policies, legal documents and customer records.

Do not fine-tune solely to add information that will change. Information embedded in model weights is harder to update, audit and remove.

2. Does the system need a repeatable process?

If yes, design the workflow using context engineering, agent skills, tools and validation rules. Define what the system should do, what it must not do and when a person must take over.

For example, a recruitment assistant may classify applications and draft questions, but the workflow can prohibit it from making final hiring decisions. That control belongs in the process design, not only in a training dataset.

3. Is the problem a measurable behaviour that persists after better prompting and workflow design?

If the model consistently produces the wrong structure, tone or classification despite strong examples, clear instructions and good context, fine-tuning may be worth testing. Build a representative evaluation set first. Do not train because the system feels imperfect; identify a repeatable failure pattern.

4. Is latency or model size the real constraint?

Fine-tuning and distillation can be valuable when a business needs a smaller, faster or cheaper model for a high-volume workflow. A large model might produce excellent results but be too slow for real-time customer support or too expensive for processing millions of routine records.

In that case, the objective is not to give the model more company knowledge. It is to transfer the required capability into a more efficient model and verify that quality remains acceptable.

5. Can the team maintain the system?

Before approving custom training, assign ownership for data preparation, evaluation, monitoring, retraining and rollback. If nobody owns those activities, a custom model is likely to become an unmanaged dependency.

A simple decision table can help:

Business requirement	First option	Why
Answer questions using changing documents	RAG	Knowledge can be updated without retraining
Follow a multi-step operating procedure	Agent skill and tools	Behaviour is expressed as a controlled workflow
Produce a consistent format or classification	Prompting, examples and evaluation; then fine-tuning if needed	Test whether the behaviour problem is persistent
Respond under strict latency or cost limits	Smaller model, distillation or fine-tuning	Optimises serving efficiency
Learn from reward signals in a specialised task	Reinforcement fine-tuning, where justified	Targets behaviour through evaluated outcomes

Decision rule: If the problem is “the model does not know this,” retrieve the information. If the problem is “the model does not follow this process,” design the skill and context. If the problem is “the model is too slow, too large or persistently inconsistent,” investigate fine-tuning.

Failure modes to plan for

The case against unnecessary fine-tuning should not become a case for simplistic RAG.

RAG can retrieve the wrong evidence. Poor chunking, weak metadata, duplicate documents and vague queries can produce irrelevant context. The model may then give a confident answer based on incomplete material. Test retrieval separately from generation.

RAG is not a substitute for reasoning. A system may retrieve every relevant document and still struggle to compare clauses, trace dependencies or reason across a large unstructured corpus. Complex tasks may require better document structure, multi-step retrieval, code or human review.

Long context can create cost and latency problems. Sending large volumes of material on every request may be technically possible but commercially inefficient. Use filtering, summaries and targeted retrieval where appropriate.

Fine-tuning can memorise bad practice. If the training set contains outdated policies, inconsistent decisions or undocumented exceptions, the model may reproduce those defects more consistently.

Benchmark results can age quickly. The source includes an unverified claim about a 2025 benchmark in which seven general-purpose models surpassed a particular legal AI company's custom model. That result should be independently verified before being used in a business case. More broadly, every comparison should record the model versions, task definitions, evaluation data and date.

Custom models create maintenance obligations. A fine-tuned model may need to be retrained or re-evaluated when the base model, business rules, customer expectations or regulatory environment changes. Those ongoing costs belong in the original proposal.

A practical pilot before training

A business can test the decision without committing to a custom model.

Choose one narrow workflow, such as answering product questions for an internal sales team. Collect 30 to 50 representative questions, including ambiguous and unanswerable cases. Establish a baseline using a general-purpose model with clear instructions. Then add an approved document set through RAG and define an agent skill for the response process.

Measure:

• Answer accuracy against the approved source
• Correct refusal when evidence is missing
• Retrieval relevance
• Response time
• Cost per task
• Human correction rate
• Escalation quality

Only after this baseline should the team test fine-tuning. If fine-tuning does not improve a defined metric enough to justify its operational cost, it is not solving the business problem.

The broader principle is straightforward: customise the workflow before customising the model. General-purpose models are improving rapidly, while retrieval, tools and structured context can be updated as the business changes. Fine-tuning remains useful, but it should earn its place through a measurable requirement rather than through the assumption that proprietary data automatically belongs in training.