RAG vs Fine-Tuning: Architecture for Production AI

Start with the failure you need to fix

An assistant gives an outdated policy answer, produces inconsistent JSON and exposes a document from the wrong customer account. These are three different defects. Adding a vector database or training an adapter does not address all three.

RAG changes the evidence supplied at inference time. Fine-tuning changes model behaviour through parameter updates or trainable adapters. Authorisation determines which evidence and actions a caller may access. A production design has to account for all three independently.

The decision framework below is an engineering proposal illustrated with a hypothetical customer-support assistant. It does not report client results or claim a universal accuracy improvement.

Choose the source of truth explicitly

A request for the current contract terms should resolve against an approved document version. A request for an order's present delivery status should query the operational system. A requirement to classify messages into an organisation's taxonomy may justify adapting the model on representative labelled examples.

Microsoft's RAG guidance distinguishes grounding in private or changing data from adapting behaviour through fine-tuning. For our support example, the contract corpus can support retrieval, while the order service remains the authority for live order state. An index refresh interval is part of the freshness contract; a document does not become searchable instantly because a file changed.

Write that contract before selecting infrastructure: which system owns each fact, how old a response may be, and what happens when the source cannot be reached. This prevents a technically sophisticated pipeline from answering the wrong version of the question.

Design retrieval around complete evidence

The original RAG research combines parametric generation with retrieved information. In an application, useful evidence may be spread across a section heading, a table, an exception and a document's effective date. Splitting solely by an arbitrary token count can separate the rule from its conditions.

Preserve document IDs, section paths, version identifiers and source locations when preparing chunks. For the hypothetical contract assistant, a candidate record could look like this:

{
  "document_id": "contract-482",
  "version": "approved-7",
  "section": "Cancellation / Exceptions",
  "effective_from": "2026-09-01",
  "tenant_id": "customer-17",
  "text": "...",
  "source_location": "page 8, section 4.2"
}

These field names are illustrative. Their purpose is to make eligibility, provenance and freshness inspectable outside the model. A generated source link should resolve to the version used to support the claim, subject to the reader's permissions.

Combine search signals, then measure their contribution

Exact identifiers, product codes and error strings can require lexical matching. Semantic retrieval helps when a user describes the same concept with different wording. A hybrid candidate set followed by reranking is a useful design to evaluate, rather than a guaranteed improvement.

Anthropic's Contextual Retrieval work explores adding chunk-specific context and combining retrieval with reranking. Treat its findings as evidence about the studied setup. Reproduce comparisons on your corpus before adopting the same configuration.

For the support assistant, compare lexical-only, vector-only and hybrid retrieval with the same authorised question set. Record whether the evidence needed for a complete answer appears in the retained results. Then inspect whether a reranker moves relevant exceptions above attractive but incomplete summaries. More candidates also add latency and operational cost.

Long context still needs selection and version control

A model with a large input window can accept more material, but acceptance does not establish that every detail will be used reliably. Lost in the Middle demonstrated position-sensitive retrieval performance in the models and tasks it evaluated. It is a reason to test evidence placement, not a timeless ranking of today's models.

An illustrative regression test can place the same exception near the beginning, middle and end of a document bundle. Add an obsolete version as a distractor. Score whether the assistant uses the currently applicable rule and identifies the supporting passage. A model upgrade should rerun this test rather than inherit a previous result.

Keep access control before the model boundary

A model instruction saying “do not reveal another customer's documents” is insufficient access control. The retrieval service must restrict eligible documents using the authenticated caller's permissions before their content enters model context. The application also needs to enforce permissions when opening citations and calling operational tools.

Azure AI Search documents document-level access patterns, including application-supplied security filters. Some native integrations are preview features; verify their current support before depending on them in production. The architectural principle is broader than any particular search service.

For the hypothetical assistant, an access test should include two tenants with similar contracts, revoked permissions, inherited group membership and stale index metadata. Keep tenant and permission scope in cache keys wherever cached content or answers depend on that scope.

Fine-tune a measured behavioural gap

Consider fine-tuning when a model repeatedly misapplies a stable classification scheme, fails a domain-specific transformation or requires consistent response behaviour that prompting has not achieved. Training examples must represent the target task, and evaluation examples must be held out from training.

LoRA is a parameter-efficient adaptation method: the base weights are frozen while low-rank updates are trained. Lower adaptation cost does not remove the need for data quality, versioning or regression tests. Neither an adapter nor full fine-tuning inherently supplies current permissions or a live operational database.

A combined design can retrieve approved contract evidence and use an adapted model to produce the required response format. Evaluate the combination against the unadapted retrieval baseline. If retrieval is failing, measure that defect directly before attributing the error to the generator.

Evaluate the pipeline in layers

Create an evaluation set that records the question, authorised scope, applicable document version, required evidence and expected behaviour. Include questions that cannot be answered from the available sources. Label ambiguous cases before comparing systems.

• Retrieval: did the retained evidence contain the complete, applicable answer?

• Generation: are the answer's claims supported, and do the cited passages support those claims?

• Behaviour: does the output satisfy the task and schema, including an appropriate refusal or clarification?

• Access: did any unauthorised content enter retrieval results, context, caches or the response?

• Operations: what are p95 latency, ingestion lag and cost per accepted answer?

This separation creates actionable failure reports. “Wrong answer” becomes “retrieval returned the obsolete exception” or “the generator ignored the correct passage.” The next change then has a defined target.

For compute and context costs, read how LLM inference works. If the assistant will execute actions, continue with agent execution and evaluation.