8 Techniques to Reduce LLM Hallucination in Production

Ground responses in trusted documents, databases, and enterprise knowledge instead of model memory alone.

A model hallucinates most confidently in exactly the gap between what it memorized during training and what the question actually needs — a policy that changed last quarter, a product that shipped after the training cutoff, an internal document it never saw at all. Retrieval-augmented generation closes that gap by fetching relevant passages from a trusted corpus at query time — typically via embedding similarity search over a vector database — and inserting them directly into the prompt as context.

This doesn't make hallucination impossible; it changes its shape. Instead of the model reconstructing a fact from parametric memory, it's now summarizing or reasoning over a passage sitting right in front of it, which is a task LLMs are considerably more reliable at. The failure mode shifts from "the model doesn't know" to "the model wasn't given the right passage" — still a real problem, but a far more debuggable one, since you can inspect exactly what the model saw.

Use clear instructions, constraints, and few-shot examples to reduce ambiguity and unsupported claims.

An ambiguous instruction leaves room the model will fill — usually with something plausible-sounding rather than something true. "Summarize this contract" invites more liberty than "summarize this contract using only clauses present in the text below, and state 'not specified' for anything the text doesn't cover." The second version removes an entire category of hallucination simply by naming the boundary explicitly.

Three moves do most of the work: give the model explicit permission to say it doesn't know (models rarely volunteer uncertainty unless the prompt tells them that's an acceptable answer); constrain the source of truth ("answer only from the provided context"); and show, not just tell, via one or two few-shot examples of the exact answer shape you want, including an example of a correct "not found" response.

# vague — invites invention
"Answer the user's question about our refund policy."

# grounded — names the boundary and the escape hatch
"Answer using ONLY the refund-policy excerpt below.
 If the excerpt doesn't cover the question, respond
 exactly: 'Not covered in the current policy document.'
 Do not use outside knowledge."

Retrieve smaller passages, then prioritize the context most relevant to the actual question.

Embedding search is fast but coarse: it ranks passages by how similar their meaning is to the query, which isn't quite the same as how well they answer it. A passage about "return windows" and a passage about "return shipping costs" can sit close together in embedding space while only one of them actually answers a specific question about international returns. Retrieve wide — the top 20 to 50 candidates — then run a second-stage reranker, typically a cross-encoder that scores the query and each passage jointly, and keep only the top 3 to 5 for the model's context window.

Chunk boundaries matter just as much as ranking. A fact split across two chunks (a number in one, its unit or condition in the next) can silently disappear from whichever chunk the reranker doesn't surface. Overlapping chunks, and chunking along natural document boundaries (sections, list items) rather than fixed character counts, catch a surprising share of otherwise invisible retrieval failures.

Use JSON schemas, templates, and validators to make responses predictable and easier to verify.

Free-form prose is hard to check programmatically; a schema turns an answer into fields you can validate mechanically. Requiring a source_id field that must reference an actual retrieved chunk ID doesn't stop the model from wanting to hallucinate a citation — but it does mean an invented one fails validation instead of quietly reaching the user. Constrained decoding (grammar-based generation that only allows schema-valid tokens) or a validate-and-retry loop both work; the point is that the output's shape becomes a checkpoint, not just a formatting preference.

{
  "type": "object",
  "properties": {
    "answer":     { "type": "string" },
    "source_ids": {
      "type": "array",
      "items": { "type": "string" },
      "minItems": 1
    },
    "confidence": { "type": "string", "enum": ["high", "low"] }
  },
  "required": ["answer", "source_ids", "confidence"]
}
// a source_id that doesn't match a retrieved chunk
// fails validation before the answer ever ships

Let the model query search engines, databases, calculators, or APIs when facts need external validation.

Arithmetic, unit conversion, current prices, live system status — these are precisely the categories where a language model will produce a confident, specific-looking number that's simply wrong, because it's pattern-matching against training data rather than computing an answer. Tool calling routes these sub-tasks to something that's actually authoritative: a calculator for math, a live API for a stock price, a database query for an account balance. The model's job narrows to deciding which tool to call and how to phrase the final answer around the result — not to generating the fact itself.

user_query = "What's 18% VAT on a £2,450 invoice, in USD?"

# the model should call two tools, not compute either number itself
vat_amount = calculator.evaluate("2450 * 0.18")      # -> 441.00 GBP
fx_rate    = fx_api.get_rate("GBP", "USD")           # -> live rate
final      = calculator.evaluate(f"{vat_amount} * {fx_rate}")

Add fact-checking, verification, or correction steps before the final answer reaches the user.

Even with good retrieval, generation can drift: a model summarizing three retrieved passages will occasionally merge details across them, soften a qualifier, or state a number slightly differently than the source did. A verification pass — sometimes called chain-of-verification or self-critique — runs a second model call (or the same model with a different prompt) that checks each claim in the draft answer against the specific source passages it cited, sentence by sentence, and flags or rewrites anything unsupported.

This roughly doubles inference cost and latency for the queries it runs on, so it's usually applied selectively: to answers heading to end users in regulated or high-stakes domains, to anything the confidence gate (#8) already flagged as uncertain, or to a sampled percentage of traffic for ongoing quality monitoring rather than every single request.

Improve domain behavior and terminology using curated data, while still grounding changing facts externally.

Fine-tuning is easy to reach for as a fix-all and a poor tool for exactly the problem people usually want it to solve. It's genuinely effective at teaching a model your domain's vocabulary, your preferred answer format, and reasoning patterns specific to your task — things that are stable over time. It's a poor way to keep a model current on facts that change, because every update requires a retraining cycle, and the model has no mechanism to "forget" a stale fact it was fine-tuned on before a newer one is trained in.

The practical split: fine-tune for how the model should behave — tone, structure, domain reasoning — and retrieve for what the model needs to know. Treating fine-tuning as a substitute for retrieval, rather than a complement to it, is one of the more common ways teams reintroduce hallucination after believing they'd fixed it.

Escalate uncertain, sensitive, or high-impact outputs for human approval instead of shipping every answer straight through.

Not every answer deserves the same amount of scrutiny, and not every system can afford to verify every answer at the depth described in #6. A confidence signal — self-consistency across several sampled generations, the token-level log-probability of the response, or a lightweight classifier trained on a labeled set of past correct and incorrect answers — gives you a cheap way to sort the easy cases from the ones that need a closer look.

Route low-confidence outputs, and anything touching money, medical, legal, or safety-sensitive territory regardless of confidence, to a human review queue before it reaches the end user. Everything else ships straight through. This is the step that turns "the model is usually right" into a system with a bounded worst case — which is the actual reliability guarantee most production deployments need.

samples = [llm.generate(prompt) for _ in range(5)]
agreement = fraction_matching(samples)  # self-consistency

if agreement < 0.6 or answer.touches_sensitive_domain():
    queue_for_human_review(answer)
else:
    return answer  # ships straight to the user

Stacking the techniques

These eight don't compete for the same job; each addresses a different failure mode, at a different cost. A practical starting stack for most production RAG systems is retrieval + reranking + prompt grounding + structured output + a confidence gate — that combination is comparatively cheap and catches the majority of ungrounded answers. Tool calling and the verification pass in Advanced RAG add real latency and cost, so they earn their place on specific query types (numeric, high-stakes) rather than every request. Fine-tuning sits outside the per-request path entirely — it's a periodic, offline investment in how the model behaves, not a substitute for anything above.

None of this makes hallucination zero. It makes it bounded, visible, and — on the cases that matter most — caught before a person ever sees the wrong answer.