Contents
At a glance
- Category
- Core agent concept
- Two kinds
- Closed-book (invented from training) and open-book (contradicts supplied sources)
- Root cause
- Models predict plausible continuations; nothing marks a token as true
- Biggest single fix
- Ground the answer in retrieved source text and require verifiable quotes
- Second fix
- Give the model an explicit permitted way to say it does not know
- Not a fix
- Temperature 0, or telling the model in the prompt not to hallucinate
What a hallucination is
A hallucination is model output that reads as authoritative and is not true: an invented order number, a policy clause that does not exist, a citation to a paper nobody wrote. The word is contested - some researchers prefer confabulation - but the operational meaning is stable and it is the single biggest reason businesses hesitate to put a model in front of customers.
It helps to split the phenomenon in two. Closed-book hallucination is the model inventing something it was never given, drawing on statistical patterns from training. Open-book hallucination is the model contradicting or embellishing source material that was right there in the prompt. The second is the one that shows up in business automations, and it is far more fixable.
Why it happens
A language model predicts likely continuations. Nothing in that process distinguishes "true" from "plausible", and a specific-sounding answer is often more probable than an admission of ignorance, because the training data is full of confident answers and thin on people saying they do not know. Under pressure - a question whose answer is not in the context, a schema demanding a field the source never mentioned - producing something is the path of least resistance.
This is why prompt design matters more than model choice for this particular failure. A prompt that says "answer from the supplied documents; if the answer is not there, reply exactly NOT_FOUND" gives the model a permitted way out. A prompt that just asks the question does not, and a schema with a required field is actively demanding invention.
What actually reduces it
Four things, in order of impact for a typical build. First, ground the answer: retrieval-augmented generation puts the real source text in the prompt so the model is summarising rather than recalling. Second, require citations - make the model quote the sentence it relied on, and validate that the quote appears in the source. That single check catches most open-book hallucinations mechanically.
Third, allow abstention and make it cheap: an explicit NOT_FOUND path, optional rather than required schema fields, and a route to a human when the model abstains. Fourth, verify anything with a canonical source: an order number should be checked against the orders table by a tool call, not trusted because it looks right. Lower temperature helps a little at the margins but is not a fix.
What you should know
Hallucination cannot currently be eliminated, only made rare and detectable. That shapes what you should automate: tasks where a wrong answer is caught by a downstream check are safe, tasks where a wrong answer goes straight to a customer are not, unless a person approves. Measure the rate rather than debating it - a twenty-case eval with fact-check assertions will tell you your real number in an afternoon.
Be careful with the word in front of clients too. "The AI hallucinated" sounds like an excuse; "the system produced an unverified value and our validation step caught it before it reached you" describes a working control, which is what a well-built automation should be able to say.
Hallucination FAQ
Why do AI models hallucinate?
Because they generate the most likely next token, and likelihood is not truth. Training data is full of confident answers and short on people saying they do not know, so a specific-sounding invention often scores higher than an admission of ignorance - especially when your prompt or schema demands an answer.
Does RAG stop hallucinations?
It reduces them substantially by putting the real text in front of the model, so the task becomes summarising rather than recalling. It does not eliminate them: models still embellish supplied sources, and if retrieval returns nothing relevant the model may answer from memory anyway. Pair retrieval with required citations.
Does setting temperature to zero prevent hallucinations?
No. Temperature 0 makes the output deterministic, not correct - it will reliably reproduce the same confident invention. It is worth setting for extraction and classification because it removes variance, but it should never be presented as an accuracy control.
How do I detect hallucinations automatically?
Make the model quote the exact source sentence it relied on, then check that string appears in the source document. Verify any identifier - order numbers, SKUs, customer IDs - against the real system with a lookup. Both are cheap deterministic checks that catch the majority of open-book cases.
What tasks are too risky for an AI that might hallucinate?
Anything where a wrong answer reaches a customer or a regulator with no check in between: medical, legal or financial advice, warranty and refund promises, delivery commitments. Those need a human approval step. Tasks where a downstream validation catches errors are safe to automate fully.
Do bigger models hallucinate less?
Generally yes on general knowledge, but the difference on your specific task is often smaller than the difference good grounding and validation make. A small model reading the right document usually beats a frontier model guessing, at a fraction of the cost. Test both on your own eval set.
Primary source
A Survey on Hallucination in Large Language Models (arXiv)
Checked August 2026. Anything priced or rate-limited moves, so verify against the source before you budget.
Want this built rather than explained?
Knowing what hallucination means is one thing; having it running in your business is another. We scope one workflow at a time, quote a fixed price in writing after a free consult, build it, and hand it over with documentation you own.
