A8gent
GlossaryAgent concepts

What is RAG?

Also called: retrieval-augmented generation, retrieval augmented generation

RAG, or retrieval-augmented generation, is the pattern of searching your own documents for the passages relevant to a question and putting them in the prompt, so the model answers from your material rather than from memory.

Written by A8gent
Contents
  1. At a glance
  2. What RAG is
  3. How the pipeline is built
  4. Cost and where it goes wrong
  5. What you should know
  6. Retrieval as a pricing decision
  7. FAQ
  8. Related terms

At a glance

Category
Agent concept - retrieval
Stands for
Retrieval-augmented generation
Origin
Lewis et al., 2020
Ingestion steps
Extract, chunk, embed, store with metadata
Query steps
Embed question, retrieve, rerank, build prompt with an abstain rule
Most common failure
Retrieval, not generation - inspect what was retrieved first

What RAG is

Retrieval-augmented generation is a two-stage pattern. Stage one searches a collection you control - policies, past quotes, product documentation, support tickets - and pulls out the handful of passages most relevant to the question. Stage two puts those passages into the prompt and asks the model to answer using them. The model contributes language and reasoning; your documents contribute the facts.

The idea comes from a 2020 paper by Lewis and colleagues, and it became the default architecture for business AI for a simple reason: it is the cheapest way to make a general model answer correctly about your specific business, and unlike fine-tuning you can update the knowledge by editing a document rather than retraining anything.

How the pipeline is built

Ingestion runs once per document and has four steps. Extract the text, split it into passages - see chunking - convert each passage into a vector using an embedding model, and store the vectors with their text and metadata in a vector database.

Query time has three. Embed the question with the same model, retrieve the nearest passages, and optionally rerank them with a more precise model before selecting the top few. Then build the prompt: the instruction, the retrieved passages clearly delimited, the question, and a rule for what to do when the passages do not contain the answer. That last rule is what stops the model quietly falling back on memory.

Cost and where it goes wrong

RAG is cheap to run and expensive to get right. Embedding a corpus is a one-off cost measured in cents per thousand pages on current models; storage is small; the per-query cost is one embedding call plus the retrieved passages as input tokens. There is no training bill.

The failures are almost never in the model. They are: chunks split mid-table or mid-clause so no passage contains a complete answer; the question phrased in customer language while the document uses internal jargon, so semantic search misses; stale documents nobody re-indexed; and the model answering from memory when retrieval returned nothing useful. Diagnose by inspecting what was retrieved before blaming generation - most teams debug the wrong half of the pipeline for a week.

What you should know

RAG reduces hallucination, it does not remove it. Require the model to quote the sentence it relied on and verify that string exists in the source; that one check catches most embellishment mechanically. Give it an explicit way to say the answer is not in the documents, and route those cases to a person.

For a business with a few hundred documents, be honest about whether you need a vector database at all. Keyword search over a well-organised folder, or a hybrid of keyword and semantic search, often outperforms a hastily built vector pipeline, and hybrid retrieval is the standard production answer because pure semantic search is weak on exact identifiers and product codes.

Retrieval as a pricing decision

The usual argument for retrieval is quality: a model given the right passage beats one given the whole book. There is a second argument that is easier to check, which is that on some models context length is a price band rather than a capacity limit.

gemini-2.5-pro is the clearest example. As published in August 2026, prompts of 200,000 tokens or fewer cost $1.25 per million input tokens and $10.00 output; above that threshold the same request costs $2.50 and $15.00. Retrieving the relevant 20,000 tokens instead of pasting a 250,000-token corpus keeps every call in the cheap band, and the difference is not marginal - the whole request reprices.

Even where no band exists, the arithmetic is the same shape. A pipeline that sends 20,000 tokens rather than 200,000 on every call spends a tenth on input. See token for current per-million rates and chunking for how the corpus gets cut up.

RAG FAQ

What is RAG in simple terms?

You search your own documents for the passages that answer the question, paste those passages into the prompt, and ask the model to answer from them. The model supplies the language; your documents supply the facts. It is the standard way to make a general model correct about your specific business.

Is RAG better than fine-tuning?

For knowledge, almost always. RAG updates when you edit a document, cites its sources, and costs nothing to retrain. Fine-tuning is for teaching format, tone or a narrow skill, not for teaching facts - facts baked into weights go stale and cannot be cited.

Do I need a vector database for RAG?

Not always. For a few hundred documents, keyword search or a hybrid of keyword and semantic search over an organised folder is often faster to build and more accurate, especially where questions contain exact product codes or reference numbers that semantic search handles poorly.

Why does my RAG system give wrong answers?

Usually retrieval, not the model. Check what was actually retrieved for the failing question. The usual culprits are chunks that split a table or clause in half, customer phrasing that does not match internal jargon, and documents that were never re-indexed after being updated.

Does RAG eliminate hallucinations?

It reduces them a lot and eliminates nothing. Models still embellish supplied sources, and if retrieval returns nothing relevant the model may answer from memory. Require it to quote the exact sentence it used, verify that string appears in the source, and give it a permitted way to say it does not know.

How much does RAG cost to run?

Less than most people expect. Embedding a corpus is a one-off cost of cents per thousand pages on current models, storage is minor, and each query costs one embedding call plus the retrieved passages as input tokens. The real cost is the engineering time spent on chunking and evaluation.

Primary source

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv)

Checked August 2026. Anything priced or rate-limited moves, so verify against the source before you budget.

Want this built rather than explained?

Knowing what rag means is one thing; having it running in your business is another. We scope one workflow at a time, quote a fixed price in writing after a free consult, build it, and hand it over with documentation you own.