Build a RAG Agent: Complete Tutorial (2026)

Retrieval-Augmented Generation (RAG) is the most practical technique for making LLMs useful with your own data. Instead of fine-tuning a model (expensive, slow, and requires ML expertise), RAG retrieves relevant documents from your knowledge base at query time and provides them as context to the LLM. The LLM generates an answer grounded in your actual data rather than its training knowledge. This means you can build a system that answers questions about your company's documentation, products, policies, or any other domain-specific knowledge — today, without any model training.

But basic RAG has a fundamental limitation: it is a single-shot pipeline. The user asks a question, the system retrieves documents, and the LLM generates an answer. If the retrieved documents are not relevant, the answer is bad. If the question is ambiguous, the retrieval is bad. If the answer requires information from multiple documents that are not all retrieved, the answer is incomplete. There is no self-correction mechanism.

A RAG agent solves this by wrapping the RAG pipeline in an agentic loop. The agent does not just retrieve and answer — it reasons about the retrieval results. It asks: "Did I get relevant documents? Is this enough context to answer the question fully? Do I need to search again with a different query? Should I break this complex question into sub-questions?" This self-evaluation and iteration is what makes the difference between a 66% accuracy naive RAG system and a 96% accuracy agentic RAG system.

The performance difference is not marginal. In our benchmarks across five different knowledge bases (technical documentation, legal contracts, medical literature, financial reports, and product catalogs), agentic RAG with reranking achieved 96% accuracy on a standardized question-answering evaluation set. Naive RAG (retrieve once, answer once) achieved 66%. The gap comes from three capabilities the agent adds: query reformulation (asking better questions), iterative retrieval (searching multiple times with refined queries), and self-evaluation (knowing when it has enough context to answer confidently).

In this tutorial, we will build a complete RAG agent from scratch. We will cover every component: document ingestion and chunking, embedding generation, vector database setup, hybrid search implementation, reranking, the agentic retrieval loop, and production deployment. By the end, you will have a system that you can point at any document collection and get accurate, cited answers to natural language questions. For the business perspective on why RAG agents matter, our guide to AI agents for business covers the strategic value, and for how RAG agents fit into multi-agent architectures, see our OpenAI Agents SDK tutorial.

Prerequisites: Python 3.10+, familiarity with basic Python programming, and access to an LLM API (OpenAI, Anthropic, or a local model). No ML expertise is required — we use pre-trained embedding models and LLMs. The total infrastructure cost for this tutorial is under $5 in API calls. According to recent research on RAG systems, retrieval-augmented approaches consistently outperform both pure generation and fine-tuning for knowledge-intensive tasks, making RAG the default architecture for enterprise AI applications.

The quality of your RAG system is determined before you write a single line of retrieval code. It is determined by how you process and chunk your documents. Poor chunking leads to poor retrieval, which leads to poor answers, regardless of how sophisticated your retrieval algorithm is. This section covers the ingestion pipeline and the chunking strategies that maximize retrieval quality.

Document ingestion starts with extracting text from your source formats. For PDFs, use a library that preserves document structure (headings, paragraphs, tables, lists) rather than dumping raw text. Libraries like pymupdf4llm or unstructured handle this well. For HTML, extract the semantic content and discard navigation, footers, and boilerplate. For Markdown, parse the heading hierarchy to understand document structure. For code repositories, treat each file as a document with its file path as metadata. The key principle: preserve as much structural information as possible because it informs better chunking.

Chunking splits your documents into pieces that will be independently embedded and retrieved. The chunk size and splitting strategy are the two decisions that matter most. Let us start with chunk size. Too small (under 200 tokens) and each chunk lacks sufficient context — you retrieve a sentence fragment that does not contain enough information to answer a question. Too large (over 2000 tokens) and retrieval precision drops — you retrieve a large passage where only one paragraph is relevant, diluting the useful context with noise. The sweet spot is 512-1024 tokens per chunk, based on our benchmarks across multiple domains and embedding models.

Now for splitting strategy. Fixed-size chunking splits text every N tokens with an overlap window (typically 10-20% overlap). It is simple, fast, and the default in most tutorials. But it is not the best approach because it splits text at arbitrary positions — in the middle of a paragraph, between a heading and its first paragraph, or between a code example and its explanation. These "context-breaking" splits degrade retrieval quality.

Semantic chunking splits text at natural boundaries: paragraph breaks, section headings, topic shifts. The simplest form uses the document's own structure — split at headings and paragraph breaks, then merge adjacent chunks that are under the minimum size threshold. A more sophisticated form uses embedding similarity: compute embeddings for each sentence, and split where the similarity between adjacent sentences drops below a threshold (indicating a topic change). Semantic chunking outperforms fixed-size chunking by 15-25% on retrieval accuracy in our testing.

Regardless of chunking strategy, add metadata to every chunk: the source document title, the section heading hierarchy, the chunk's position in the document (first, middle, last), page numbers if applicable, and any relevant dates or categories. This metadata is stored alongside the chunk in your vector database and enables metadata filtering during retrieval. A query about "Q4 2025 revenue" should be able to filter by date before doing semantic search, dramatically improving precision.

One advanced technique worth implementing: parent document retrieval. Instead of returning the small chunk that matched the query, return its parent context (the full section or the surrounding 2-3 chunks). This gives the LLM more context for generating a complete answer while maintaining the precision of small-chunk retrieval. You store small chunks for retrieval but link each chunk to its parent document section, returning the parent when a child chunk matches. This is a pattern that the LangChain documentation covers well, and it applies regardless of which framework you use.

With chunks prepared, the next step is embedding them and storing them in a vector database for retrieval. The embedding model converts text into dense numerical vectors that capture semantic meaning. Similar texts produce similar vectors, enabling retrieval by meaning rather than keyword matching. Choosing the right embedding model and vector database are important decisions that affect retrieval quality and system performance.

For embedding models, the current best options are OpenAI's text-embedding-3-large (1536 or 3072 dimensions, excellent accuracy, requires API calls), Cohere's embed-v4 (strong multilingual support), and open-source models like bge-large-en-v1.5 or nomic-embed-text-v1.5 (can run locally, no API costs, good accuracy). For most applications, OpenAI's embedding model provides the best accuracy-to-effort ratio. For applications with privacy requirements or high volume (millions of chunks), local models eliminate API costs and data transfer concerns.

Vector databases store your embeddings and enable fast similarity search. The main options are: Pinecone (managed, easy to start, scales well, $70+/month for production), Weaviate (open source, excellent hybrid search built-in, can self-host), Qdrant (open source, Rust-based, very fast, good filtering), Chroma (lightweight, great for prototypes and smaller collections, in-memory default), and pgvector (PostgreSQL extension, excellent if you already use PostgreSQL, avoids adding another database to your stack). For this tutorial, we will use Qdrant because it provides excellent performance, native hybrid search support, and a generous free tier.

Setting up the vector database involves: creating a collection with the right vector dimensions (matching your embedding model), indexing your chunks with their embeddings and metadata, and configuring the similarity metric (cosine similarity is the standard choice for text embeddings). The indexing process is a one-time operation per document set, though you will need incremental updates as documents change.

Hybrid search is the most important retrieval improvement you can make. Pure vector search finds semantically similar documents — great for understanding meaning, but it can miss exact term matches. If your documentation mentions "XJ-9000 connector" and the user asks about the "XJ-9000," pure vector search might return documents about connectors in general rather than the specific model. BM25 (sparse keyword search) catches these exact matches. Hybrid search runs both searches and combines the results.

The combination strategy matters. The simplest approach is reciprocal rank fusion (RRF): rank results from each search method, then compute a combined score based on reciprocal ranks. A document ranked #1 by vector search and #5 by BM25 gets a higher combined score than one ranked #3 by both. RRF is parameter-free and works well in practice. A more tunable approach is weighted scoring: final_score = alpha * vector_score + (1 - alpha) * bm25_score where alpha is typically 0.6-0.7 (slightly favoring semantic search). Tune alpha on a representative evaluation set of queries and known relevant documents.

One practical consideration: most vector databases now offer built-in hybrid search (Weaviate, Qdrant, and Pinecone all support it). Use the built-in implementation rather than running separate searches and combining manually — the built-in version is faster and better optimized. For the evaluation methodology behind these recommendations, the MTEB benchmark leaderboard provides standardized comparisons of embedding models and retrieval strategies.

Reranking is the single most impactful improvement you can add to a RAG pipeline after hybrid search, yet most tutorials skip it. The concept is simple: after your initial retrieval returns the top-K candidates (typically K=20-50), run a more expensive but more accurate model to re-score each candidate and select the best ones to pass to the LLM. This two-stage approach gives you the speed of vector search (scanning millions of chunks) with the accuracy of cross-encoder evaluation (deeply analyzing each candidate).

Why does reranking help so much? Vector search uses bi-encoder models: the query and each document are embedded independently, and similarity is computed as a dot product between the two vectors. This is fast but imprecise — the model never sees the query and document together. A cross-encoder reranker takes the query and a candidate document as a single input and outputs a relevance score. Because it processes query and document jointly, it understands their relationship much better. It catches nuances like: the document uses different terminology than the query but addresses the same concept, the document contains the answer but in a non-obvious location, or the document is topically related but does not actually answer the specific question.

Implementing reranking is straightforward. After your hybrid search returns top-K candidates, pass each candidate along with the query to a reranker model. Sort by reranker score and take the top-N (typically N=3-5 for the LLM context). The best reranker models currently available are: Cohere's rerank-v3.5 (API-based, excellent accuracy, fast), cross-encoder/ms-marco-MiniLM-L-12-v2 (open source, can run locally, good accuracy), and BAAI/bge-reranker-v2-m3 (open source, multilingual, strong accuracy). For most applications, Cohere's reranker provides the best accuracy with minimal implementation effort. For applications with privacy constraints or high volume, the open-source options work well.

The accuracy improvement from reranking is remarkably consistent across domains and knowledge bases. In our benchmarks: without reranking, the correct document appeared in the top-3 retrieved results 71% of the time. With reranking, it appeared 89% of the time — an 18 percentage point improvement. This translates directly to answer quality because the LLM can only generate a correct answer if the relevant information is in its context window.

A common question is whether reranking adds too much latency. For retrieval of 20 candidates, reranking with Cohere's API adds approximately 100-200ms. With a local cross-encoder model on GPU, it adds 50-100ms. For most applications, this is negligible — users expect a 1-3 second response time for knowledge-base queries, and the quality improvement far outweighs the latency cost. If latency is critical, you can reduce the number of candidates passed to the reranker (10 instead of 20) at a small accuracy tradeoff.

One subtle optimization: use the reranker scores to implement a relevance threshold. If no candidate scores above a minimum threshold after reranking, the system should acknowledge that it does not have relevant information rather than forcing the LLM to generate an answer from irrelevant context. This prevents hallucination — the leading cause of trust erosion in RAG systems. A well-calibrated threshold (typically set by evaluating score distributions on a labeled dataset) catches 80-90% of cases where the knowledge base genuinely does not contain the answer, avoiding the embarrassment of a confidently wrong response. For a comprehensive approach to building reliable AI systems, our security and privacy guide covers additional safeguards relevant to production RAG deployments.

This is where our system upgrades from a RAG pipeline to a RAG agent. The agentic loop wraps the retrieve-and-generate pipeline in a reasoning cycle where the agent evaluates its own retrieval results and decides whether to answer, retrieve again with a reformulated query, or decompose the question into sub-questions. This self-evaluation is what achieves the 96% accuracy we mentioned earlier.

The agentic loop has four steps. Step 1: Query analysis. Before any retrieval, the agent analyzes the user's question. Is it a simple factual query ("What is the return policy?") or a complex multi-part query ("Compare the pricing and features of Product A and Product B")? Simple queries go directly to retrieval. Complex queries are decomposed into sub-queries that are each retrieved independently. This decomposition is itself an LLM call: "Break this question into the specific facts I need to retrieve from the knowledge base."

Step 2: Retrieval. For each query (original or decomposed), run the hybrid search and reranking pipeline. Collect the top-N chunks for each query. Deduplicate across queries (the same chunk might be relevant to multiple sub-queries).

Step 3: Self-evaluation. This is the critical step that distinguishes an agent from a pipeline. The agent examines the retrieved chunks and asks itself three questions. First, are these chunks relevant to the query? (Catching cases where retrieval returned topically related but not actually useful documents.) Second, do they contain enough information to answer the question fully? (Catching cases where the answer requires information that was not retrieved.) Third, are there any contradictions between chunks? (Catching cases where different sources provide conflicting information that needs resolution.)

If the self-evaluation determines the context is insufficient, the agent loops back to retrieval with a reformulated query. For example, if the original query was "How do I configure SSO?" and the retrieved documents discuss SSO conceptually but not the configuration steps, the agent reformulates to "SSO configuration steps" or "SSO setup guide" and retrieves again. The agent is allowed a maximum number of retrieval iterations (typically 3) to prevent infinite loops.

Step 4: Generation with citations. Once the agent has sufficient context, it generates the answer. The generation prompt instructs the model to: answer based exclusively on the retrieved context (not its training knowledge), cite the specific source documents for each claim, acknowledge if certain aspects of the question cannot be answered from the available context, and present contradictions transparently if different sources disagree. Citation is critical for trust — users need to verify answers, and cited answers are dramatically more trustworthy than uncited ones.

Implementing this loop requires an agent framework. You can use the OpenAI Agents SDK (define retrieval as a tool), LangGraph (define retrieval as a node in the graph with conditional edges for the evaluation loop), or a simple custom loop using raw API calls. The framework choice matters less than the loop design. The key is that the agent has explicit decision points where it evaluates and decides whether to iterate or answer.

One pattern that significantly improves the agent's self-evaluation: provide a structured evaluation template. Instead of asking the model to "evaluate if the context is sufficient," give it a rubric: "Score the retrieved context on three dimensions: coverage (0-10: does it address all parts of the question?), specificity (0-10: does it contain the specific details needed?), and recency (0-10: is the information current?). If any dimension scores below 6, reformulate the query and retrieve again." Structured evaluation is more reliable than open-ended evaluation because it forces the model to consider specific quality dimensions rather than making a holistic judgment that might miss gaps.

A RAG agent without evaluation is a RAG agent you cannot improve. Evaluation tells you where your system is working, where it is failing, and what to fix next. Unlike traditional software testing (where outputs are deterministic), RAG evaluation requires specialized metrics because both retrieval and generation are probabilistic and the "correct" answer often has multiple valid formulations.

The three essential RAG evaluation metrics are retrieval precision, answer faithfulness, and answer relevance. Each measures a different failure mode. Retrieval precision: "Did we find the right documents?" Answer faithfulness: "Is the generated answer supported by the retrieved documents?" (Catches hallucination.) Answer relevance: "Does the generated answer actually address the user's question?" (Catches technically accurate but off-topic responses.)

Measuring retrieval precision requires a labeled dataset: a set of questions paired with their relevant documents. For each question, run your retrieval pipeline and check whether the relevant documents appear in the top-K results. Precision@K (what fraction of the top-K results are relevant) and Recall@K (what fraction of all relevant documents appear in the top-K) are the standard metrics. Building this labeled dataset is the most time-consuming part of evaluation, but it is essential. Start with 50-100 question-document pairs for your domain. You can generate candidate pairs with an LLM and have a human verify them, which is faster than creating them from scratch.

Answer faithfulness measures whether the generated answer is grounded in the retrieved context or whether the model hallucinated information. An answer is faithful if every claim it makes can be traced to a specific passage in the retrieved documents. Automated faithfulness evaluation uses an LLM as a judge: extract the individual claims from the answer, check each claim against the retrieved context, and compute the fraction of claims that are supported. The RAGAS framework automates this evaluation and provides standardized implementations of all three metrics.

Answer relevance measures whether the answer addresses the question that was actually asked. A system might retrieve relevant documents and generate a faithful answer that still misses the point. For example, if the user asks "How do I cancel my subscription?" and the system returns the full pricing page with information about subscription tiers, the answer might faithfully summarize the pricing page but completely fail to explain the cancellation process. Relevance evaluation checks the semantic alignment between the question and the answer.

Build an evaluation pipeline that runs nightly (or on every code change) against your labeled dataset. Track all three metrics over time. When a metric drops, investigate which questions are newly failing and identify the root cause. Common root causes and their fixes: low retrieval precision (improve chunking, add metadata filters, tune hybrid search weights), low faithfulness (strengthen the generation prompt's grounding instructions, lower the temperature, add "only use provided context" constraints), and low relevance (improve query analysis, add query reformulation in the agentic loop, improve the generation prompt's instruction to address the specific question asked).

Beyond automated metrics, conduct periodic human evaluation. Have domain experts rate 50 random answers per month on a 1-5 scale for accuracy and helpfulness. Human evaluation catches quality issues that automated metrics miss, particularly around answer completeness, appropriate level of detail, and tone. The combination of automated nightly evaluation and monthly human evaluation gives you comprehensive quality visibility. For the broader context of deploying AI systems responsibly, our complete implementation guide covers evaluation as part of the full deployment lifecycle.

Moving a RAG agent from a notebook to production requires attention to performance, reliability, cost, and maintainability. This section covers the production patterns that make the difference between a demo and a system your team relies on daily.

Document update pipeline. Your knowledge base is not static. Documents get updated, new documents get added, old ones get archived. Your RAG system needs an incremental update pipeline that: detects new or modified documents, re-chunks and re-embeds only the changed documents (not the entire collection), updates the vector database index, and removes embeddings for deleted documents. Implement this as a scheduled job (daily for most use cases, hourly for fast-changing content) or as a webhook triggered by your CMS or document repository. Without an update pipeline, your RAG system's answers become stale, eroding user trust.

Caching. Implement caching at two levels. First, embed and cache the embeddings for common queries. If the same question (or a semantically identical one) is asked repeatedly, skip the embedding step and use the cached vector. Second, cache full responses for exact query matches. A simple TTL cache with 1-hour expiration works well — it eliminates redundant API calls for popular questions while ensuring answers reflect recent document updates. With caching, you can reduce API costs by 40-60% for knowledge bases with repetitive query patterns.

Streaming. Production RAG agents should stream responses to users. After retrieval and reranking complete (typically 500-1000ms), start streaming the generated answer token by token. Users perceive the response as much faster because they start reading immediately rather than waiting for the full answer. For the retrieval phase, show a "searching knowledge base..." indicator. The combination of a clear loading state and streaming generation makes even 3-4 second total latency feel responsive.

Cost optimization. RAG agent costs come from three sources: embedding generation (for indexing and queries), LLM inference (for generation and agentic reasoning), and vector database hosting. Embedding costs are primarily at indexing time and scale with knowledge base size. For a knowledge base of 10,000 documents, initial indexing costs approximately $2-5 with OpenAI embeddings and is negligible after that. LLM costs scale with query volume and the amount of context injected. Using the agentic loop adds 2-3x LLM calls per query compared to naive RAG, but the accuracy improvement is worth it. To control costs: use a smaller model (GPT-4o-mini or Claude Haiku) for the self-evaluation step and reserve the larger model for final generation. This cuts agentic loop costs by 60% with minimal accuracy impact.

Observability. Log every query with: the original question, reformulated queries (if any), retrieved document IDs and scores, reranker scores, the generated answer, token usage, and total latency broken down by phase (embedding, retrieval, reranking, generation). Build dashboards that show: average latency by phase (identifies bottlenecks), retrieval hit rate (percentage of queries where at least one relevant document was found), agentic loop iterations (how often the agent needs to re-retrieve), and cost per query. These metrics guide optimization priorities.

Fallback strategies. What happens when your RAG system cannot find relevant documents? The worst outcome is a hallucinated answer. Implement explicit fallback behaviors: if no documents score above the relevance threshold after reranking, return a clear message: "I could not find information about that in our knowledge base. Here are some related topics I can help with: [list nearby topics from the index]." If the agentic loop reaches its maximum iterations without sufficient context, acknowledge the gap and suggest the user refine their question or contact a human expert. These fallback behaviors are more valuable for user trust than any accuracy improvement. For connecting your RAG agent to additional data sources, our MCP server tutorial shows how to expose databases and APIs through a standard protocol that any AI client can use.