RAG in AI: what retrieval-augmented generation is and how it works

Ask a chatbot about your company's refund policy, last week's price change or a clause in your own contract, and it will either admit it does not know or, worse, make something up. Large language models only know what was in their training data, and your documents were never in it.

Retrieval-augmented generation, or RAG, is the standard fix. This guide explains what RAG is in plain English, walks through the architecture one stage at a time, shows a small RAG pipeline in Python that we ran ourselves, and covers the choices that decide whether a RAG system works: chunking, search type, RAG vs fine-tuning, and when agentic RAG is worth the extra cost.

What is RAG in AI?

RAG stands for retrieval-augmented generation. It is a way of building AI applications in which the system first retrieves relevant information from a knowledge source, then augments the prompt with that information, and only then lets the language model generate an answer. AWS describes it as optimizing an LLM's output so it references an authoritative knowledge base (opens in a new tab) outside its training data before responding.

A useful picture is the difference between a closed-book and an open-book exam. A plain LLM answers from memory. A RAG system lets the model open the right page of the right book first. NVIDIA's explainer on RAG (opens in a new tab) uses a courtroom version of the same idea: the model is a judge with broad general knowledge, and retrieval is the clerk sent to the law library to fetch the precedents for this specific case.

If you are new to the model side of this, our explainer on what LLM stands for and how large language models work covers tokens, training and why models state false things confidently. RAG exists largely to manage that last problem.

Where RAG came from

The term comes from a 2020 paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (opens in a new tab), by Patrick Lewis and 11 co-authors. Lewis was then a doctoral student at University College London and working at Meta's London AI lab, according to NVIDIA. The paper paired two kinds of memory: the knowledge stored inside a pretrained model's parameters, which the authors called parametric memory, and a dense vector index of Wikipedia that a retriever could search, which they called non-parametric memory. The combination set new state-of-the-art results on three open-domain question answering tasks and produced more specific and factual text than the model alone.

Why LLMs need RAG

RAG solves four practical problems that every team hits when it puts an LLM in front of real users.

• Knowledge cutoffs. A model knows nothing after its training date. RAG lets you feed it today's price list or this morning's incident report. Google Cloud notes that LLMs limited to their pre-trained data (opens in a new tab) give outdated and potentially inaccurate responses.
• Private data. Your contracts, tickets and wikis were never in any public training set. Retrieval brings them in at question time, without sending them off to train a model.
• Hallucinations. Supplying the facts in the prompt gives the model something to answer from instead of guessing. It helps a great deal, but as covered below, it does not make errors impossible.
• Trust and control. Because the answer comes from known passages, the system can show its sources. AWS also points out that developers can restrict retrieval by authorization level (opens in a new tab), so an employee only gets answers drawn from documents they are allowed to see.

It is also cheap compared with the alternatives. Updating a RAG system means updating documents in an index, which AWS calls a more cost-effective way to introduce new data than retraining a foundation model.

RAG architecture: how a RAG pipeline works

Every RAG system has two halves. An indexing pipeline runs ahead of time and prepares your documents. A query pipeline runs every time someone asks a question. Since a picture of boxes and arrows only goes so far, here is the full RAG architecture diagram written out as a table you can map onto any framework or cloud service.

AWS's own description of the flow matches these stages: embedding models convert your data into numerical representations stored in a vector database, the user query is converted into a vector and matched against it, the relevant data is added to the prompt, and documents and their embeddings are updated asynchronously (opens in a new tab), in real time or in periodic batches. Stages 2 and 5 can be skipped in the simplest systems, but they are where much of the quality work happens in serious ones.

A minimal RAG example in Python

The fastest way to understand RAG is to build a tiny one. This script uses only Python's standard library. It indexes four short policy snippets, retrieves the two most relevant to a question using TF-IDF keyword scoring, and builds the augmented prompt you would send to a model. Real systems swap the scoring for an embedding model and a vector database, but the shape of the code stays the same.

import math
import re
from collections import Counter

# 1. Your knowledge base: in real life, chunks of your own documents.
chunks = [
    {"id": "refunds-1", "text": "Customers can request a full refund within 30 days of purchase. Refunds go back to the original payment method."},
    {"id": "refunds-2", "text": "Annual plans cancelled after 30 days get a prorated refund for the unused months."},
    {"id": "shipping-1", "text": "Standard shipping takes 3 to 5 business days. Express shipping arrives in 1 to 2 business days."},
    {"id": "support-1", "text": "Support is available by chat from 9am to 6pm Eastern, Monday to Friday."},
]

def tokenize(text):
    return re.findall(r"[a-z0-9]+", text.lower())

# 2. Index: score words by TF-IDF so rare, specific words count more.
docs = [Counter(tokenize(c["text"])) for c in chunks]
df = Counter(word for d in docs for word in d)
idf = {w: math.log(len(docs) / n) + 1 for w, n in df.items()}

def vectorize(counts):
    vec = {w: tf * idf.get(w, 0.0) for w, tf in counts.items()}
    norm = math.sqrt(sum(v * v for v in vec.values())) or 1.0
    return {w: v / norm for w, v in vec.items()}

index = [vectorize(d) for d in docs]

# 3. Retrieve: rank chunks by cosine similarity to the question.
def retrieve(question, k=2):
    q = vectorize(Counter(tokenize(question)))
    scored = [(sum(q.get(w, 0) * v for w, v in vec.items()), c)
              for vec, c in zip(index, chunks)]
    return sorted(scored, key=lambda s: s[0], reverse=True)[:k]

# 4. Augment: put the retrieved chunks into the prompt, with IDs to cite.
def build_prompt(question):
    hits = retrieve(question)
    for score, c in hits:
        print(f"retrieved {c['id']} (score {score:.2f})")
    sources = "\n".join(f"<source id=\"{c['id']}\">{c['text']}</source>" for _, c in hits)
    return (f"{sources}\n\nAnswer the question using only the sources above. "
            f"Cite the source id. If the answer is not in the sources, say so.\n\n"
            f"Question: {question}")

# 5. Generate: send this prompt to any LLM (API call or chat window).
print(build_prompt("Can I get a refund on my annual plan after 30 days?"))

Running python3 mini_rag.py prints exactly this:

retrieved refunds-2 (score 0.59)
retrieved refunds-1 (score 0.29)
<source id="refunds-2">Annual plans cancelled after 30 days get a prorated refund for the unused months.</source>
<source id="refunds-1">Customers can request a full refund within 30 days of purchase. Refunds go back to the original payment method.</source>

Answer the question using only the sources above. Cite the source id. If the answer is not in the sources, say so.

Question: Can I get a refund on my annual plan after 30 days?

The retriever ranked the annual-plan chunk first because it shares the specific words "annual" and "plans" with the question, and it skipped the shipping and support chunks entirely. Paste that prompt into any chatbot or send it through an LLM API, and the model has exactly what it needs to answer correctly and cite refunds-2.

Where this simple retriever breaks

Now ask the same index a question that uses different words from the documents:

# Replace the last line of mini_rag.py with:
for score, c in retrieve("How long does delivery take?"):
    print(c["id"], round(score, 2))

# Output
refunds-1 0.0
refunds-2 0.0

Both scores are zero, and the shipping chunk is not even in the results. It says "shipping", the question says "delivery", and a keyword retriever has no idea those mean the same thing. This is the reason production RAG uses embeddings: an embedding model places "delivery" and "shipping" close together, so a vector search finds the right chunk. The opposite failure also happens. Anthropic's contextual retrieval research (opens in a new tab) gives the example of a query for "Error code TS-999": an embedding search may return general material about error codes, while a keyword method like BM25 matches the exact string. That is why the strongest systems use both.

What makes a RAG system accurate

Most RAG systems that disappoint have a retrieval problem, not a model problem. These are the levers with the best evidence behind them.

Hybrid search: keywords plus meaning

Google Cloud describes search engines that use semantic search and keyword search together (opens in a new tab), which it calls hybrid search, followed by a re-ranker that scores the results so the top ones are the most relevant. Semantic search catches paraphrases. Keyword search catches names, product codes and exact phrases. You want both.

Chunks that keep their context

A chunk cut out of a long document often loses the information that made it meaningful, such as which company or which quarter a sentence is about. Anthropic's 2024 contextual retrieval technique adds a short, model-written explanation of 50 to 100 tokens to each chunk before indexing it. In its tests, the share of questions where the right chunk was missing from the top 20 results fell from 5.7 percent to 1.9 percent once every technique was combined.

Those figures come from Anthropic's own evaluation, so treat them as a strong signal about which techniques matter rather than a number you will hit on your data. The broader lesson holds up: combining keyword and vector search and then reranking beats either search alone.

A prompt that demands grounding

Retrieval is only half the job. The prompt should tell the model to answer only from the supplied sources, to cite them and to say plainly when the answer is not there, exactly as the demo above does. Our Claude prompt engineering guide covers related techniques, such as wrapping documents in tags, putting them before the question and asking for quotes before analysis.

Finally, measure retrieval on its own. Build a test set of real questions with the chunk that should answer each, and check that the chunk shows up in the top results before you judge the final answers. If retrieval fails, no amount of prompt tuning will fix it.

RAG vs fine-tuning vs long context

RAG is not the only way to give a model knowledge, and choosing wrong wastes weeks. OpenAI's guide to optimizing LLM accuracy (opens in a new tab) draws the line clearly: optimize the context when the model lacks knowledge that was not in its training set, and optimize the model itself with fine-tuning when it produces inconsistent results or the wrong format. In the guide's words, "RAG only solves our in-context learning issues." It also describes a case where adding retrieved context made results worse, because the problem was the model's behavior, not missing facts.

Long context windows add a third option. Anthropic notes that if a knowledge base is smaller than about 200,000 tokens, roughly 500 pages (opens in a new tab), you can skip retrieval and include the whole thing in the prompt, using prompt caching to keep it fast and affordable.

These are not exclusive. A common pattern is a fine-tuned or carefully prompted model that has learned your output format, fed by a RAG pipeline that supplies the facts. If cost is the worry, RAG also pairs well with smaller models: a small model given the right passage often does the job, a point we expand on in how small language models cut inference costs.

What is agentic RAG?

Basic RAG retrieves once, the same way, for every question. That works for "What is the refund window?" but falls short on "Compare our refund policy with the three competitors mentioned in last quarter's board report." A January 2025 survey of agentic RAG (opens in a new tab) describes traditional RAG as constrained by static workflows and lacking the adaptability needed for multi-step reasoning. Agentic RAG hands control of retrieval to an agent that can plan, use tools, reflect on what it found and search again.

IBM's overview of agentic RAG (opens in a new tab) lists the roles agents typically play:

• Routing agents decide which knowledge source or tool fits the question, such as the HR wiki, the product database or a web search.
• Query planning agents break a complex question into smaller sub-questions, send each to the right place and combine the answers.
• ReAct agents alternate reasoning and action step by step, adjusting the next search based on what the last one returned.
• Plan-and-execute agents work out all the steps up front and then run the whole workflow, which can cut cost compared with checking back after every step.

The trade-off is cost and reliability. IBM notes that more agents mean more tokens and more latency, that agents can fail on complex tasks, and that agentic RAG is not always better than the simple version. Start with a plain pipeline, measure where it fails, and add agentic steps only for the question types that need them. If you want to build the agent loop itself, our guide on how to build an AI agent shows a tested tool-calling loop, and the Model Context Protocol is a standard way to connect an agent to the data sources it retrieves from. For the broader distinction between agents and content generators, see agentic AI vs generative AI.

RAG limitations: what still goes wrong

• It still hallucinates. A 2024 study by Varun Magesh and colleagues, "Hallucination-Free?" (opens in a new tab), tested three commercial legal research tools built on RAG and found they hallucinated between 17 and 33 percent of the time, even though vendors had marketed the approach as eliminating or avoiding hallucinations. They did hallucinate less than general-purpose GPT-4.
• Bad retrieval in, bad answer out. If the right passage is not retrieved, the model either says it does not know or, worse, answers from the wrong passage with full confidence.
• Stale indexes. RAG is only as current as its last re-index. A policy updated in the wiki but not in the vector database will still be quoted in its old form.
• Security. Retrieved text goes straight into the prompt, so a document containing hidden instructions can try to steer the model. Treat retrieved content as untrusted input and enforce permissions at retrieval time, not in the prompt.

When to use RAG: a quick checklist

1. Is the knowledge outside the model? Private, recent or specialized information points to RAG. General knowledge does not need it.
2. Is it bigger than about 500 pages or changing often? If yes, use RAG. If it is small and stable, try putting it all in the prompt first.
3. Do answers need citations or per-user permissions? Both are much easier with RAG than with any alternative.
4. Is the real problem format or behavior? If the model knows enough but answers inconsistently, look at prompting or fine-tuning before retrieval.
5. Do questions need several lookups or sources? Start with simple RAG, then add agentic routing or query planning for the question types where it falls short.

The bottom line

RAG in AI means retrieval-augmented generation: look up the relevant facts first, put them in the prompt, then let the model answer. It is the most practical way to make an LLM answer from your own, current information with sources attached, and it costs less than retraining a model. The quality of a RAG system is mostly the quality of its retrieval, so invest in hybrid search, good chunking and a test set before you tune anything else. For more plain-English guides on models, agents and the tools around them, browse our AI guides.