What is agentic RAG, and how is it different from ordinary RAG?

**Ordinary retrieval-augmented generation (RAG)** usually follows a predefined pipeline: receive a query, search one or more indexes, retrieve a fixed number of passages, place them in the prompt, and generate an answer.\n\n**Agentic RAG** gives an agent control over retrieval. The agent can decide whether retrieval is needed, reformulate the question, choose among sources or indexes, issue several searches, inspect intermediate results, follow references, resolve conflicts, and stop when it has sufficient evidence.\n\nUse ordinary RAG for predictable, high-volume questions over a well-structured knowledge base. Use agentic RAG for ambiguous, multi-part, cross-source, or investigative questions where one search is unlikely to be enough.

RAG grounds a model in information retrieved at request time. It is valuable because the knowledge can be current, private, permission-controlled, and linked back to evidence rather than memorised in model weights.\n\nA conventional RAG pipeline commonly performs these steps:\n\n1. convert the user's request into a search query;\n2. search an index;\n3. select the top-ranked chunks;\n4. add the chunks to the model context;\n5. generate an answer;\n6. attach the retrieved sources.\n\nThe application designer chooses the search strategy, indexes, number of results, filters, and sequence in advance. This simplicity improves latency, predictability, cost control, and reproducibility.\n\nAgentic RAG treats retrieval as one or more tools inside an agent loop. The agent may:\n\n- decompose a broad question into subquestions;\n- search different collections for different aspects;\n- switch between semantic, keyword, graph, database, and web retrieval;\n- inspect whether a source is current and authoritative;\n- retrieve a full document after finding a relevant chunk;\n- search for contradictory evidence;\n- repeat retrieval when evidence is incomplete;\n- ask a clarifying question;\n- abstain when the available sources do not support an answer.\n\nMicrosoft's architecture guidance describes this distinction directly: standard RAG follows a fixed orchestrated sequence, while agentic RAG lets an agent invoke retrieval on demand, evaluate intermediate results, and iterate across heterogeneous data sources.\n\nAgentic RAG can improve difficult answers, but it introduces new risks:\n\n- more searches, tokens, and latency;\n- loops that fail to converge;\n- inconsistent search strategies between runs;\n- retrieval from an unauthorised source;\n- prompt injection in retrieved documents;\n- source duplication mistaken for corroboration;\n- the model stopping too early or continuing wastefully;\n- harder evaluation and debugging.\n\nA robust agentic RAG system needs:\n\n1. **Source routing rules** — which index is appropriate for which data.\n2. **Access control** — permission checks before retrieval, not merely before display.\n3. **Freshness metadata** — valid dates, versions, and superseded records.\n4. **Provenance** — source identity and claim-level evidence links.\n5. **Budgets** — maximum searches, tokens, time, and tool calls.\n6. **Stopping criteria** — sufficient evidence, unresolved conflict, or escalation.\n7. **Injection defences** — retrieved content is evidence, not trusted instruction.\n8. **Evaluation** — retrieval recall, citation correctness, completeness, and answer quality.\n9. **Observability** — queries, results, filters, reranking, and final evidence set.\n10. **Fallback** — ordinary retrieval or human review when the agent cannot resolve the question.\n\nFor TRACE, agentic RAG would be useful when a user asks a broad or comparative question that spans knowledge entries, recent stories, source documents, and live web evidence. A simple canonical question with one authoritative entry may be better served by deterministic retrieval.

- [Microsoft Azure Architecture Center — Develop an agentic RAG solution](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-agentic) — distinguishes fixed RAG pipelines from agent-controlled iterative retrieval. - [Microsoft Learn — Retrieval-augmented generation and indexes](https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation?view=foundry-classic) — explains RAG for private, current, and grounded information. - [Google Cloud — Vertex AI RAG Engine](https://docs.cloud.google.com/vertex-ai/generative-ai/docs/rag-overview) — documents ingestion, retrieval, grounding, and managed RAG components. - [OpenAI — File search](https://platform.openai.com/docs/guides/tools-file-search) — documents model-driven retrieval from vector stores using semantic and keyword search. - [NIST — Building evaluation probes into agentic AI](https://www.nist.gov/programs-projects/building-evaluation-probes-agentic-ai) — identifies grounding, citation correctness, completeness, and source-supported claims as agent-evaluation concerns.