A base model does not automatically know your newest policies, customer records, product catalog, case files, or internal documents. RAG can fetch only the relevant approved evidence at query time.
Updated September 2026 · Visual architecture guide
The evolution of RAG architectures
Retrieval-augmented generation has evolved from simple top-k vector search into a family of retrieval patterns. The useful question is not “what is the most advanced RAG?” It is “what failure are we fixing, and what evidence proves the added architecture is worth it?”
Architecture rule
Escalate only when evaluation exposes a failure.
1 Long-context or flat retrieval baseline
2 Hybrid retrieval + reranking
3 Contextual / hierarchical structure
4 Graph / corrective retrieval
5 Bounded agentic orchestration
Why RAG exists
RAG gives models controlled access to knowledge they do not automatically carry in parametric memory
The original page framed RAG as solving “knowledge cutoff” and hallucination. That is directionally useful but incomplete. Modern systems can use long context and tools; RAG remains valuable because it provides selective access to current or private knowledge, permissions, citations, freshness, and retrieval observability.
Retrieval can provide explicit source material for an answer, but it does not guarantee correctness. Bad retrieval, stale content, or weak generation can still produce unsupported claims.
More tokens are not always better. RAG helps choose what enters the context window, which can improve focus, enforce access boundaries, reduce cost, and make failures easier to diagnose.
Before building RAG: test a long-context baseline when the relevant knowledge set is bounded and fits comfortably in the model context. RAG earns its complexity when corpus scale, freshness, permissions, citations, filtering, or retrieval quality require explicit control.
Modern pipeline
The foundational two-phase blueprint still works—but production RAG has more control points
The original ingestion/inference split is preserved. The update adds governance, contextualization, routing, hybrid retrieval, reranking, verification, and fallback.
Phase 1 · Knowledge preparation
Offline / continuous1. Ingest & parseFiles, APIs, databases, HTML, PDFs, tables, images and transcripts.
2. Govern & normalizePermissions, tenancy, provenance, PII policy, canonical IDs and freshness.
3. Chunk & enrichChunk boundaries, parent links, headings, metadata, contextual headers and summaries.
4. RepresentDense embeddings, lexical terms, multi-vector representations, entities and relationships.
5. IndexVector, lexical, metadata, graph, parent-child, hierarchical or multimodal indexes.
Phase 2 · Query-time reasoning
Online6. RouteClassify intent, tenant, query type and approved data/tool path.
7. TransformRewrite, decompose, expand or create subqueries only when useful.
8. RetrieveDense, sparse, SQL, graph, web, document or multimodal candidate retrieval.
9. Fuse & rerankCombine candidate sets, remove duplicates and prioritize evidence.
10. Generate & verifyAnswer with evidence, citations, confidence rules, checks and fallback.
Architecture escalation ladder
RAG does not “mature” by becoming more complicated
Use this as a debugging ladder, not a ranking. Begin with the simplest architecture that can meet the target. Move to the next layer only when your evaluation set shows a specific failure.
Start left/simple. Add structure only for a measured reason.Patterns often combine; they are not mutually exclusive product tiers.
0
Long-context baseline
For bounded knowledge sets, test whether providing the relevant source material directly is simpler and good enough before building retrieval infrastructure.
1
Flat vector RAG + metadata
Embed chunks, retrieve top-k by semantic similarity, enforce metadata and permission filters, then generate from the retrieved evidence.
2
Hybrid retrieval + reranking + query transformation
Add lexical retrieval for exact terms, rank fusion for dense+sparse candidates, query rewriting/decomposition when phrasing is weak, and reranking when candidate ordering is noisy.
3
Context-aware and multi-level indexing
Use contextual retrieval, parent-child retrieval, hierarchical/RAPTOR-style indexing, or multi-vector late interaction when flat chunks lose meaning or fine-grained matching matters.
4
Structured and corrective retrieval
Add GraphRAG when relationships or corpus-level themes matter. Add CRAG or self-reflective retrieval when weak evidence must be detected and corrected instead of blindly passed to the model.
5
Bounded agentic and multimodal retrieval
Let a policy or agent select among approved vector, lexical, SQL, graph, web, document, image, or tool-based retrieval paths when a static pipeline cannot serve all query types.
Decision matrix
Choose the pattern from the failure mode—not from architecture fashion
This replaces the old radar chart and 1–10 complexity scores. The table uses observable symptoms and the evidence you should collect before promoting a pattern to production.
Observed failure
Pattern to test
What should improve
Exact identifiers, names, codes or citations are missed
Hybrid lexical + vector
Recall by query type, especially exact-term queries; measure fusion sensitivity and false positives.
The right evidence is retrieved but ranked below weak chunks
Reranking
nDCG/MRR, context precision and answer quality versus added latency and reranker cost.
Chunks become ambiguous after being removed from the source document
Contextual retrieval
Retrieval failure rate and downstream answer accuracy. Anthropic reported sizable gains in its own multi-domain experiments; reproduce them on your corpus.
Search needs small chunks, but answers need wider context
Parent-child retrieval
Retrieval precision using child chunks and answer completeness using parent context.
Questions span long documents or multiple levels of abstraction
Hierarchical / RAPTOR
Multi-hop or long-document QA improvement versus index-build and update cost.
Questions ask about relationships, neighborhoods, communities or themes
GraphRAG
Quality by relationship/global-query class versus graph extraction, indexing and query cost.
Retriever sometimes returns weak evidence and the pipeline cannot recover
CRAG / self-reflective RAG
Recovery rate, false correction rate, added calls, latency and unsupported-claim rate.
Different questions require different approved sources or retrieval tools
Agentic routing
Task success, tool-selection accuracy, step count, cost, loops, policy violations and fallback rate.
GraphRAG today
GraphRAG is not one search mode
Microsoft's current GraphRAG query engine distinguishes several modes. The right one depends on whether the question is local, global, or needs a broader starting point with iterative exploration.
Baseline vector RAG
Top-k text-unit retrieval. Useful as a control when graph structure is not necessary.
Entity-centric reasoning
Combines graph entities, relationships and linked source chunks for questions about specific entities.
Whole-corpus reasoning
Uses community reports and map-reduce style synthesis for themes or questions about the dataset as a whole.
Local + broader exploration
Starts with community information and expands into detailed follow-up retrieval to balance breadth with local detail.
Evaluation
The graph you need most is not “complexity”—it is evidence across the RAG system
Keep a versioned golden query set with known source evidence and segment it by query type. Diagnose retrieval and generation separately so you know which knob to change.
Retrieval
- Recall@k / hit rate
- MRR or nDCG
- Context precision & recall
- Filter / route accuracy
Generation
- Groundedness
- Citation correctness
- Answer completeness
- Unsupported-claim rate
System
- p50 / p95 latency
- Cost per successful answer
- Index freshness
- Retry / fallback rate
Business
- Task success
- Escalation / override rate
- User trust / adoption
- Time to resolution
RAGCHECKER is one example of a research framework built specifically to diagnose retrieval and generation components separately. The specific evaluation tool matters less than preserving component-level evidence across versions.
DataKnobs fit
Treat RAG architecture choices as measurable, governable Knobs
The evolution of RAG is really the growth of adjustable choices. Those choices should not disappear into code, prompts, or framework defaults.
Build ingestion, parsing, contextualization, embeddings, lexical search, graph/index creation, reranking and answer workflows as reusable capabilities.
Govern permissions, source boundaries, PII, lineage, freshness, citations, tool access, budgets, review and production evidence.
Experiment with chunk size, context generation, embedding model, dense/sparse fusion, top-k, reranker, graph mode, routing thresholds and agent autonomy.
Current references
Sources behind the 2026 update
These first-party documentation and research sources support the patterns and terminology used in the refreshed visual guide.
Microsoft GraphRAG query engineCurrent Basic, Local, Global and DRIFT search modes.
Anthropic — Contextual RetrievalContextual embeddings, contextual BM25, reranking, and long-context baseline guidance.
RAPTORTree-organized retrieval across multiple levels of abstraction.
Corrective RAG (CRAG)Retrieval evaluation and corrective actions when retrieved evidence is weak.
Self-RAGAdaptive retrieval and self-reflection instead of fixed retrieval for every query.
ColBERTv2Late-interaction retrieval with token-level representations.
RAGCHECKERFine-grained RAG evaluation across retrieval and generation modules.
Anthropic — Context engineeringWhy context selection and attention remain important even as model context windows grow.
FAQ
Common RAG architecture questions
Is agentic RAG always better than simple RAG?
No. Agentic retrieval adds routing, tool use, extra latency, cost and operational risk. Use it only when a fixed pipeline cannot reliably choose among heterogeneous sources or recover from failed retrieval.
When should I use hybrid search and reranking?
Use hybrid retrieval when exact terms and semantic meaning both matter. Add reranking when first-stage retrieval finds the right evidence but orders too many weaker candidates above it.
When is GraphRAG worth the extra cost?
Use it for query classes that depend on relationships, entity neighborhoods, multi-hop paths, communities or corpus-level themes. Evaluate its quality gains against graph extraction, indexing and query cost.
Should I test long context before building RAG?
Yes. Long context is a useful baseline for bounded knowledge sets. RAG becomes more valuable as corpus scale, freshness, permissioning, citations, filtering and retrieval observability become first-order requirements.