AI Architecture

Retrieval is the product.

Everyone demos naive RAG in an afternoon. Then production arrives: exact-match queries miss, chunks lack context, and the LLM confidently answers from the wrong paragraph. The distance between a demo and a product is retrieval strategy.

↓ Why naive RAG breaks ↓ Skip to the decision tree

Six architectures, one decision: how much retrieval machinery does your problem actually need? More is not better — matched is better.

In this piece

The baseline
The landscape
The strategies
The decision
Sources

Ali Reza Rashidi8 min readUpdated 6 August 2026

01—The baseline

Naive RAG — and where it breaks#

The starter architecture is four steps and genuinely useful — for the sixty percent of questions that are easy. The other forty percent is why this page exists.

01ChunkSplit documents into passages — usually by fixed size, blind to meaning.

02EmbedTurn each chunk into a vector; park them in a vector database.

03RetrieveEmbed the question; fetch the top-K nearest chunks.

04GenerateStuff the chunks into the prompt; the LLM answers.

Exact terms miss“Error E47” has no semantic neighborhood — pure vector search shrugs; keyword search wouldn’t.

Chunks lack contextA paragraph about “it” and “the figure” means nothing without its neighbors.

One shot, no reflectionBad retrieval in, confident answer out — nothing checks whether the context was even relevant.

No relationship map“How are A and B connected?” needs graph traversal, not nearest neighbors.

02—The landscape

Six architectures on one map#

Every strategy is a point on the same trade: latency and money in, answer quality out. Plot them honestly and the hype sorts itself.

The cost–quality frontier

How to read this: the knee of the curve is the advanced trio — most of the quality for a fraction of the cost. Everything right of it must argue for its budget.

Architecture	Latency	Cost / query	Reach for it when
Naive	100–500 ms	~$0.001–0.01	Docs are clean, questions are simple, volume is high
Advanced trio	0.5–2 s	~$0.003–0.01	The enterprise default — fixes most naive failures
Query transforms	+200–600 ms	+1–3 LLM calls	Questions are vague, multi-part, or vocabulary-mismatched
GraphRAG	1–5 s	~$0.02–0.15	Answers need multi-hop relationships across the corpus
Self-RAG / CRAG	+1–4 s	+2–5 LLM calls	Wrong retrievals are expensive; reflection pays for itself
Agentic	2–10+ s	~$0.01–0.10	Multi-step questions needing tools, iteration, planning

80% of naive-RAG failures typically fixed by the advanced trio — chunking, hybrid, rerank (illustrative)

300–500 tokens per chunk — the sweet-spot band for most enterprise documents

15% typical chunk overlap, so sentences don’t die at the boundaries

4 RAGAS dimensions to evaluate: faithfulness, answer relevance, context precision & recall

03—The strategies

Each one, properly#

Six architectures, each with a home turf, a bill, and a failure mode. The trick is not knowing them — it is knowing which one your problem is.

Three upgrades fix the bulk of naive failures. Semantic chunking splits documents at meaning boundaries (300–500 tokens, 10–15% overlap) instead of arbitrary character counts. Hybrid search runs BM25 keyword matching beside dense vectors and fuses ranks — exact terms like “Error E47” finally land. Reranking re-scores the top 50 candidates with a cross-encoder and keeps the best 5.

What it kills

Exact-term misses: product codes, error strings, names.
Orphan chunks: paragraphs that lost their context.
Near-miss retrieval: vaguely relevant, specifically wrong.

What it costs

Latency: roughly 0.5–2 s end to end.
Money: a reranker call (~$1 per 1K searches) plus embedding upkeep.
Complexity: two indexes and a fusion step to maintain.

Most enterprise RAG is fine right here.Master the trio before reaching for graphs or agents. Boring, measured, working beats exotic, unmeasured, demoed.

Sometimes the bottleneck is the query itself. HyDE has the LLM draft a hypothetical answer first and embeds that — answers live nearer to answers than questions do. Multi-query rewrites the question three to five ways and fuses the retrievals. Decomposition splits “compare X and Y across Z” into sub-questions answered separately, then merged.

When to reach for it

Vocabulary mismatch: users say “refund,” docs say “credit memo.”
Multi-part questions: two comparisons hiding in one sentence.
Vague questions: “the thing we discussed last quarter” has no embedding neighborhood.

Cost: a few hundred milliseconds and one to three extra LLM calls.Cheap insurance when the question — not the index — is the weak link.

GraphRAG has the LLM extract entities and relationships into a knowledge graph at index time, precompute community summaries, and answer by traversing structure instead of nearest neighbors. “What connects our top three churned accounts?” is a graph question — no single chunk holds it, so chunk retrieval can’t see it.

Built for

Multi-hop questions: A affects B affects C.
Global questions: “what are the main themes in this corpus?”
Investigative work: intelligence, legal, research corpora.

The bill

Indexing: an LLM pass over the whole corpus — heavy.
Query cost: roughly $0.02–0.15 and 1–5 s per question.
Freshness: new documents mean graph maintenance, not just new rows.

If your questions are about single documents, a graph is an expensive ornament.GraphRAG earns its cost only when relationships are the product.

Self-RAG trains the model to critique itself: should I retrieve at all? Is this context relevant? Is my answer actually supported by it? CRAG adds a grader that scores retrieved documents and falls back to web search when confidence runs low. Both turn one-shot retrieval into a checked loop — you pay in latency and win in trust.

Self-RAG

Reflection tokens: retrieve? relevant? supported?
Best when wrong answers cost more than slow ones.

CRAG

Grades retrieval confidence per document.
Falls back to web search when the corpus comes up short.

Reflection is a latency-for-trust trade.Expect one to four extra seconds and two to five extra model calls. Worth it where a confident wrong answer is the expensive outcome.

A classifier reads each incoming question and picks a strategy: simple → answer directly with no retrieval; medium → single retrieval pass; complex → multi-step with decomposition or agents. Skipping retrieval for easy questions saves money and accuracy — retrieval can pollute a question the model already knows cold.

Why it wins at scale

Cost follows difficulty: easy questions stop paying the complex-question tax.
Latency follows difficulty too: simple answers in milliseconds, research questions in seconds.
One front door: users stop choosing modes; the router chooses.

The router itself must be evaluated.A misrouted complex question gets a cheap, wrong answer — the worst cell in the matrix. Gate routing accuracy like any other metric.

Here the loop owns the tools. An agent plans the research, issues searches, reads what comes back, decides what to fetch next, and stops when it can defend an answer. This handles the genuinely multi-step questions — “compare our Q3 retention cohorts against the pricing change timeline” — that break every fixed pipeline.

Built for

Multi-step research: questions that need a plan, not a lookup.
Tool mixing: retrieval plus SQL plus calculators plus APIs.

The bill

Latency: 2–10+ seconds, occasionally much more.
Cost: roughly $0.01–0.10 per query, variable by design.
Governance: evals, logs, and a cost ceiling — it’s an agent, supervise it like one.

Agentic RAG is RAG inside an agent — not an agent inside your RAG.If your questions are lookups, this is a very expensive way to be slow.

04—The decision

A tree, not a hype cycle#

Choose by question shape, not by what’s trending. Four questions route almost every workload.

Which RAG are you?

Does one good chunk answer the question?

Stay naive — plus a reranker. Spend your budget elsewhere.

Do answers span two or three documents?

The advanced trio: semantic chunking, hybrid search, reranking. This is most enterprise RAG, done right.

Are answers about relationships and connections?

GraphRAG earns its indexing cost here. Multi-hop questions are its home turf.

Multi-step, tool-using, plan-then-search?

Agentic RAG — with evals, logs, and a cost ceiling. Supervise it like any agent.

A mixed bag of all of the above?

Adaptive RAG: route each query to the cheapest strategy that can handle it.

The meta-rule: start with the advanced trio, measure with RAGAS, and add machinery only when the evals demand it.

05—Grounding

Sources#

Latency and cost figures on this page are practitioner ranges; the architecture claims below are peer-reviewed or primary. Here they are.

Lewis et al. (2020) — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. The paper that named the four-step baseline: chunk, embed, retrieve, generate. arXiv:2005.11401
Gao et al. (2023) — Retrieval-Augmented Generation for Large Language Models: A Survey. The reference taxonomy behind this page’s naive → advanced → modular progression. arXiv:2312.10997
Gao et al. (2022) — HyDE: Precise Zero-Shot Dense Retrieval without Relevance Labels. ACL 2023. Hypothetical-document embeddings: embed the imagined answer, not the question. arXiv:2212.10496
Asai et al. (2023) — Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. ICLR 2024. Reflection tokens for retrieve?/relevant?/supported? — the self-checking loop. arXiv:2310.11511
Yan et al. (2024) — Corrective Retrieval Augmented Generation (CRAG). A retrieval grader with web-search fallback when corpus confidence runs low. arXiv:2401.15884
Edge et al. (2024) — From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Microsoft Research. Entity/relationship extraction plus community summaries — the GraphRAG this page prices. arXiv:2404.16130
Es et al. (2023) — RAGAS: Automated Evaluation of Retrieval Augmented Generation. EACL 2024 demo. The four dimensions in the stat strip: faithfulness, answer relevance, context precision and recall. arXiv:2309.15217

RAG Strategies

Advantages of data science

Agentic AI vs AI Agent

Retrieval is the product.

Naive RAG — and where it breaks#

Six architectures on one map#