Why RAG Complexity Should Be Earned

Retrieval-Augmented Generation (RAG) architectures have expanded considerably beyond the original retrieve-and-generate pattern over the last few years. Contemporary systems increasingly incorporate dense and lexical retrieval, query rewriting, rank fusion, neural reranking, question decomposition, corrective retrieval, reflection, and agent-based orchestration. These techniques can materially improve performance on complex information-seeking tasks. However, they are also frequently introduced before the underlying retrieval subsystem has been independently evaluated.This article looks at how RAG complexity should be introduced in response to measured retrieval failure modes rather than adopted as an architectural default. Recent studies show that comparatively conventional retrieval methods remain highly competitive: lexical retrieval performs well in specialized domains, hybrid retrieval with reranking provides solid baselines, and even agentic systems perform better when they are built on top of stronger retrieval.The implication is not that agentic RAG is unnecessary. Rather, retrieval quality and agentic reasoning address different parts of the problem: when evidence can be recovered through a well-defined retrieval step, the main concern is usually search quality; when retrieval is iterative, multi-hop, or depends on intermediate evidence, agentic retrieval becomes much more useful.···Increasing Complexity in RAG ArchitecturesThe following is an increasingly common pattern in modern RAG systems:A retrieval problem appears. Before it has been established whether retrieval itself is functioning correctly, the architecture accumulates query rewriting, routing, multiple retrieval passes, reflection, corrective retrieval, an agent deciding whether additional evidence is required, a reranking model, and sometimes another model tasked with verifying the final answer.The resulting architecture appears sophisticated. However, the system may still fail for a much simpler reason: the relevant evidence was ranked outside the candidate set and never entered the model’s context. This distinction is important because retrieval and reasoning represent separate system capabilities.If the dominant failure is:The required evidence was not retrieved.then additional reasoning after retrieval after retrieval is unlikely to fix the root cause. It may instead increase latency, token consumption, nondeterminism, and the number of components requiring evaluation.This is not an argument against agentic RAG. There are information-seeking tasks for which planning, decomposition, heterogeneous source selection, and iterative retrieval are necessary.The argument is narrower:Architectural complexity should correspond to a demonstrated failure mode.Agency makes sense when conventional retrieval is structurally insufficient. However, when the evidence already exists in a retrievable unit but fails to enter the context window, there is a higher probability that the dominant problem lies within the search and retrieval subsystem.···Retrieval and Generation as Distinct System ComponentsA RAG system performs two conceptually separate operations:Retrieval: identifying information likely to contain the information necessary to answer a query.Generation: interpreting the retrieved information and constructing an appropriate response.These operations are often evaluated together because the generated answer is the user-visible output. From a system-design perspective, however, their failure modes should be separated.Consider a query against a collection of commercial agreements:What termination provisions apply if the supplier repeatedly fails its SLA?Assume that the retriever returns:a paragraph describing general supplier obligations,a payment clause,several passages containing the phrase service level,and a definition of contractual breach.The actual termination clause though, is ranked outside the retrieved top-k set.A sufficiently capable language model may still produce a plausible answer based on general contractual patterns. It may even sound correct. The system has nevertheless failed at retrieval. No change to the generation prompt can recover evidence that never entered the model’s context.This leads to a fundamental diagnostic question for RAG systems:Was the evidence required to answer the query present within the retrieved candidate set?That question should generally be answered before modifying the reasoning or generation layer.Figure 1. Retrieval and generation represent separate sources of failure. End-to-end answer accuracy alone cannot identify which subsystem caused an incorrect result. Image by author. ···Lexical Retrieval Remains a Competitive BaselineThe widespread adoption of embedding-based retrieval has sometimes produced an implicit assumption that semantic similarity is a strict successor to lexical search. Empirical evidence does not really support that interpretation. BM25 remains a highly competitive retrieval mechanism, particularly in domains where exact lexical matches carry significant information.Its ranking function can be expressed approximately as:BM25⁡(D,Q)=∑qi∈QIDF(qi)f(qi,D)(k1+1)f(qi,D)+k1(1−b+b∣D∣avgdl⁡)\operatorname{BM25}(D,Q)= \sum_{q_i\in Q} IDF(q_i) \frac{f(q_i,D)(k_1+1)} {f(q_i,D)+k_1\left(1-b+b\frac{|D|}{\operatorname{avgdl}}\right)}where f(qi, D) represents the frequency of query term qi within document D, while the remaining terms account for factors such as term-frequency saturation and document-length normalization.The exact formulation is less important here than the distinction between lexical and semantic retrieval. Lexical search is especially effective when queries contain identifiers or terminology where exact matching matters, such as error codes, contractual identifiers, legal citations, product numbers, function names, ticker symbols, acronyms, medical terminology, database fields, or precise names and dates.For example, an embedding model may correctly infer that TS-999 refers to a technical error, but still rank a semantically similar passage above the only document containing the literal identifier TS-999. BM25 has the opposite bias: exact lexical evidence is strongly rewarded. BM25 has the opposite bias: exact lexical evidence is heavily rewarded.This matters even more in specialized corpora, where terminology and identifiers often carry more weight than broad semantic similarity. A 2026 benchmark covering 23,088 financial questions and 7,318 mixed text-and-table documents compared ten retrieval strategies, including sparse retrieval, dense retrieval, hybrid fusion, reranking, query expansion, contextual retrieval, and adaptive retrieval. In that setting, BM25 outperformed the evaluated state-of-the-art dense retrieval method.The appropriate conclusion is not that lexical retrieval is generally superior to embeddings. It is that lexical and semantic retrieval solve different retrieval failures.···Hybrid Retrieval and Two-Stage Ranking ArchitecturesThe strength of lexical retrieval, however, does not remove the need for semantic search. It highlights a different limitation: lexical methods work best when the query and the source share enough vocabulary to match reliably. Dense retrieval becomes valuable when lexical overlap is weak.A user might ask:Under what circumstances can an employee resign voluntarily?while the underlying policy refers only to:employee-initiated termination.Lexical retrieval may struggle because the wording is different. Dense embeddings can capture that semantic relationship and recover the relevant material.This is where hybrid retrieval becomes useful. Rather than choosing between lexical and semantic search, the system can use both to generate candidates and then combine their rankings.A representative architecture is:Figure 2. A two-stage hybrid architecture uses inexpensive retrieval mechanisms to maximize candidate recall before applying a more computationally expensive reranker to optimize precision. Image by author. One common approach to combining rankings is Reciprocal Rank Fusion:RRF(d)=∑r∈R1k+rank⁡r(d)RRF(d)=\sum_{r\in R}\frac{1}{k+\operatorname{rank}_r(d)}Instead of directly comparing BM25 scores with embedding similarity scores, which are not necessarily on comparable scales, RRF combines results based on relative rank.The resulting candidate pool can subsequently be evaluated by a cross-encoder or other reranker. This distinction between candidate generation and reranking is important. The first retrieval stage is primarily responsible for recall. It should recover a broad enough candidate set that relevant material is unlikely to be discarded. The reranker operates on a substantially smaller set and can therefore spend more computation estimating relevance.For example:Figure 3. Two-stage hybrid retrieval architecture combining lexical and dense candidate generation, rank fusion, and cross-encoder reranking before context assembly.Recent evidence supports this approach. The 2026 financial retrieval benchmark mentioned earlier found that the best-performing approach was not BM25 alone. A two-stage architecture combining hybrid retrieval and neural reranking achieved Recall@5 of 0.816 and MRR@3 of 0.605, outperforming the evaluated single-stage methods by a substantial margin.Anthropic reported a similar pattern in its Contextual Retrieval experiments. Combining contextual embeddings with contextual BM25 reduced top-20 retrieval failures by 49% relative to its baseline, while introducing reranking increased the reduction to 67%.These results support a relatively conventional conclusion:retrieval mechanisms are often complementary rather than mutually exclusive.The newest component is not necessarily a replacement for the older one. In many cases, the strongest architecture comes from combining their respective strengths.···Document Representation as a Retrieval ConstraintSo far, we have talked about retrieval methods. However, retrieval quality is determined partly before a retrieval query is ever issued.Parsing, document segmentation, metadata propagation, table handling, and chunk construction determine the units over which retrieval operates.Consider the following source document:A fixed-token splitter may produce:and:Chunk 42 still contains the critical factual condition but it has lost the information required to identify what that condition refers to. Once that structure is removed during chunking, the embedding model cannot reliably reconstruct it.Similar failures occur when:headings are separated from their sections,table headers are removed from table values,document titles disappear from chunks,parent-child relationships are discarded,timestamps or reporting periods are removed,access-control metadata is not propagated,PDF layout is flattened incorrectly.This makes chunking more than a token-management problem. It is also an information representation problem.Anthropic’s Contextual Retrieval experiments explicitly address this issue by prepending short document-derived context to each chunk before indexing. In its reported experiments, contextual embeddings reduced top-20 retrieval failure from 5.7% to 3.7%. Combining contextual embeddings and contextual BM25 reduced it further to 2.9%, and reranking reduced it to 1.9%.The broader implication is more important than the specific technique:Retrieval quality begins at ingestion, not at query time.An increasingly elaborate query-time architecture cannot fully compensate for information that has been structurally degraded during indexing.···The Appropriate Role of Agentic RetrievalThere are information needs for which static retrieval is genuinely insufficient, and this is where agentic retrieval becomes useful. Consider a financial analysis query:Which company had the higher operating margin in 2025, Company A or Company B, and what did each company identify as the primary cause of its year-over-year change?No individual passage necessarily contains the complete answer.The system may need to:identify the appropriate reporting period for Company A,retrieve Company A’s operating margin,retrieve management commentary explaining the change,repeat the process for Company B,reconcile differences in terminology or reporting periods,compare the resulting evidence.The limitation here is not simply poor retrieval quality. The information need itself requires multiple dependent retrieval steps.A reasoning layer could transform the original request into several retrieval operations:The resulting evidence can then be merged and reranked before generation. Question decomposition has empirical support in this class of problem. A 2025 study evaluated an LLM-based decomposition and reranking pipeline on MultiHop-RAG and HotpotQA and reported a 36.7% improvement in MRR@10 and an 11.6% improvement in answer F1 relative to standard RAG baselines.This is an appropriate use of additional reasoning because the failure originates in the structure of the information need.Other defensible cases for agentic retrieval include:Heterogeneous source selectionA system may need to determine whether the requested information belongs in:a relational database,a document index,a knowledge graph,an internal API,a code repository,or an external search source.In such cases, the system must first decide where to retrieve from before it can decide what to retrieve.Evidence-dependent retrievalThe next query cannot be formulated until an intermediate result has been observed.A fixed pipeline is less well suited to this kind of branching search because the retrieval path emerges during execution.Ambiguity resolutionAn initial search may surface multiple plausible interpretations of a query. Additional retrieval may then be required to resolve the ambiguity, narrow the search space, or determine whether clarification is needed.Multi-hop evidence aggregationSome questions can only be answered by combining facts distributed across multiple documents or systems, where one intermediate fact is required to locate the next.In all of these cases, agency adds something that a fixed retrieval pipeline may not express cleanly: adaptive control over the retrieval process itself.This leads to a more useful design question:What specific retrieval failure requires adaptive reasoning?That framing is more useful than treating RAG and agentic RAG as competing architectural categories. In some cases, conventional retrieval is entirely sufficient. In others, the retrieval process itself is iterative, conditional, or multi-step, and the additional complexity is justified by the problem.···Retrieval Quality as a Constraint on Agentic ReasoningOne of the more interesting recent results suggests that stronger retrieval can make agentic systems substantially better. A 2026 scaling study compared lexical, dense, graph-based, and agentic retrieval across 28 nested corpus sizes ranging from approximately 1,000 to 512,000 documents.BM25 occupied the low-cost end of the Pareto frontier at every measured scale and led accuracy from the middle corpus sizes onward. The raw file-system agent was competitive at small scale but deteriorated considerably as corpus size increased, while consuming substantially more query-time tokens.The most informative result, however, appeared when the retrieval mechanism underneath the agent was changed.At full scale:Figure 4. Results from https://arxiv.org/abs/2607.26497The stronger retrieval layer did not make the agent redundant. It made the agent considerably better. This result helps separate two capabilities that are often discussed as though they are interchangeable. Retrieval determines what evidence becomes available to the system; reasoning determines how that evidence is used and what should happen next. A weak retrieval substrate limits the evidence available to an agent, while a stronger one gives the same reasoning layer a better basis for subsequent decisions.The relationship is therefore better represented as:rather than:The study also illustrates why debates framed as BM25 versus agents or classical RAG versus agentic RAG are often conceptually unhelpful since they operate at different levels of the architecture.···Computational and Operational Costs of Adaptive RetrievalWe have made a case so far that additional retrieval complexity can be justified when it addresses a specific limitation. That architectural complexity, however, has costs beyond inference spend.A relatively constrained pipeline may execute:An adaptive architecture might execute:The latter architecture introduces obvious token and latency costs. More importantly, it introduces a larger failure surface. An incorrect response may originate from:intent classificationquery decompositiontool selectionquery rewritinglexical retrievaldense retrievalfusionrerankingevidence-sufficiency judgementstopping criteriagenerationverificationEach adaptive branch also introduces additional nondeterminism. The operational problem therefore changes from:Did the model answer correctly?to:Which trajectory produced the answer, and which component was responsible for failure?This has direct implications for observability. Production agentic retrieval systems increasingly require traces containing:Without this information, an improvement in end-to-end accuracy may conceal a deterioration elsewhere in the system. A more complex architecture is therefore justified not merely when it improves an evaluation score, but when the improvement is large enough to justify its additional latency, inference cost, operational burden, and failure modes.···Independent Evaluation of Retrieval and GenerationEnd-to-end answer accuracy alone is insufficient for diagnosing RAG systems.Four broad outcomes are possible:Figure 5. Retrieval and Generation failure modes and their interpretationThe third case is particularly dangerous.A model may correctly answer a question using parametric knowledge even though the RAG system failed to retrieve supporting evidence. An answer-only evaluation may record a success. However, a grounded enterprise system should generally record a failure.Retrieval should therefore be evaluated independently using conventional information-retrieval metrics.Recall@kRecall@k measures how much of the relevant evidence appears within the first (k) retrieved items.Recall@k=∣Relevant documents in top-k∣∣Relevant documents∣Recall@k= \frac{|\text{Relevant documents in top-k}|} {|\text{Relevant documents}|}For many RAG applications, recall is a critical first-stage metric because evidence discarded before generation cannot subsequently be recovered.Precision@kPrecision@k measures the fraction of retrieved items that are relevant.Precision@k=∣Relevant documents in top-k∣kPrecision@k= \frac{|\text{Relevant documents in top-k}|}{k}High recall accompanied by very poor precision creates a different failure mode: the required evidence is present, but it is surrounded by enough irrelevant context to degrade model performance.Mean Reciprocal RankMean Reciprocal Rank evaluates how early the first relevant result appears:MRR=1∣Q∣∑i=1∣Q∣1rankiMRR= \frac{1}{|Q|} \sum_{i=1}^{|Q|} \frac{1}{rank_i}A system that consistently retrieves the correct document at position 18 has a materially different operating profile from one that ranks it first.Normalized Discounted Cumulative GainnDCG becomes useful when relevance is graded rather than binary. Highly relevant evidence appearing near the top of the ranking receives greater value than weaker evidence appearing later.Generation can then be evaluated separately using dimensions such as:factual correctness,completeness,faithfulness to retrieved evidence,citation correctness,abstention behavior,contradiction handling.This separation enables more useful diagnosis.···Evaluation Datasets Must Represent Production Retrieval ConditionsGood metrics are still insufficient if the evaluation corpus does not reflect production behavior. Real queries contain:spelling errors,incomplete entity names,unexplained acronyms,terminology mismatches,ambiguous references,temporal constraints,contradictory documents,missing information,questions for which no answer exists.Evaluation sets constructed from clean documents and equally clean synthetic questions often underestimate retrieval difficulty.A mature RAG evaluation suite should therefore contain several classes of queries:Different retrieval strategies are likely to fail on different subsets, and a single aggregate score may conceal architectural weaknesses. A dense retriever may perform well on semantic paraphrases while struggling with identifiers; BM25 may show the opposite pattern. An agentic system may perform particularly well on multi-hop questions while adding unnecessary cost to straightforward factual lookups. The purpose of evaluation should therefore not merely be to identify the architecture with the highest aggregate number. It should also identify which failure modes each architectural component resolves, and where it introduces new trade-offs.···A Progressive Architecture for RAG System ComplexityA more useful approach to RAG design is to treat complexity as an escalation path.Each additional mechanism should correspond to evidence that the preceding architecture cannot adequately resolve an important class of queries.Figure 6. A progressive architecture for RAG systems. Higher levels are not inherently superior; each introduces capabilities that should correspond to an observed limitation at lower levels. Image by author. Level 0: Establish Whether Retrieval Is RequiredNot every knowledge-grounded application requires a retrieval subsystem.For sufficiently small and stable corpora, directly supplying the source material may be operationally simpler. Anthropic has noted, for example, that knowledge bases below approximately 200,000 tokens may in some circumstances be supplied directly to the model rather than retrieved dynamically.The exact threshold depends on the model, workload, latency requirements, and cost profile, but the broader architectural principle still holds: retrieval should be introduced because the application requires it, not simply because the system is being described as RAG.Level 1: Establish Corpus RepresentationIf retrieval is required, the next step is to make sure the corpus is represented correctly. Before retrieval optimization, the system should:validate parsing,preserve document hierarchy,propagate metadata,handle tables explicitly,establish meaningful chunk boundaries,preserve parent-child relationships where necessary,apply access-control metadata during ingestion.Level 2: Establish a Lexical BaselineA lexical baseline provides a useful reference point. BM25 is computationally inexpensive, interpretable, and particularly strong where exact terminology matters.Its purpose is not to become the final retrieval architecture. It establishes whether later additions produce measurable improvements over a competent conventional baseline.Level 3: Introduce Dense RetrievalDense retrieval is justified when evaluation shows meaningful lexical-semantic mismatch. At this stage, the important comparison is not only downstream answer quality, but also the candidate sets returned by lexical and dense retrieval.If dense retrieval consistently recovers relevant evidence that lexical retrieval misses, the additional complexity is addressing a measurable failure mode.Level 4: Introduce Hybrid RetrievalHybrid retrieval becomes justified when lexical and dense methods demonstrate complementary recall. Rather than selecting one approach as the default, the system can combine their respective strengths.This is often the point at which the retrieval architecture becomes more robust across a wider range of query types.Level 5: Introduce RerankingIf candidate recall is already strong but the ordering of results remains weak, reranking becomes the next logical step.The retrieval layer can remain broad and recall-oriented, while a more expensive reranker focuses on improving top-k precision over a smaller candidate set.Level 6: Improve RepresentationIf failures persist, the problem may still lie in how the corpus is represented rather than in the retrieval algorithm itself. Observed failure modes may justify contextual chunks, parent-document retrieval, domain-specific embeddings, late-interaction architectures, table-specific indexing, or alternative segmentation strategies.At this stage, the question is less about adding another retrieval mechanism and more about improving the information that those mechanisms operate over.Level 7: Introduce Query TransformationQuery rewriting and decomposition become useful when evaluation shows systematic query-document mismatch or when the information need contains multiple separable components.Level 8: Introduce Agentic RetrievalAgency becomes justified where the retrieval process itself requires adaptive decisions:selecting among heterogeneous sources,deciding what information to obtain next,using retrieved evidence to formulate subsequent searches,determining whether sufficient evidence has been collected,executing variable-length multi-hop retrieval trajectories.At this point, the architecture is not adding an agent merely because agents are available. It is introducing adaptive control flow because the information-seeking problem requires it.···Fixed and Adaptive Retrieval Should CoexistA further implication follows from this framework: not every query sent to an agentic RAG system should necessarily invoke an agentic workflow.A production system may instead distinguish between query classes.Figure 7. A heterogeneous RAG system can route different information needs to retrieval strategies with different computational and reasoning requirements. This architecture treats agentic retrieval as one capability within a larger retrieval system rather than as the universal execution path.A query such as:What is the cancellation period in Contract 4827?may require little more than lexical and metadata-constrained retrieval.A query such as:Compare the cancellation obligations across the current contracts for suppliers responsible for the three services with the highest SLA violation rate last quarter.is structurally different. It may require:querying operational data,identifying suppliers,retrieving multiple contracts,locating relevant clauses,normalizing terminology,comparing evidence.Applying identical retrieval orchestration to both requests is difficult to justify. A more robust system should route queries according to the structure of the information need, using simpler retrieval where it is sufficient and adaptive retrieval where the task genuinely requires it.···Complexity Should Follow Demonstrated FailureThe central issue is not whether advanced RAG techniques work. Many of them clearly do. The problem arises when these mechanisms become architectural defaults rather than responses to observed limitations.LLMs make architectural augmentation unusually easy. If retrieval performs poorly, an additional model can rewrite the query. If the rewritten query fails, another retrieval pass can be introduced. If the candidate set is noisy, a model can grade it. If evidence appears incomplete, a reflection stage can initiate another search. If the final response remains incorrect, another model can verify it.Each addition may improve some queries while also obscuring a defect earlier in the pipeline. A sufficiently complicated reasoning system can therefore produce better end-to-end results while simultaneously making the architecture more difficult to understand, evaluate, and operate.The relevant engineering objective is not architectural minimalism.It is architectural justification.The presence of a component should be explainable in terms of a measured system limitation:Hybrid retrieval exists because dense and lexical retrieval exhibit complementary recall on the target corpus.Reranking exists because candidate recall is sufficient while top-k precision is inadequate.Query decomposition exists because multi-hop questions systematically fail under single-query retrieval.An agent exists because subsequent retrieval actions depend on evidence discovered during execution.This framing converts RAG architecture from a collection of currently popular techniques into a sequence of testable engineering decisions.···ReferencesAnthropic. Introducing Contextual Retrieval. 2024. Experiments examining contextual embeddings, contextual BM25, hybrid retrieval, and reranking across multiple knowledge domains.Akarsu, M., Karaman, R. K., & Mierbach, C. From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents. 2026. Benchmark of ten retrieval strategies across 23,088 financial QA queries and 7,318 documents.Wang, P., Xu, B., Wang, S., et al. Which RAG Paradigm Wins at Scale? A Scaling Study of Retrieval-Augmented Generation Paradigms. 2026. Controlled comparison of lexical, dense, graph-based, and agentic retrieval over corpus sizes ranging from approximately 1,000 to 512,000 documents.Ammann, P. J. L., Golde, J., & Akbik, A. Question Decomposition for Retrieval-Augmented Generation. 2025. Evaluation of LLM-driven question decomposition and reranking on MultiHop-RAG and HotpotQA.