Routing precedes retrieval, not the embedding dimension

KDnuggets' June 2026 survey of retrieval-augmented generation failures reports a global manufacturing company that budgeted $400K for its RAG system, spent $1.2M in year one, and landed at 23% accuracy on technical-documentation queries before terminating the project. The article's diagnosis is structural, not tuning-level: retrieval irrelevance, context poisoning, and a chunk-size conflict that no embedding dimension resolves. Its prescription is four architectures chosen by query type, with the load-bearing line "the key change is making routing explicit. Every query gets classified before any retrieval runs." (Nate Rosidi, KDnuggets, "Your RAG Pipeline Is Probably Useless. Here's a Better Alternative", published 2026-06-29, retrieved 2026-08-23, https://www.kdnuggets.com/your-rag-pipeline-is-probably-useless-heres-a-better-alternative). The Honest Architect reads the piece as Theorem 3 applied to retrieval: a property (factual relevance) is guaranteed exactly when its mechanism (explicit routing before retrieval) is implemented and measuring — and adding embedding dimensions to a design that lacks that mechanism makes the failure more expensive, not less.

[UNIQUE INSIGHT]: The over-engineering trap is Theorem 3 in miniature. You cannot guarantee factual relevance by adding more of a mechanism (higher-dimensional embeddings, more reranking, multi-step retrieval) that does not implement relevance. Relevance is a routing property — does this query belong to this corpus, this version, this document — not a similarity property. Scaling the wrong mechanism compounds the cost without producing the property.

Key takeaways

• Retrieval irrelevance is a missing-mechanism failure. Theorem 3: the property factual-relevance is guaranteed by routing the query to the right corpus, version, and document type before retrieval, not by vector similarity. The article: a parental-leave query returns the 2022 policy, the 2024 policy, and a cultural blog post, all high on embedding distance, none answering. Production ✅ for the diagnosis.
• Context poisoning is a missing-mechanism failure. Theorem 3: the property contradiction-surfaced is guaranteed by version routing plus contradiction detection, not by blending chunks. The article: when the retriever returns contradictory policy versions, the model "picks one, blends both, or presents a confident synthesis" and "neither the user nor the model knows it." Production ✅ for the diagnosis.
• The chunk-size conflict is structural, not tunable. Recall needs 100–256-token chunks; coherence needs 1024 or more. No embedding dimension resolves a trade-off where the two properties pull the parameter in opposite directions. Production ✅ for the structural claim.
• Over-engineering compounds the failure. The article: a 72% first-year failure rate for enterprise RAG in 2025, a $400K-to-$1.2M budget overrun, 23% accuracy, $75K per month in vector-database costs by month six at a healthcare enterprise. Adding complexity to a broken retrieval design "raises compute costs and delays the more useful question, which is whether the retrieval architecture was the right choice at all." Partial ⚠️ (figures survey-cited, not independently verified by Everythink).
• Routing precedes retrieval. Self-Route (EMNLP 2024) classifies whether a query needs full context or focused retrieval before retrieval runs; 15 to 30% precision improvements reported for hybrid search and reranking. Theorem 3: the property right-strategy-per-query is guaranteed by classify-before-retrieve, not by committing to one strategy at build time. Production ✅ for the form; Partial ⚠️ for the specific percentages.
• "The space is the router" is the upstream routing mechanism RAG lacks. Everythink's network→community→room topology routes a question to the right context before anything responds — the same form as Self-Route, one domain upstream. Partial ⚠️ (same form — route-before-respond — separate domains).
• Scope: civil/defensive. Retrieval architecture is civil infrastructure. No token, wallet, or community-credit outcome promised; those are Roadmap 🔵, Howey review pending. Everythink is a forecasting platform, not a RAG vendor; the cross-domain parallels are Partial ⚠️ illustrations, not endorsements of KDnuggets, StrataScratch, Microsoft GraphRAG, or any specific tool.

When RAG fails: the property is not guaranteed

The article opens with the failure that demos hide. A user asks about parental leave. The retriever returns the 2022 version, the 2024 version, and a cultural blog post. Each chunk scores high on embedding distance because it shares vocabulary with the query. None answers the question. The model does not know the retrieved content is outdated or off-topic; it blends the chunks into a confident, detailed, factually wrong answer. The article calls this "topical similarity without factual relevance, and it is the dominant failure mode in production RAG systems."

The Honest Architect reads this as Theorem 3. The property is factual-relevance. The mechanism the system implements is vector-similarity, which guarantees answer-resembles-question, not factual-relevance. The two coincide when the corpus is small, current, and single-version — the demo case. They diverge when the corpus holds multiple versions, off-topic vocabulary matches, and contradictions. The system asserts relevance by retrieving similar chunks; the mechanism does not implement relevance, so the property is not guaranteed. Production ✅ for the diagnosis.

The subtler failure, context poisoning, is the same form. Enterprise knowledge bases hold the same policy in multiple versions. The retriever returns chunks from both. The model "does not surface the contradiction. It picks one, blends both, or presents a confident synthesis. The reader gets an answer. The answer may be wrong. Neither the user nor the model knows it." The property is contradiction-surfaced. The mechanism is blend-and-synthesize. No mechanism detects the contradiction, so the property is not guaranteed. The system produces an answer; it does not produce a known-correct answer. Production ✅.

[PERSONAL EXPERIENCE]: Building the HAI Engine since 2016, we learned the same lesson in a different domain. A forecast is not made relevant by computing it over more data; it is made relevant by routing the question to the right room — the right community, the right scoped context — before any model runs. Scaling the model over the wrong context produces a confident, detailed, off-target forecast, the way scaling embeddings over the wrong chunks produces a confident, detailed, off-target answer. The mechanism that produces relevance is routing, not volume. Production ✅.

The over-engineering trap is Theorem 3 in miniature

The article's most useful section is the one engineers skip. When standard RAG underperforms, the common fix is to make it more complicated: higher-dimensional embeddings, more sophisticated reranking, multi-step retrieval. The article's verdict: "This compounds the problem."

The data is blunt. A global manufacturing company budgeted $400K, spent $1.2M in year one, landed at 23% accuracy, and terminated the project. A healthcare enterprise hit $75K per month in vector-database costs by month six. The article cites a 72% first-year failure rate for enterprise RAG in 2025. The Honest Architect reads these as the cost of scaling the wrong mechanism. Higher embedding dimensions implement finer-grained similarity, not factual-relevance. Reranking implements re-ordering, not contradiction-surfaced. Multi-step retrieval implements retrieve-then-retrieve-again, not route-before-retrieve. Each adds compute to a design that still lacks the missing mechanism, so the property stays unguaranteed while the bill grows. Partial ⚠️ on the figures (survey-cited, not independently verified by Everythink); Production ✅ on the form — scaling a mechanism that does not implement the property cannot produce the property.

This is Theorem 3 in miniature. A property is guaranteed exactly when its mechanism is implemented and measuring. Relevance is the property. Routing before retrieval is the mechanism. Embedding dimension is a different mechanism, measuring a different property. You cannot buy relevance with embedding dimensions any more than you can buy fire-resistance with paint thickness. The article's line — "raises compute costs and delays the more useful question, which is whether the retrieval architecture was the right choice at all" — is the operational version of the theorem. Production ✅.

The article also names a structural conflict no tuning resolves: recall needs small chunks (100–256 tokens), coherence needs large ones (1024 or more), and every RAG designer picks one and accepts the trade-off. The Honest Architect reads this as the boundary of the chunk-embed-retrieve mechanism — not a tuning problem but a mechanism-selection problem. The four alternatives are not patches on that conflict; they are different mechanisms, each guaranteeing a different property for a different query type. Production ✅ for the framing; the tool names (Self-Route, GraphRAG) are Partial ⚠️ (research-reported, not independently verified by Everythink).

The four alternatives are routing by situation

Long-context: skip retrieval when the corpus fits

The article's first alternative is to skip retrieval entirely. If the corpus fits the model's context window, load it and let the model read. A benchmark (arXiv 2501.01880) found long-context LLMs consistently outperformed RAG on QA tasks when compute was available, with chunk-based retrieval lagging the most. The cost trade-off is real: at 1M tokens, latency runs 30 to 60 times slower than a RAG pipeline, at roughly 1250 times the per-query cost; prompt caching can make long-context cost-competitive for high-traffic applications. The decision rule: if the corpus fits the window and query volume is moderate, long-context is the cleaner starting point; add retrieval only when the corpus exceeds the window, latency violates SLOs, or query volume crosses the break-even. Theorem 3: the property answer-from-full-context is guaranteed by loading the whole corpus, not by retrieving chunks of it. The routing decision (does this corpus fit?) precedes the architecture choice. Production ✅ for the form; Partial ⚠️ for the cost multipliers.

Memory compression: summarize before retrieving

When the corpus is too large for the window, the article's second alternative is to summarize before retrieving rather than pulling raw chunks. Summarization-based retrieval performs comparably to full long-context methods while chunk-based retrieval lags both. A concrete result: an order-preserving RAG approach using 48K well-chosen tokens outperformed full-context retrieval at 117K tokens by 13 F1 points, at one-seventh the token budget. Theorem 3: the property relevant-context-in-budget is guaranteed by compressing to relevance before injection, not by retrieving raw chunks and hoping the model filters. A well-compressed relevant document beats a raw dump of tangentially related chunks. Production ✅ for the form; Partial ⚠️ for the 13-F1-point figure (single study).

Structured retrieval: classify the query before retrieval runs

The article's third alternative maps most directly to the missing mechanism. When retrieval is the right architecture, the solution is routing by query type rather than applying better embeddings uniformly. Self-Route, presented at EMNLP 2024, lets the model classify whether a query needs full context or focused retrieval before running it. Simple factual lookups go to focused RAG. Complex multi-hop questions go to a long context. The result: better overall accuracy at lower computational cost. Adaptive systems using this hybrid approach have shown 15 to 30% retrieval-precision improvements through hybrid search and reranking. The article's load-bearing line: "The key change is making routing explicit. Every query gets classified before any retrieval runs, and the system stops treating all queries as identical embedding problems."

Theorem 3: the property right-strategy-per-query is guaranteed by the mechanism (classify the query, then retrieve), not by committing to one strategy at build time. The classification is the measurement; the routing is the mechanism. The system that classifies before retrieving implements the property; the system that retrieves-then-hopes does not. Production ✅ for the form; Partial ⚠️ for the specific percentages.

[ORIGINAL DATA]: The Honest Architect notes the structural parallel between Self-Route's classify-before-retrieve and Everythink's "the space is the router." Self-Route classifies the query, then routes to a retrieval strategy. Everythink routes the question to a room — network→community→room — before any model runs. Both implement the property relevance by routing before responding, not by broadcasting and filtering. The difference is domain and scope: Self-Route routes inside one retrieval system; "the space is the router" routes across an entire network of scoped contexts. Partial ⚠️ (same form — route-before-respond — separate domains).

Graph-based: route relational queries to a graph

The article's fourth alternative is for queries that require understanding relationships across a dataset rather than fetching a passage. These are the multi-hop questions: which decisions did the board reverse in Q3, and what was the stated reason each time? No single chunk answers this; the answer lives in the connections between documents. Microsoft Research introduced GraphRAG in 2024: build a knowledge graph from the corpus, traverse entity relationships rather than matching vectors. The trade-off is cost — knowledge-graph extraction runs 3 to 5 times more expensive than baseline RAG and requires domain-specific tuning — worth it for thematic analysis and multi-hop reasoning, not for single-passage lookups. Theorem 3: the property cross-document-relations is guaranteed by entities plus typed relationships, traversed, not by vector similarity. Production ✅ for the form; Partial ⚠️ for the cost multipliers. A deeper treatment of GraphRAG's mechanism forms sits in our earlier piece on matching the mechanism to the query type.

The space is the router — the mechanism RAG is missing

The four alternatives converge on one mechanism: route before you retrieve. Long-context routes by corpus size. Memory compression routes by budget. Structured retrieval routes by query type. Graph-based routes by relational structure. Each is a routing decision made before the retrieval mechanism runs, and each guarantees its property because the routing matches the query's actual shape.

Everythink's "the space is the router" is the same mechanism, one domain upstream. The network→community→room topology routes a question to the right scoped context before anything responds. A question asked in a room about payments retry logic is already routed to the payments community, the engineering network, the relevant document scope — before any Sister drafts, before the Oracle merges, before any retrieval runs. The routing is structural, not computed at query time. The property relevance is guaranteed by the topology, not by broadcasting the question across the whole network and filtering the responses. Production ✅ for the form (the HAI Engine has run this routing in production since 2016); Partial ⚠️ for the cross-domain claim.

The Honest Architect does not claim "the space is the router" is a retrieval system. Everythink is a forecasting platform, not a RAG vendor. The claim is structural: the missing mechanism in failed RAG pipelines is explicit routing before retrieval, and that mechanism has a production-proven analogue in Everythink's topology. The Sisters-to-Oracle pipeline is the downstream analogue of the article's map-reduce: each Sister drafts in parallel (the map stage), the Oracle merges them into a normalized Ensemble with entropy on every merge (the reduce stage), and the property calibrated-forecast is guaranteed by the mechanism, not by one model producing the whole forecast. Partial ⚠️ (same form — parallel-draft-plus-measured-merge — separate domains).

Frequently asked questions

Is RAG useless, as the headline says?

No, and the article does not argue that. It argues RAG is a reasonable default that breaks in predictable ways — retrieval irrelevance, context poisoning, the chunk-size conflict — and that the fix is routing by query type, not over-engineering the embeddings. The mechanism must match the property. Production ✅ for the diagnosis; the "useless" framing is editorial.

Why does adding embedding dimensions not fix retrieval irrelevance?

Because embedding dimensions implement similarity, not relevance. Retrieval irrelevance is a missing-routing failure: the query returns vocabulary matches that do not answer the question. Scaling the wrong mechanism compounds the cost without producing the property. Theorem 3: factual-relevance is guaranteed by routing before retrieval, not by finer-grained similarity. Production ✅.

What is the routing mechanism RAG is missing?

Explicit classification before retrieval. Self-Route classifies whether a query needs full context or focused retrieval before any retrieval runs. The property right-strategy-per-query is guaranteed by classify-then-retrieve, not by committing to one strategy at build time. Production ✅ for the form.

How does "the space is the router" relate?

It is the same form, one domain upstream. Everythink's network→community→room topology routes a question to the right scoped context before any model runs, the way Self-Route routes a query to the right retrieval strategy before retrieval runs. Both implement relevance by routing before responding. Partial ⚠️ (same form — route-before-respond — separate domains). Everythink is a forecasting platform, not a RAG vendor.

Does Everythink endorse GraphRAG or any retrieval tool?

No. Everythink is a forecasting platform. The article's figures are Partial ⚠️. The cross-domain parallels are illustrations, not endorsements. Scope is civil/defensive. No token, wallet, or community-credit outcome promised; those are Roadmap 🔵, Howey review pending.

Sources

• Nate Rosidi, KDnuggets, "Your RAG Pipeline Is Probably Useless. Here's a Better Alternative", published 2026-06-29, retrieved 2026-08-23, https://www.kdnuggets.com/your-rag-pipeline-is-probably-useless-heres-a-better-alternative

If your team is ready to route before it retrieves — to classify the query before any embedding runs, the way the space is the router classifies the question before any Sister drafts — read the 21 papers or book a demo. The HAI Engine has run that routing mechanism in production since 2016; the Sisters draft in parallel, the Oracle merges with entropy on every run, and Theorem 3 holds: the property is guaranteed exactly when its mechanism is implemented and measuring.