Three things cause retrieval to miss a fact sitting right there.
<p><b>Key facts.</b></p>
<ul>
<li>Retrieval, not generation, is where grounded agents most often break: RAG shifts the hallucination surface to the retrieval and context-assembly stage rather than removing it (Barnett et al., Seven Failure Points of RAG, arXiv:2401.05856).</li>
<li>Even purpose-built legal RAG tools hallucinated on 17% (Lexis+ AI) to 33% (Westlaw AI-Assisted Research) of 202 expert queries; base GPT-4 hallucinated on 43% (Stanford RegLab/HAI, Magesh et al., Journal of Empirical Legal Studies, 2025).</li>
<li>Retrieved facts placed in the middle of a long context are systematically under-used: accuracy follows a U-shaped curve and degrades sharply for mid-context evidence (Liu et al., Lost in the Middle, arXiv:2307.03172).</li>
</ul>
<h2>What is actually happening when the data exists but the answer is wrong?</h2><p><!-- conviction-start -->Pipeline leaks first; a frontier upgrade inherits the rework.<!-- conviction-end --> (<a href="https://arxiv.org/abs/2401.05856" target="_blank" rel="noopener">arXiv:2401.05856</a>)</p>
<p>The context that reached the model never had the right fact. Or it had the fact cut into a form the model could not use. RAG does not kill hallucination. It moves it earlier, from the model's memory to the retrieval and assembly step. If the right chunk is not fetched, or not ranked high enough, or buried where the model stops attending, the model fills the gap from training instead of refusing. The database was fine. The grounding layer between the database and the model leaked, and leaked quietly, because the wrong answer still reads fluent and confident.</p>
<h2>Why does retrieval miss a fact that is sitting right there?</h2>
<p>Three things cause retrieval to miss a fact sitting right there. <b>Retrieval miss:</b> the chunk is indexed, but embedding similarity scores it below the top-k cutoff, so the model never sees it. This is common with exact identifiers like codes or case numbers. <b>Embedding mismatch:</b> the embedding model puts the query and the right chunk far apart in vector space. Cosine similarity picks a different chunk, one that is thematically nearby but factually off. <b>Ranking failure:</b> similar-looking chunks score higher and the right one sits just below the cutoff, or gets cut at the token limit. In all three cases the right data is in the index. The model just never sees it. A bigger model does not fix this. The bottleneck is what reaches the context, not how well the model reasons once it arrives.</p>
<h2>Why does chunking make the agent invent a value?</h2>
<p>Fixed-size chunking splits documents at arbitrary token boundaries. A rule and its exception end up in different chunks. A table row gets separated from its header. A record loses one of its fields. Retrieval returns a partial piece and the model gap-fills the missing field with a plausible invented value instead of saying it could not find it. The fact was in your source system. Chunking cut it in half before the model ever saw it. Semantic chunking, which splits at natural topic and section boundaries and keeps tables intact, is one of the highest-impact fixes available.</p>
<div class="fig"><img src="/blog/article1-diagram.png" alt="The leaky retrieval pipeline: query, retrieval with a dropped chunk, ranking, a context window with the right chunk buried in the middle, and a wrong answer"/></div>
<h2>Why does the agent ignore a fact it did retrieve?</h2>
<p>Even when the right chunk reaches the context window, the model can under-weight it. Liu et al. found a U-shaped accuracy curve: key facts at the start or end of the context get used, facts buried in the middle get skipped. This is the lost-in-the-middle effect, and it holds even for long-context models. With a large top-k and verbose chunks, the correct evidence is right there in the context and the model still guesses from training. Retrieval succeeded and the answer is still wrong. Teams who only check whether a document was retrieved miss the failure entirely.</p>
<h2>A real case: the data existed and the agent still got it wrong</h2>
<p>In 2024 an Air Canada customer asked the airline's chatbot about bereavement fares. The bot confidently described a policy allowing a retroactive discount claim. The correct policy existed on Air Canada's own site and did not work that way. The customer relied on the wrong answer, was denied, and took it to the British Columbia Civil Resolution Tribunal, which ruled against Air Canada and awarded damages of roughly 812 Canadian dollars, rejecting the argument that the chatbot was a separate entity responsible for itself. The ground truth was in the source system. The agent synthesized a confident, wrong version anyway. That is this failure mode in public, with a legal ruling attached.</p>
<h2>How do you stop it?</h2>
<table style="width:100%;border-collapse:collapse;margin:1.5rem 0;font-size:0.97rem;"><tr><th style="text-align:left;padding:10px 12px;border-bottom:2px solid #e5e7eb;color:#5E6AD2;font-weight:600;">Mechanism</th><th style="text-align:left;padding:10px 12px;border-bottom:2px solid #e5e7eb;color:#5E6AD2;font-weight:600;">Signal</th><th style="text-align:left;padding:10px 12px;border-bottom:2px solid #e5e7eb;color:#5E6AD2;font-weight:600;">Fix</th></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Retrieval miss / ranking</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Right doc exists, never retrieved</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Hybrid search (BM25 + dense, fused) plus a cross-encoder reranker</td></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Embedding mismatch</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Thematically close, factually wrong chunk wins</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Domain-tuned embeddings; exact-match for identifiers</td></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Bad chunking</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Partial facts, invented fields</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Semantic / structure-aware chunking</td></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Lost in the middle</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Retrieved but ignored</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Rerank to the top, compress context, smaller top-k</td></tr>
<tr><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Missing-field gap-fill</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Confident invented value</td><td style="padding:10px 12px;border-bottom:1px solid #eee;vertical-align:top;">Grounding / faithfulness checks; allow "not found"</td></tr>
</table>
<p>Hybrid search plus reranking gets the right chunk in front of the model. Semantic chunking keeps facts whole. Metadata filtering removes stale and contradictory versions. Corrective or agentic RAG lets the agent check whether its answer is supported and re-retrieve or escalate if not. A grounding check that permits an honest "I could not find this" stops the silent gap-fill. None of these is a bigger model. Every one of them is an engineering layer above the model. That layer, knowing where your retrieval is trustworthy and where it is guessing, is what VibeModel builds as the Pattern Intelligence Layer.</p>
<aside class="jaside v-warn"><span class="col"><svg class="hook" width="28.5" height="34.5" viewBox="0 0 57 69" fill="none" preserveAspectRatio="none" xmlns="http://www.w3.org/2000/svg"><path fill="var(--pageBg)" d="M54 0V0.716804C54 25.9434 35.0653 47.1517 10 50L0 57V0H54Z"/><path fill="var(--acc)" d="M56.9961 4.15364C57.0809 2.49896 55.8083 1.08879 54.1536 1.00394C52.499 0.919082 51.0888 2.19168 51.0039 3.84636L56.9961 4.15364ZM9.09704 51.7557L8.49716 48.8163L9.09704 51.7557ZM6 69V59.2227H0V69H6ZM9.69692 54.6951L14.3373 53.7481L13.1375 47.8693L8.49716 48.8163L9.69692 54.6951ZM14.3373 53.7481C38.202 48.8777 55.7486 28.4783 56.9961 4.15364L51.0039 3.84636C49.8967 25.4384 34.3213 43.5461 13.1375 47.8693L14.3373 53.7481ZM6 59.2227C6 57.0268 7.54537 55.1342 9.69692 54.6951L8.49716 48.8163C3.55195 49.8255 0 54.1756 0 59.2227H6Z"/></svg><span class="rail"></span></span><span class="glyph"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3Z"/><path d="M12 9v4"/><path d="M12 17h.01"/></svg></span><h4>Watch out</h4><p>The grounding layer between the database and the model leaked, and leaked quietly, because the wrong answer still reads fluent and confident.</p></aside>
Frequently asked questions
Is a bigger or smarter model the fix for this?
No. The failure is in the grounding layer: what gets retrieved, ranked, and positioned before the model writes. A stronger model reasons better over its context, but if the right fact never reached that context, or was buried in the middle of it, the answer is still wrong, just more fluently.
How do I tell a retrieval failure from a model failure?
Trace whether the correct chunk was retrieved and where it sat in the context. If it was never retrieved, that is a retrieval or ranking problem. If it was retrieved but ignored, that is lost-in-the-middle or a weak grounding check. Measuring only the final answer hides which one it is.
Does RAG eliminate hallucination?
No. RAG moves the hallucination surface from the model's memory to the retrieval and assembly stage. Done well it sharply reduces hallucination; done with fixed chunking, weak ranking, and no grounding check, it produces confident wrong answers over data that exists.
What is the single highest-impact fix?
Add a reranker and a grounding check. Reranking gets the right chunk to the top of the context; the grounding check verifies every claim is supported by a retrieved source and lets the agent say "not found" instead of inventing a value.