Scaling RAG to 10M Documents: The Hallucination Fixes That Worked Were the Boring Ones

The biggest thing scaling a retrieval pipeline past 10M documents taught me: hallucination is a retrieval problem wearing a generation costume. I spent the first weeks tuning prompts and swapping in bigger models, and the fabrication rate barely moved. What actually moved it was unglamorous, killing duplicate and contradictory chunks before they reached the model, adding a confidence gate that let the system answer "I don't know," and verifying every claim against the retrieved text after generation. None of that is a model upgrade. RAG narrows hallucination but never erases it S1, and at scale the leverage lives in retrieval precision and the licence to abstain, not in the LLM. Here's the order I learned that in.

Why didn't a bigger model fix the hallucinations?

That was my first move, because it's the easy one. Swap the generator, rewrite the system prompt to "only use the retrieved passages," and wait for the numbers to improve.

They barely did.

The research lines up with what I saw on my own corpus: RAG reduces hallucination, but it does not eliminate it S1S4. Worse, a model will often hallucinate rather than abstain, and it does this more under RAG than in a closed-book setting when the retrieved context is insufficient S5. You hand it weak evidence, it fills the gap confidently.

The lesson landed hard. The smartest model you can afford is still capped by the evidence you put in front of it S2. If the retrieval layer feeds it noise, no amount of prompt scolding saves the answer. So I stopped tuning the generator and went upstream.

What was actually poisoning the answers?

Duplicates and contradictions.

Most knowledge bases aren't truth-shaped, they're document-shaped. You end up with the 2022, 2023, and 2024 version of the same policy sitting in the index, all equally retrievable S3.

So the retriever does its job and pulls back two chunks: "Refunds take 5 days" and "Refunds take 14 days." The model has no mechanism to know which one is current S3. It picks one, or splits the difference, and now you've shipped a confident wrong answer that looks grounded because it technically came from a real document.

This was boring fix number one, and it moved the needle more than anything else: drop near-duplicates at ingestion, before they ever touch retrieval.

# At ingestion: drop near-duplicate chunks before they reach the index
seen = MinHashLSH(threshold=0.85)
for chunk in chunks:
    sig = minhash(chunk.text)
    if seen.query(sig):          # a near-identical chunk already indexed
        continue
    seen.insert(chunk.id, sig)
    index.add(chunk, metadata={"version": chunk.version, "source": chunk.source})

Pair that with metadata filtering so version and recency are first-class at query time, and a whole category of "contradiction hallucinations" disappears, not because the model got smarter, but because it stopped being handed a contradiction.

Why did raising top-k make things worse, not better?

My next instinct was the obvious one: if the right answer might not be retrieved, just retrieve more. Push k from 5 to 20 so the correct chunk is somewhere in the prompt.

That backfired.

Models prioritize information at the very start and end of a long context and quietly skip the middle, the "lost-in-the-middle" effect S3S10. If the critical chunk lands fifth out of twenty, the model may conclude the information isn't there at all, then fall back to its training data. A hallucination born from good intentions S3.

The fix isn't fewer candidates or more candidates, it's a two-stage retrieve. Cast a wide net with cheap recall, then use an expensive, precise reranker to reorder and keep only the few that matter S7S8.

The bi-encoder that fetched your top 30 might rank chunk #17 as most relevant. A cross-encoder reads the actual query against each chunk's actual text and realizes chunk #3 is the real match. That reordering is the difference between a good RAG and a great one, and it only ever runs on the top 15–30 candidates, never the full corpus S7S8.

How do you make the system admit it doesn't know?

This was boring fix number two, and it bought the most user trust per line of code.

Every retrieved chunk carries a relevance score by the time it reaches the prompt. So gate on it. If nothing clears the bar, don't generate, return an honest "I couldn't find this in the knowledge base."

def answer(query, index, threshold=0.65):
    candidates = hybrid_retrieve(query, index, k=30)     # BM25 + dense
    ranked = cross_encoder_rerank(query, candidates)     # precise reorder
    top = ranked[:5]
    if max(c.score for c in top) < threshold:
        return Abstain("Insufficient evidence in the knowledge base.")
    return generate_grounded(query, top)

This felt wrong at first, I was deliberately making the system answer fewer questions. But a confident wrong answer is infinitely worse than an honest abstention S5S6S13. The literature converges on the same move under different names: selective RAG driven by a sufficient-context signal S5, a calibrated "reject option" S6, and uncertainty thresholding that decides whether to answer, hedge, or abstain S12.

An abstention isn't the system failing. It's the system working correctly.

How do you catch a hallucination after the model has already written it?

The gate stops the model from answering with no evidence. It doesn't stop the model from drifting away from evidence it did receive. So I added boring fix number three: a post-hoc grounding check.

Split the generated answer into individual claims. For each one, verify it's actually supported by the retrieved chunks, entailment via an off-the-shelf NLI model, or fuzzy matching, since the model paraphrases rather than quotes S6S11.

def verify(answer, context_chunks):
    claims = split_into_claims(answer)
    flagged = [c for c in claims if not entailed_by(c, context_chunks)]
    faithfulness = 1 - len(flagged) / max(len(claims), 1)
    return faithfulness, flagged     # surface flagged claims, don't silently drop them

I run this asynchronously: stream the answer to the user immediately, run verification in parallel, and surface a warning if a claim couldn't be traced to a source. Crucially, I don't auto-suppress the response, I show the user which specific claim failed verification S6S11. That's far more useful than a blanket "this might be wrong," and it turns the verifier into a debugging tool, not just a guardrail.

What about retrieval quality itself, embeddings, hybrid, chunking?

This is the part everyone wants to optimize, and it matters, but it's table stakes, not the differentiator.

Hybrid retrieval beats either method alone. Dense embeddings capture meaning but miss exact identifiers, error codes, and rare terms; BM25 catches the literal string but misses paraphrase. Most production pipelines combine both S9.

Chunking is a genuine tradeoff with no free answer: chunks too small lose their surrounding context, chunks too large smuggle in irrelevant noise or blow the context budget S9S10. Structure-aware splitting plus a one-line context prefix on each chunk was the best balance I found.

But here's the thing I'd tell my earlier self: I could have perfected chunking and embeddings for a month and still shipped hallucinations, because the failures weren't coming from there. They were coming from contradictions, missing abstention, and unverified claims. Get retrieval good enough, then spend your real effort on the three boring fixes above.

How did you know any of this was working?

You can't improve what you don't measure, and you can't measure it once and walk away.

I track two families of metric continuously in production, not just in an offline test set S8: faithfulness (is the answer grounded in the retrieved context?) and context relevance (did retrieval even fetch the right material?). Tools like RAGAS and DeepEval score these directly, and a well-tuned pipeline can hit faithfulness in the mid-90s S8.

The split is what makes it actionable. Low context relevance is a retrieval failure, fix chunking, embeddings, or reranking. Low faithfulness with good context is a generation failure, tighten the prompt or the grounding check S8S12. Same symptom on the surface, completely different repair underneath. Before I separated them, I was guessing.

Where does hallucination actually come from, and what fixes it?

Should you even build this?

Not every problem needs a 10M-document pipeline. A few honest disqualifiers I wish I'd checked first:

• Skip RAG if your data fits a long-context window. If the whole corpus is 200k–1M tokens, a long-context model reading it directly is simpler and often more accurate on a small, static set S3. RAG earns its complexity at scale, not below it.
• Don't use vector search for structured queries. "Which customers spent more than $5,000 in December?" is a SQL question. Vector search can't do aggregations or joins S3.
• Build this when you have millions of documents, frequent updates, contradictory or versioned sources, a need for citations and provenance, and a domain where a confident wrong answer carries real cost S1.

If you land in that last bucket, the order is: dedup first, gate second, verify third. Retrieval polish underneath all of it.

Sources

1. S1A Systematic Review of Key Retrieval-Augmented Generation Systems: Progress, Gaps, and Future Directions
2. S2Enhancing LLMs with Domain-specific RAG: Long-form Consumer Health QA in Ophthalmology
3. S3How to Stop AI Hallucinations in Enterprise RAG Systems (A Complete Guide), Redgate Simple Talk
4. S4Reducing Hallucination in Structured Outputs via Retrieval-Augmented Generation
5. S5Sufficient Context: A New Lens on Retrieval Augmented Generation Systems
6. S6HALT-RAG: A Task-Adaptable Framework for Hallucination Detection with Calibrated NLI Ensembles and Abstention
7. S7MEGA-RAG: Multi-Evidence Guided Answer Refinement for Mitigating Hallucinations
8. S8LLM-Assisted Question-Answering on Technical Documents Using Structured Data-Aware RAG
9. S9RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines
10. S10A Survey of Scaling in Large Language Model Reasoning
11. S11HalluSearch at SemEval-2025 Task 3: A Search-Enhanced RAG Pipeline for Hallucination Detection
12. S12Hallucination Mitigation: RAG, Decoding, and Training, Michael Brenndoerfer
13. S13Enhancing RAG with Active Learning: Reject Incapables and Answer Capables

Written by

Syed Moinuddin

Full Stack Engineer writing about AI tooling, agentic systems, and shipping things that survive production. Follow along for more deep dives on the tools changing how we ship software.