Break Your Own RAG Pipeline Before Users Do

IntroductionRetrieval-augmented generation (RAG) answers questions using information retrieved from a set of documents. That document collection is called a corpus. A typical evaluation sends a well-written question to a tidy corpus and checks whether the retriever returns the correct passage.Production document collections rarely stay tidy. Old pages remain searchable after a policy changes. Users type "warehuse" instead of "warehouse." Optical character recognition (OCR), the software that extracts text from scans, may read "SSO" as "SS0." A table can also be divided at a page boundary, separating a number from its label.Each problem can send the retriever to the wrong passage. The language model then receives incorrect or incomplete context, so even a well-written answer may be wrong.In this article we will see a small retriever and fault injection, which means deliberately adding problems to test how a system responds. I added an outdated policy page, an OCR character swap, a query typo, and a divided table. ···Hey there, I'm Sara Nóbrega, an AI engineer with background in physics. If you're working on similar problems or want feedback on applying these ideas, I collect my writing, resources, and mentoring links here).👉 Read: 5 AI Skills That Will Keep Data Scientists Relevant in 2027 | Towards Data Science···The Toy PipelineA retriever searches documents and returns the passages that appear most relevant to a question. RAG systems usually divide each document into smaller passages called chunks before indexing them for search.This example keeps the retrieval method deliberately simple. It has 4 short documents and one scoring function. The function splits the query and each document into lowercase words, called tokens. It then gives a document 1 point for every query token found in that document. The document with the highest score wins.Each Document stores an ID, its text, a topic, and the date it was updated. Production systems often use more complex matching methods. Exact token matching makes each failure easy to see, and the same input problems can affect those systems.Stale documents that contradict each otherThe returns policy changed from 60 days to 30 days at the start of 2026. The ingestion process copies source documents into a searchable index. Here, it added the new page without removing the old one.Both pages receive the same token-overlap score because each contains "returns" and "days." The retriever needs a rule for equal scores, known as a tiebreaker. Here it uses list order, so it returns the first page: the policy from 2024. A customer could be told they have 60 days to return an item when the current limit is 30.Image by Author | Claude Design.The correction adds a recency tiebreaker. When documents receive the same score, the retriever prefers the one with the later updated date.This rule fits policies with a clear replacement date. Other collections may need an explicit status such as current or retired, especially when a newer document does not replace an older one.OCR errors that hide the matching wordsOCR converts scanned pages into searchable text. Similar-looking letters and numbers cause common extraction errors. The faulty OCR output changes "Enterprise" to "Enterpr1se" and "SSO" to "SS0."The extracted text contains enterpr1se and ss0, so the query tokens enterprise and sso each receive 0 points. All 4 documents tie at 0, and list order sends the returns policy back as the result.Image by Author | Claude Design.A normalization function corrects known OCR substitutions before tokenization. Applying it to both queries and documents gives the scorer consistent text.Production normalization rules need care. Converting every 0 to o, for example, could damage product codes or measurements. Build substitutions from errors found in your own extracted documents and limit them to fields where the change is safe.A typo in the queryUsers make spelling mistakes. "warehuse sync" is missing one letter from "warehouse sync," and an exact token matcher treats the 2 words as unrelated.The misspelled token contributes 0 points. This leaves sync to determine the result. Both the CRM document and the warehouse inventory document contain it, so the tie goes to the CRM document because it appears first.Image by Author | Claude Design.Fuzzy matching compares words by spelling similarity and gives close matches partial credit. With fuzzy matching enabled, warehuse is close enough to warehouse for the inventory document to score higher.The similarity threshold matters. Set it too low and unrelated words can match; set it too high and common typos still fail. Tests based on real queries provide better thresholds than a handful of invented spelling mistakes.A table divided across a page boundarySome PDF extraction tools create one chunk per page. If a table continues onto the next page, the first chunk may contain a row label while the second contains its value.Image by Author | Claude Design.Image by Author | Claude Design.The first chunk contains all 3 query tokens: basic, plan, and storage, so it ranks first. Its text ends after Basic |. The value, 10 GB, is in the next chunk and may never reach the language model.This correction belongs in the ingestion process. Detect table fragments and join related pages before creating searchable chunks.Joining every pair of pages would create oversized chunks and mix unrelated text. Limit the rule to detected tables or carry enough neighboring content forward to preserve each row.Test the evidence inside each resultThe returned document ID confirms which source ranked first. An evidence check confirms whether its chunk contains enough information to answer the question. The divided table demonstrates the difference: a chunk from the correct limits document could contain Basic and Storage while leaving 10 GB on the next page.Add an evidence check to each test. The check names the words or values that must appear in the retrieved text for the language model to produce a supported answer.The returns-policy test should require both the current document ID and the current value:The table test should require the row label and its value in the same retrieved chunk:These assertions also make failures easier to diagnose. A wrong ID points to ranking or filtering. A correct ID with missing text points to extraction or chunking. A correct ID with the required evidence gives the generation step enough source material to answer, although the final answer still needs its own evaluation.If your retriever returns several chunks, apply the same check to the combined text passed to the language model. The evaluated text will then match the context the model receives.···What the 4 tests catchRunning the 4 corrupted inputs against the basic retriever produces 4 failures. After the matching correction is enabled for each case, all 4 return the expected document.Image by Author | Claude Design.The fixes are small: use document dates to resolve a tie, normalize known OCR errors, allow close spelling matches, and preserve table rows during ingestion. Each one addresses a different cause. The separate test results identify which protection is missing.Run these tests alongside a standard relevance evaluation. Relevance tests measure whether retrieval works on expected inputs. Fault-injection tests measure whether it still works after a realistic defect is added to the query or document collection.The 4 cases are a starting test set. When production returns the wrong document, add a regression test: a repeatable check that confirms the bug stays fixed after later code changes. ···Thanks for reading! My name is Sara Nóbrega and I’m an AI engineer with background in physics.Useful links: