Why Better RAG Starts With Better Ingestion

Part 2 of 6: Building a production retrieval layer, one failure at a time. TL;DR: Same 5,000 documents. Same embedding model. Same top-5 retrieval, same prompt, same everything downstream. I changed only the ingestion pipeline — structure-aware chunking, provenance on every chunk, and explicit supersession — and the parental leave bug from Part 1 became impossible rather than unlikely. Here is the rebuilt pipeline, the schema, and an honest account of what this approach still cannot do. Where we are The system currently does this: extract text from a document, cut it into 512-token pieces, embed each piece, store the piece and its vector in Postgres, retrieve the five nearest to a question, paste them into a prompt. It works well on most questions and fails invisibly on one particular class of question. In Part 1 it told an employee they had 12 weeks of parental leave, citing a genuine document. The number was four years out of date. Two chunks — the 2021 policy and the current one — scored 0.847 and 0.839 against the query. Near-identical, because they are near-identical. The obsolete one won by 0.008. The autopsy found that no better embedding model and no reranker could have prevented it, because the information that distinguishes the two chunks — the revision date — had been thrown away hours earlier by a four-line extraction function. This post rebuilds those four lines. Nothing downstream changes. The claim Retrieval quality is capped by ingestion. That's the whole argument, and I want to state it precisely because the loose version of it is wrong. I am not claiming embedding models don't matter, or that rerankers are useless. They matter. But they operate on chunks that were finalised before any query existed. If the information needed to answer a question is not present in the chunk, retrieval cannot recover it — it can only rank what it was given. Most work labelled "improving RAG" happens downstream of the damage. This post moves upstream. Here's the architecture with the ingestion stage rebuilt. Everything to the right of the dashed line is untouched from Part 1. ┌──────────┐ ┌──────────────┐ ┌──────────────┐ │ DOCUMENTS│──▶│ PARSE │──▶│ STRUCTURE- │ │ │ │ + METADATA │ │ AWARE CHUNK │ └──────────┘ └──────────────┘ └──────┬───────┘ │ ┌──────────────┐ ┌──────▼───────┐ │ SUPERSESSION │◀──│ CONTEXTUAL │ │ MARKING │ │ HEADER │ └──────┬───────┘ └──────────────┘ │ ─────────────────────┼────────────── unchanged ────── ▼ ┌──────────┐ ┌──────────┐ ┌────────┐ ┌────────┐ │ EMBED │─▶│ PGVECTOR │─▶│RETRIEVE│─▶│ LLM │ └──────────┘ └──────────┘ │+ FILTER│ └────────┘ └────────┘ Fix 1: Stop throwing away the document The schema in Part 1 had three columns: id, content, embedding. Here is the replacement. CREATE TABLE chunks ( id BIGSERIAL PRIMARY KEY, content TEXT NOT NULL, embedding vector(1536), -- provenance source_path TEXT NOT NULL, source_title TEXT, heading_path TEXT[], chunk_index INT NOT NULL, -- time effective_date DATE, ingested_at TIMESTAMPTZ NOT NULL DEFAULT now(), -- lifecycle status TEXT NOT NULL DEFAULT 'current', superseded_by BIGINT REFERENCES chunks(id), -- change detection content_hash TEXT NOT NULL ); CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops); CREATE INDEX ON chunks (status) WHERE status = 'current'; CREATE INDEX ON chunks (content_hash); Every column here exists because of a specific failure, not because it seemed tidy. effective_date and status are what the parental leave bug needed. heading_path is what the flattened tables need. content_hash is what makes re-ingestion cheap enough to actually run. chunk_index lets you fetch a chunk's neighbours when a retrieved passage is obviously truncated mid-thought. The extraction function now returns a document, not a string: from dataclasses import dataclass, field from datetime import date from pathlib import Path @dataclass class Document: path: Path title: str text: str effective_date: date | None = None metadata: dict = field(default_factory=dict) Where the date comes from is the unglamorous part, and it's different for every corpus. In ours: wiki pages carry a revision timestamp in their frontmatter, PDFs sometimes have an effective date in the header and sometimes only in the filename, and a handful have nothing at all. I wrote a small cascade — frontmatter, then a regex over the first 500 characters, then file mtime as a last resort — and, critically, recorded which source the date came from, because a date inferred from file mtime deserves less trust than one printed on the document. This stage is boring, specific to your data, and worth more than any model upgrade you will make this year. Fix 2: Chunk along the document's own seams Fixed-size chunking cuts every 512 tokens regardless of what is there. It will cut through the middle of a table, split a policy from its own heading, and merge the end of one section with the start of an unrelated one. Structure-aware chunking splits on the document's existing boundaries first, then subdivides anything still too large: import re HEADING = re.compile(r"^(#{1,6})\s+(.*)$", re.MULTILINE) def split_by_heading(text: str) -> list[tuple[list[str], str]]: """Return (heading_path, section_text) pairs.""" sections, stack, cursor = [], [], 0 matches = list(HEADING.finditer(text)) for i, m in enumerate(matches): level, title = len(m.group(1)), m.group(2).strip() end = matches[i + 1].start() if i + 1 str: header_parts = [f"Document: {doc.title}"] if heading_path: header_parts.append("Section: " + " > ".join(heading_path)) if doc.effective_date: header_parts.append(f"Effective: {doc.effective_date.isoformat()}") return "\n".join(header_parts) + "\n\n" + body The chunk that gets embedded now begins: Document: Parental Leave Policy Section: Entitlement > Duration Effective: 2021-03-01 Eligible employees may take up to 12 weeks of paid parental leave... Two things happen. The embedding now encodes what this passage is, not just what it says — so a query mentioning a policy name or a year has something to match against. And the language model receiving the chunk can see the date with its own eyes, which means it can hedge or flag staleness rather than answering flatly. Store the header-prefixed version in content and embed that same string. Keeping the embedded text and the stored text identical will save you a genuinely miserable debugging session later. Fix 4: Make supersession explicit Now the actual bug. Two versions of one policy both live in the corpus, both true-sounding, one obsolete. The instinct is to filter by date at query time — take the most recent. Don't. Plenty of legitimate questions are historical ("what was the limit before the 2023 change?"), and silently discarding old documents replaces a wrong-answer problem with a no-answer problem. Instead, mark the relationship at ingestion: def mark_supersession(conn) -> None: """Within a document lineage, the newest effective_date wins.""" with conn.cursor() as cur: cur.execute(""" WITH ranked AS ( SELECT id, lineage_key, heading_path, effective_date, ROW_NUMBER() OVER ( PARTITION BY lineage_key, heading_path ORDER BY effective_date DESC NULLS LAST ) AS rank, FIRST_VALUE(id) OVER ( PARTITION BY lineage_key, heading_path ORDER BY effective_date DESC NULLS LAST ) AS newest_id FROM chunks ) UPDATE chunks c SET status = 'superseded', superseded_by = r.newest_id FROM ranked r WHERE c.id = r.id AND r.rank > 1 """) conn.commit() lineage_key is how you decide two documents are versions of the same thing. Ours is a normalised title with version markers stripped — "Parental Leave Policy (2021 rev)" and "Parental-Leave-Policy-v3" both reduce to parental-leave-policy. This is a heuristic and it is imperfect. It is also enormously better than nothing, and every incorrect grouping it produces is at least visible in a table you can query, which the old system could not offer at all. Retrieval gains one clause: cur.execute(""" SELECT content, source_title, effective_date FROM chunks WHERE status = 'current' ORDER BY embedding %s::vector LIMIT 5 """, (q_vec,)) Historical questions get routed to an unfiltered variant. That routing decision is a Part 4 problem; for now, the default is current-only. Fix 5: Make re-ingestion cheap None of the above is worth much if the pipeline only runs once. A corpus that was correct at ingestion time drifts, and freshness has to be a property of the running system rather than of the day you built it. The content_hash column makes this trivial. On each run, hash every chunk; skip anything whose hash already exists; embed only what changed. On our corpus, a nightly run touches a low single-digit percentage of chunks, which turns re-embedding from a budget conversation into a cron job. The rerun Same question. Same model. Same top-5. score 0.891 [status: current] "Document: Parental Leave Policy Section: Entitlement > Duration Effective: 2025-01-15 Eligible employees may take up to 18 weeks of paid parental leave..." The 2021 chunk is still in the database. It is retrievable, it is auditable, and a historical query can still reach it. It simply cannot be returned as the current answer any more, because it is explicitly marked as not being the current answer. Note where the improvement actually came from. The score went up — 0.847 to 0.891 — because the contextual header gave the embedding more to match against. But the score is not what fixed the bug. The bug was fixed by a WHERE clause, which is to say by a fact about the world that we bothered to record. If you take one thing from this post: the fix was structural, not statistical. What this still doesn't solve Four honest limitations, because a post that claims a clean win is lying to you. Dates you don't have. Roughly one in nine of our documents had no recoverable effective date. They default to current, which is a guess. The pipeline makes that guess visible instead of invisible — you can query for it — but it does not make it correct. Lineage grouping is a heuristic. Two genuinely different policies with similar titles will get incorrectly linked. We catch these by reviewing the supersession table when it changes, which does not scale forever. Table extraction is still the weakest link. Structure-aware chunking keeps tables intact once they're found. Finding them reliably inside arbitrary PDFs is a harder problem than anything else in this post, and our solution is best described as "mostly." The cost moved, it didn't vanish. Ingestion is now slower and more complex, and it has failure modes of its own. You have traded an invisible retrieval problem for a visible pipeline problem. That is a good trade, but it is a trade. Next Retrieval is now correct. It is not yet fast, and it is not yet complete. Two things break in Part 3. At 5,000 documents, pgvector with an HNSW index is comfortable — but the index has recall characteristics nobody warns you about, and they get worse as the corpus grows. And there's a class of query that vector search handles badly no matter how good your ingestion is: someone pastes an exact error code, and semantic similarity confidently returns five passages that are about errors and none that contain that string. Part 3 covers the storage and search layer: what pgvector's index is actually doing, when a dedicated vector database earns its operational cost, and why hybrid search fixes the error-code problem that neither better chunking nor a better model will touch. Part 3: pgvector Was Fine Until It Wasn't — coming soon. Part 1: https://hackernoon.com/better-embeddings-wont-fix-missing-provenance-in-rag