Better Embeddings Won’t Fix Missing Provenance in RAG

Part 1 of 6: Building a production retrieval layer, one failure at a time.Someone asked our internal assistant how much parental leave they were entitled to.It answered immediately. Clear number, confident tone, and a citation pointing at a real document that really did exist in our wiki — the parental leave policy, exactly the document you would want it to cite.The number was from 2021. The policy had been revised twice since.Nobody noticed for three weeks. Not the person who asked, not the three people who asked variations of the same question after them, and not me. There was nothing to notice. The answer was fluent, the citation resolved, the document was genuine. It was wrong in the one way a retrieval system can be wrong without leaving fingerprints.When I finally traced it, I found something I didn't expect. The embedding model had done its job perfectly.That was the problem.What this series isThis is the first of six posts that build one system. Not six disconnected tutorials — one architecture, developed the way real architectures actually get developed: something breaks, you add a component, something else breaks.Each post starts from the working system the last post left behind and ends by breaking it.The naive baseline — you are hereFixing ingestion — chunking, metadata, hierarchy, freshnessStorage and search — pgvector vs. a dedicated vector database, hybrid search, rerankingStructured data — text-to-SQL and routingRelationships — the graph layer and multi-hop questionsKnowing it works — evals, regression testing, cost and latency budgetsThis post builds the baseline honestly. Not a strawman - the version a competent engineer ships in week one, because it is the version I shipped in week one. Then I break it in public and show you the autopsy.If you have a RAG system in production right now, you probably have this bug. You just haven't found it yet.The corpusFive thousand documents of internal company documentation. If you have worked anywhere with more than fifty people, you have seen this pile:PDFs with tables in them. Benefits summaries, compensation bands, expense limits. The information that matters most lives inside table cells.Wiki pages, layered by history. The same parental leave policy exists in three versions spread across five years. All three are live. None is marked as superseded.A changelog nobody reads. Which is, it turns out, the only place the revision dates are recorded.Hold on to those three properties. Each one detonates in a later post. For now, just note that this is an ordinary corpus. Nothing about it is unusually hostile. It is the pile you actually have.The baselineHere is the whole architecture. Four stages, no framework, about eighty lines. ┌──────────┐ ┌────────┐ ┌───────────┐ ┌──────────┐ │ DOCUMENTS│───▶│ EXTRACT│───▶│ CHUNK │───▶│ EMBED │ └──────────┘ └────────┘ └───────────┘ └────┬─────┘ │ ▼ ┌──────────┐ ┌────────┐ ┌───────────┐ ┌──────────┐ │ ANSWER │◀───│ LLM │◀───│ RETRIEVE │◀───│ PGVECTOR │ └──────────┘ └────────┘ │ (top 5) │ └──────────┘ └───────────┘ I am writing this in plain Python rather than a framework. Not because frameworks are bad, but because this series is about the machinery a framework hides. If something else does your chunking, you cannot see that chunking is what broke you.Pull the text out. That is the entire ambition of this stage.from pypdf import PdfReader from pathlib import Path def extract(path: Path) -> str: if path.suffix == ".pdf": reader = PdfReader(path) return "\n".join(page.extract_text() or "" for page in reader.pages) return path.read_text(encoding="utf-8") This is the obvious choice and it is what almost everyone starts with. It produces a single flat string per document. Remember that word: flat.2. ChunkDocuments are too long for an embedding model's context and too long to stuff into a prompt, so they get split. The standard approach is fixed-size chunks with a little overlap so a sentence straddling a boundary appears in both neighbours.import tiktoken encoder = tiktoken.get_encoding("cl100k_base") def chunk(text: str, size: int = 512, overlap: int = 50) -> list[str]: tokens = encoder.encode(text) out, start = [], 0 while start list[list[float]]: response = client.embeddings.create( model="text-embedding-3-small", input=texts, ) return [item.embedding for item in response.data] def ingest(conn, documents: list[Path]) -> None: for path in documents: pieces = chunk(extract(path)) for i in range(0, len(pieces), 100): batch = pieces[i : i + 100] with conn.cursor() as cur: cur.executemany( "INSERT INTO chunks (content, embedding) VALUES (%s, %s)", list(zip(batch, embed(batch))), ) conn.commit() 4. Retrieve and answerEmbed the question with the same model, find the five nearest chunks by cosine distance, paste them into a prompt.def answer(conn, question: str) -> str: q_vec = embed([question])[0] with conn.cursor() as cur: cur.execute( "SELECT content FROM chunks ORDER BY embedding %s::vector LIMIT 5", (q_vec,), ) context = "\n\n---\n\n".join(row[0] for row in cur.fetchall()) completion = client.chat.completions.create( model="gpt-4o-mini", messages=[ { "role": "system", "content": ( "Answer using only the context provided. " "If the context does not contain the answer, say so." ), }, {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}, ], ) return completion.choices[0].message.content That is the whole system. It runs. It is not a toy, it is not deliberately crippled, and with a system prompt that explicitly forbids answering beyond the context, it is arguably more careful than a lot of what ships.The demo that fooled everyoneI want to be fair to this baseline before I take it apart, because an autopsy on a strawman proves nothing."What's the expense limit for client dinners?"- correct figure, correct policy document, correct caveat about pre-approval above a threshold."How do I request access to the analytics warehouse?" - correct process, correct approver, and it noticed the process differs for contractors."What's our on-call escalation policy?" - pulled the right runbook and summarised the tiers accurately.I demoed this. People were impressed, and they were right to be. Three weeks earlier the answer to all three questions had been "ask in Slack and hope."Then somebody asked about parental leave.The autopsyHere are the two chunks that mattered, with their cosine similarity to the question "How much parental leave do I get?"score 0.847 "...eligible employees may take up to 12 weeks of paid parental leave following the birth or adoption of a child. Leave must be taken within the first 12 months..." score 0.839 "...eligible employees may take up to 18 weeks of paid parental leave following the birth or adoption of a child. Leave may be taken in up to three separate blocks within the first 24 months..." The first chunk is the 2021 policy. The second is current. The gap between them is 0.008.The system retrieved both — they were both in the top five — and the language model, presented with two nearly identical passages and no signal distinguishing them, went with the one that appeared first. Which was the one that scored fractionally higher. Which was the obsolete one.Now, the two fixes you are already reaching for. I reached for both."Use a better embedding model." This does not solve the underlying problem, and it is worth being precise about why. An embedding model measures semantic similarity. Both of these chunks are semantically excellent answers to the question. That is not an error; it is the model doing exactly what it was asked to do. If the information that distinguishes the current policy from the obsolete one was discarded during ingestion, a better embedding model has no reliable signal to rank one above the other. You are asking a ruler to tell you the temperature."Add a reranker." A reranker re-scores the retrieved chunks against the question using a slower, more precise model. It may reshuffle the results, but it still cannot reliably resolve freshness or policy versioning if that evidence is missing from the chunks it sees. Both passages remain plausible answers. A better judge cannot compensate for evidence the pipeline threw away upstream.The date existed. It was on the wiki page — in the revision history, in the page metadata, in the changelog. Three separate places.We threw all three away in stage one, when extract() flattened the document into a string, and then threw away what remained in stage three, when we inserted content and embedding and nothing else.By the time the question arrived, the information needed to answer it correctly had been destroyed hours earlier by a function that was four lines long.What this actually meansThe chunk is the unit of truth in a RAG system. Not the document — the chunk. It is what gets scored, what gets retrieved, and what the language model treats as fact.And a chunk stripped of its origin is a claim with no provenance. It cannot tell you when it was written, what it superseded, who is bound by it, or whether it is still true. It is a sentence floating in space with a confidence score attached.Which reframes where the work is. Most effort labelled "improving RAG quality" happens at the retrieval end — better models, rerankers, query rewriting, prompt tuning. All of that operates on chunks that were finalised long before any query arrived. If the necessary information is not in the chunk, no amount of downstream cleverness will recover it. You are optimising the search over a corpus that has already lost the answer.The retrieval stage got a lot of attention because it is where the failure becomes visible. It is almost never where the failure happens.NextPost 2 rebuilds stage one and stage two. Same corpus, same embedding model, same five thousand documents — nothing downstream of ingestion changes at all. Chunking becomes structure-aware, chunks carry their source and their revision date, superseded versions get marked as superseded, and freshness becomes something the system can actually reason about.Then we run the same parental leave question and look at the numbers.Part 2: Your RAG Retrieval Problem Is a Data Pipeline Problem — coming soon.