My RAG System Answered a Stock-Price Question It Had No Data For

My retrieval system was working. That was the problem. I built this prototype to test one thing. Could a local, auditable RAG system support financial-document research without turning retrieval scores into false confidence? The pipeline ran over SEC filings and earnings-call transcripts. The corpus covered three consumer staples companies: Coca-Cola, PepsiCo, and Mondelez. Twelve PDFs in total. Hybrid retrieval, a cross-encoder reranker, grounded generation with mandatory citations. There was also a refusal gate, meant to decline when the documents could not support an answer. It ran in Docker. It answered questions about margin pressure, citing the exact transcript. Then I ran my evaluation set through it. One line in the query log stopped me: {"question": "What is Coca-Cola's stock price today?", "refused": false, "top_retrieval_score": 0.9990252256393433, "top_retrieved_sources": [ {"company": "KO", "doc_type": "10-Q", "period": "2025Q3", "section_or_speaker": "Item 1"}, {"company": "KO", "doc_type": "10-K", "period": "FY2025", "section_or_speaker": "Item 5"}]} My corpus contains no market data. No price feeds, no quotes, nothing time-sensitive. A static annual filing cannot tell you what a stock traded at today. The system answered anyway. Its reranker relevance score was 0.999, the highest in the entire run. Two lines below it, in the same log, this: {"question": "What did Coca-Cola's management say about pricing in Q1 2026?", "refused": true, "refusal_reason": "The model determined the retrieved passages don't answer the question.", "top_retrieval_score": 0.2726505696773529} That question is answerable. The Q1 2026 earnings call discusses pricing at length. Retrieval pulled the right transcript. The system refused it. Here is the full picture from that run: Expected Class Answered Refused Total Answerable 6 6 12 Unanswerable 1 2 3 Total 7 8 15 Six of twelve answerable questions were refused. One of three unanswerable questions was answered. Overall decision accuracy was 8 of 15. That number needs context, since the set is unbalanced. Twelve of fifteen questions were answerable. So a system that answered everything would score 12 of 15 on overall accuracy, while failing the entire point. The honest read is per class. The system accepted half the answerable questions and refused two-thirds of the unanswerable ones. This article is about why that happened and what I did about it. The cause was structural, not a threshold set two decimal places wrong. Test Setup Everything below comes from recorded runs. Here is the exact configuration, held constant across all of them unless stated. Component Value Generation model llama3.2:1b via Ollama 0.32.1 Decoding temperature 0.0, top_p 0.9, top_k 40, seed 0 Embedding model BAAI/bge-base-en-v1.5 Reranker BAAI/bge-reranker-base (cross-encoder) Vector store Chroma, persistent on disk Keyword index BM25 Retrieval LangChain EnsembleRetriever, equal dense and sparse weights Refusal threshold 0.1 on the top reranker score Evaluation RAGAS 0.3.9: Faithfulness, AnswerRelevancy, LLMContextPrecisionWithoutReference RAGAS judge llama3.2:1b Hardware Laptop, CPU inference The corpus was twelve PDFs, four per company: Company Q3 2025 FY2025 Q1 2026 Coca-Cola (KO) 10-Q 10-K 10-Q + transcript PepsiCo (PEP) 10-Q 10-K 10-Q + transcript Mondelez (MDLZ) 10-Q 10-K 10-Q + transcript Two facts matter later. RAGAS answer-level metrics were computed only over non-refused responses. Moreover, the decoding temperature was set to 0, deliberately to make generation as reproducible as possible. Use Case: Why an Asset Manager Needs Refusal More Than Fluency The target user was a research analyst at a mid-size asset manager covering several companies. They need answers fast. The source material is fragmented across formats. There are 10-K and 10-Q filings, earnings-call transcripts, and internal investment memos. In this workflow, an analyst may need to search across all three before they can support a single comparison. Answers vary between people asking the same question. Sourcing is hard to audit afterward. That last point drives the design. In most RAG applications, a hallucination is embarrassing. In an asset manager, it is worse. A fabricated statement attributed to a CFO can enter an investment memo, a research note, or a compliance review. The gap between a wrong answer and no answer is severe here. Severe enough that refusal ceases to be a defensive feature. It becomes the core product requirement. The goal was never to answer everything. It was to answer only what the documents support, cite it, and decline honestly otherwise. That is what my system failed at. Architecture Architecture of the financial-document RAG pipeline. Pre-generation deterministic checks (evidence type, scope coverage, score threshold) run before the model. Post-generation checks include one model-generated self-refusal and one deterministic citation-resolution check. The request flows top to bottom. PDFs are ingested with metadata, indexed into a vector store and a keyword index, and retrieved through a hybrid ensemble with reranking. The retrieved evidence passes through the pre-generation checks, generation, and post-generation checks. The system then returns either an answer with citations or a refusal with a reason. The checks are grouped by when they run and how they decide. The three at the top are deterministic and run before the model is called. The two at the bottom run after generation, and only one of them delegates its decision to the model. That grouping is the spine of this article, and the two green checks marked "added" did not exist in my first version. How I arrived at them is the rest of the piece. Everything runs on a laptop. That constraint matters when you read the numbers. Question One: A Working Query, Layer by Layer Start with a question that worked. The failures only make sense once you see success. "What did PepsiCo say was driving margin pressure in Q1 2026?" This returned in a few seconds. Nothing was refused. It cited three sources: two chunks from the PepsiCo Q1 2026 transcript, and one from Item 7 of the FY2025 10-K. Ingestion: Financial PDFs Have Structure, and Generic Chunkers Destroy It A 10-K is not an unstructured document. It has a mandated skeleton. Item 1 covers Business. Item 1A covers Risk Factors. Item 7 covers Management's Discussion and Analysis. A fixed-size chunker with a 1000-character window ignores that. It slices through those boundaries. The result is a chunk that begins with a risk disclosure and ends with a liquidity discussion. Later, when someone asks about risk factors, retrieval returns a chunk that is half about something else. So, chunking follows the document's own section boundaries. Every chunk carries metadata assigned at ingestion: @dataclass class ChunkMetadata: company: str # KO, PEP, MDLZ doc_type: str # 10-K, 10-Q, transcript period: str # FY2025, 2026Q1, 2025Q3 section_or_speaker: str | None # "Item 7", "Dara Mohsenian" source_path: str chunk_id: str That section_or_speaker field carries two things, depending on document type, and both matter in finance. In a filing, the item number is the item. In a transcript, the speaker is the one speaking. That distinction is material. It matters whether the CFO stated something, the CEO stated it, or an analyst merely asked. A chunker that drops speaker identity turns a citable commitment into an anonymous fragment. The company and period fields are, in fact, the most important part of this article. They are what the deterministic checks use later. Retrieval: Why Dense Search Alone Struggles on Peer Companies Three large consumer-staples companies discussing commodity and input costs can produce remarkably similar language. Dense search alone can struggle to distinguish peers when disclosures read alike. It will sometimes return PepsiCo text for a Coca-Cola question, because the semantics are close. The answer is hybrid retrieval, dense plus sparse, followed by reranking: ensemble = EnsembleRetriever( retrievers=[vector_retriever, bm25_retriever], weights=[0.5, 0.5], ) candidates = ensemble.invoke(question) # cast a wide net, cheaply reranked = cross_encoder.rank(question, candidates) # then rank precisely Two stages, two purposes. Retrieve wide and cheap. Then, rerank narrow and sharp. BM25 catches exact terms, but dense search glosses over them. The cross-encoder reorders the merged pile by relevance. For the PepsiCo margin question, this worked. Top reranker score of 0.4297. Three PepsiCo sources. A grounded answer. The pipeline behaved. Question Two: Where Coverage Quietly Breaks Now a harder question, the kind a sector analyst actually asks. "How did KO, PEP, and MDLZ each describe input-cost or commodity pressure in Q1 2026?" The system answered it. No refusal. Here is what it retrieved: "top_retrieved_sources": [ {"company": "MDLZ", "doc_type": "transcript", "period": "2026Q1"}, {"company": "MDLZ", "doc_type": "10-Q", "period": "2025Q3"}, {"company": "MDLZ", "doc_type": "10-K", "period": "FY2025"}] Three sources, all Mondelez. The question names three companies. Two are absent from the context. The system answered anyway. I later regenerated this exact question to see what it produced. The model attributed a passage about cocoa-cost pressure to Coca-Cola. The cited passage discussed Mondelez's cocoa costs and came from a Mondelez chunk, but the answer attributed it to Coca-Cola. The citation number resolved correctly, so the answer looked grounded. The company attribution was invented. That is the failure an asset manager cannot tolerate. A cited, confident, grounded-looking answer, wrong about who said it. This was not isolated. One question asked how each company characterized consumer demand. Retrieval returned PepsiCo, Mondelez, and Mondelez, with no Coca-Cola. Another asked whether Coca-Cola's volume commentary changed from Q3 2025 to Q1 2026. All three chunks were dated 2026Q1. The comparison period was missing. Here is what makes this worse than a retrieval miss. My context precision on that run was 0.7967, which looks respectable. It stayed high for a reason. Each Mondelez chunk genuinely is relevant to commodity pressure. The LLMContextPrecisionWithoutReference metric estimates whether retrieved chunks are relevant to the generated response, and whether the useful ones appear earlier in the list. It does not test whether the evidence set covers every company, period, or comparison the question requires. Precision and scope coverage are different properties. The frustrating part came next. The capability to catch this already existed in the codebase. def answer_question( question: str, company: str | None = None, period: str | None = None, ) -> Answer: result = retrieve(question, company=company, period=period) The metadata filter is built. It works. It scopes retrieval by company and period. It was simply never invoked. Nothing inferred scope from the question text, and the harness called answer_question without arguments. The system knew which company every chunk belonged to. It never checked that against what the question asked for. Question Three: When Retrieval Is Right and the Answer Is Impossible Back to the stock price question, and that 0.999. The score is not an error. The reranker did its job. It searched for passages about Coca-Cola's stock and surfaced Item 5 of the FY2025 10-K. That item is titled Market for Registrant's Common Equity, Related Stockholder Matters and Issuer Purchases of Equity Securities. Item 5 was the reranker's strongest topical match among the retrieved candidates. It scored 0.999 because, on topic, it was right. The question is still unanswerable. A static annual filing cannot contain today's price, no matter how much equity discussion it holds. This reframed the whole project for me. Semantic similarity measures whether a passage is about the right subject. It cannot measure whether the passage contains the answer. Those are different questions. My refusal gate was using the first as a proxy for the second. The score distribution proves it. My three highest reranker scores tell the story: 0.9990, the stock price question, was wrongly answered 0.9988, a risk factors question 0.9541, "What is PepsiCo's detailed plan for 2030?", correctly refused despite the high score Two of the top three belong to questions that should have been refused. At the other end, the correctly answered margin question scored 0.4297. A legitimate question about Mondelez's foreign-currency effects scored 0.0560 and was refused. The scores are not ordered by answerability. They are ordered by topical similarity, a different axis entirely. One caveat on the number. The 0.999 is the reranker score used by my pipeline. It is not a calibrated estimate of answerability, and I should not read it as a probability that the question can be answered. Three Gates, but None Checked Evidence Sufficiency I thought I had built a refusal gate. I had actually built three, and the interesting part is not which are deterministic. Two of the three are. The interesting part is that none of them checked whether the evidence could support an answer. Gate one, the score threshold. Deterministic, and it runs before the LLM: def should_refuse_on_retrieval( reranked: list[tuple[Document, float]], threshold: float ) -> bool: if not reranked: return True top_score = reranked[0][1] return top_score str | None: q = question.lower() for phrase, evidence_type in rules.items(): # e.g. "stock price" -> "live_market" if phrase in q: return (f"This question asks for live/current market data " f"(matched phrase '{phrase}'), which this corpus of static " f"SEC filings and transcripts does not contain.") return None Scope coverage check: Parse the company and period combinations from the question names, then compare them against what retrieval actually returned. Two implementation details matter here. First, check company-and-period pairs, not companies and periods independently. A question can need KO in Q1 2026 and PEP in Q1 2026. Retrieval might return KO in Q3 2025 and PEP in Q1 2026 instead. The company set and the period set both look complete, yet KO in Q1 2026 is missing. Pair coverage catches that. Second, sort before reporting. That keeps the refusal message byte-identical across processes, rather than depending on set iteration order: def refuse_on_scope(question, reranked, aliases) -> str | None: required_pairs = extract_company_period_pairs(question, aliases) retrieved_pairs = { (d.metadata["company"], d.metadata["period"]) for d, _ in reranked } missing = sorted(required_pairs - retrieved_pairs) if missing: detail = ", ".join(f"{co}/{pd}" for co, pd in missing) return f"The retrieved evidence does not cover: {detail}." return None Both checks use deterministic information available before generation. The evidence-type check applies keyword rules to the question. The scope check compares the required company-period pairs against metadata attached at ingestion. Neither check requires an LLM call. I verified they are deterministic by running the known-failure questions repeatedly and getting identical output. The scope check does not overfire. A question that names only Coca-Cola still answers, even when a Mondelez chunk happens to appear in the retrieved set. The check requires only the pairs the question asks for, not every pair that was retrieved. That distinction separates a useful coverage check from one that refuses everything. One honest limitation. In this prototype, an incomplete scope means immediate refusal. A production version would first trigger scoped retrieval for each missing company or period. It would rerank the merged evidence and refuse only if coverage still failed. That turns the check into a recovery step. I have not built that yet. What the Fix Actually Changed I re-ran the same fifteen questions with the checks in place, everything else held constant: same 1B model, same threshold, same judge. All figures below compare against the first 1B run, the same baseline used in the opening matrix. Run Answerable accepted Unanswerable refused Overall correct Fabrication caught First 1B baseline 6/12 2/3 8/15 NO 1B + deterministic checks 6/12 3/2 9/15 YES Overall accuracy went from 8 of 15 to 9 of 15, a net gain of one correct decision. Three decisions changed, and they are worth separating, because the net figure hides what actually happened. The stock-price question is now refused, every time, by the evidence-type check. Its reranker score is still 0.999 in the log. The check overrides it deterministically, before the model runs, in 1.6 seconds instead of a full generation. This added one correct refusal. The three-company commodity question is now refused by the scope check. Its reason: "The retrieved evidence does not cover: KO/2026Q1, PEP/2026Q1." The retrieved set held Mondelez evidence but omitted both Coca-Cola and PepsiCo for the quarter. That is exactly the pair-based gap the check is built to catch. The fabricated cocoa attribution cannot occur because the question never reaches generation when the required company-period evidence is missing. This reduced the corpus-level accuracy count by one, because the question was answerable but is now refused. That result exposes a weakness in the metric: it had counted the previously fabricated answer as a success. The scope check made the system safer, even though the headline accuracy moved the wrong way for this query. That is exactly the trade this system exists to make. A third change was unrelated to the checks. A guidance question that the baseline declined to answer was answered in the gated run. Neither new check touched it. That flip is the same temperature-zero nondeterminism from the section above, showing up again, and it restored one answerable question. It is a reminder that the model-made gate remains a moving part. So, the net rose from 8 of 15 to 9 of 15, but the count hides the real work. The evidence-type check added a correct refusal on the flagship failure. The scope check converted a counted-correct but fabricated answer into a safe refusal. That lowered corpus-level accuracy by one, even as it improved safety. A separate model-made flip added one answerable question back. The blocked fabrication matters more than the net score. The RAGAS scores moved slightly in the fixed run's favor. However, that was on a different answered subset and was too small a sample to trust, so I would not read anything into it. The Numbers, With Their Caveats Attached Three caveats belong on every number here. I would rather state them than have a reader find them. Judge is weak: RAGAS scores faithfulness with an LLM, and mine was the 1B local model. A small model judges grounding poorly. Some of that 0.25 may be judgment error rather than unsupported generation. I held the judge fixed across runs so comparisons remain internally consistent, but the absolute value should be interpreted loosely. Sample is small: Fifteen questions, with answer-level metrics over the handful that were answered. Both runs produced occasional judge-side parser exceptions and timeouts. Movements of a few hundredths are not a signal. Hardware is modest: Everything ran on a laptop with local models. That is why one question took over a minute in an earlier run. These are the results this configuration produced, not a general benchmark. At 0.25 faithfulness, this run is nowhere near strong enough to justify deployment, even allowing for judge error. The checks fixed a class of refusal failure. They did not make the generation trustworthy. Those are separate problems. Refusal Is an Architectural Property, Not a Safety Feature It is tempting to treat refusal as something you bolt on at the end. A confidence check, a threshold, a polite decline. That framing produced every failure in this article. The information needed to refuse well is generated early, then thrown away. Ingestion knows which company and period each chunk belongs to. Retrieval knows the composition of the set it assembled. Generation knows whether a claim traces to a passage. By the time a single number reaches the gate, all of that has been compressed out. The fix was not a smarter model. It used metadata I already had before the model ran. Four things I would tell anyone building a retrieval over regulated documents: A topical similarity score is not an answerability signal. The gap between them is where confident wrong answers live. Coverage is separate from relevance. Compare what the question asked for against what retrieval returned, pairing the company with the period, so a partial match cannot pass as complete. Keep evidence-sufficiency checks deterministic where possible and make sure they measure the right property. Determinism does not help when the check reads “topical similarity” instead of “answerability”. A decision left to the model can vary by model and, even at temperature zero, between identical runs. Write the evaluation set before the code, including questions the corpus cannot answer. Without those, a confident failure looks identical to success. That last point is what made this article possible. My fifteen questions existed before any pipeline did. Three were deliberately out-of-corpus, where the correct answer was a refusal. Without those three, the stock price failure would have slipped invisibly. Moreover, without regenerating the three-company answer, I would never have seen the fabricated attribution that had been disguised by a resolved citation. The pipeline runs end-to-end. Every module executes. What it now does, that it did not before, is decline for reasons I can read in a log and reproduce on demand. That is a smaller claim than "it works." It is also the only one the evidence supports.