What Building My First RAG Application Taught Me

I thought connecting an LLM to my documents would be the hard part. It wasn't. The real challenge was getting the AI to retrieve the right information, use it correctly, and admit when it didn't know. I thought building a RAG application would be straightforward: take some documents, split them into chunks, generate embeddings, put the vectors into a database, retrieve the most relevant chunks, send them to an LLM, and get an answer. That's the diagram you see everywhere, and technically, it works — you can build a basic RAG prototype surprisingly quickly. What nobody tells you is that getting a RAG application to give consistently useful answers is a completely different problem. The LLM is often not the hardest part. The vector database isn't the hardest part. Even prompt engineering isn't the hardest part. The difficult part is making sure the model receives the right information at the right time. I learned this after building my own RAG application, and the experience changed how I think about AI development. What Is RAG Actually Trying to Fix? Let's start with the problem. Large language models are very good at generating natural language, but generation isn't the same thing as having access to the right knowledge. Imagine that I want to build an AI assistant that can answer questions about a private collection of documents — company documentation, product manuals, internal policies, PDFs, research papers, technical documentation, customer support articles, recently updated information. I can't expect a general-purpose LLM to automatically know all of this, and I don't want to retrain the entire model every time a document changes. This is one of the problems RAG addresses. Retrieval-Augmented Generation combines information retrieval with language generation: instead of asking the LLM to answer from its learned knowledge alone, the application first retrieves relevant information and passes it to the model as context. The simplified architecture looks like this: text RAG APPLICATION User Question │ ▼ Query Embedding │ ▼ Vector Search │ ▼ Relevant Chunks │ ▼ Context Construction │ ▼ LLM │ ▼ Generated Answer Simple enough — but every step in that pipeline can fail, and that's where things get interesting. Why Not Just Use an LLM? This was one of the first questions I asked myself: if modern models know so much, why do we need retrieval at all? There are several reasons. 1. Knowledge can become outdated A language model doesn't automatically know about information that appeared after its relevant training data. If your documentation changes today, the model doesn't magically receive the update. 2. Private data isn't part of the model's general knowledge Your company's internal documentation isn't automatically available to a public LLM. Neither are your private PDFs, databases, product specifications, or internal knowledge bases. 3. Fine-tuning isn't a database Fine-tuning is useful for changing model behavior, style, and learned patterns, but it isn't necessarily the best mechanism for constantly changing factual information. If your knowledge base changes every week, updating a retrieval index can be much more practical than retraining a model. 4. Hallucinations are still a problem An LLM can generate a convincing answer that isn't supported by reality, and RAG doesn't magically eliminate hallucinations. What it does is give the application a mechanism for providing external evidence to the model — which is a much more useful way to think about it. The Architecture I Built I separated my application into two pipelines: one that handles documents, and one that handles questions. The ingestion pipeline text Documents │ ▼ Text Extraction │ ▼ Chunking │ ▼ Embeddings │ ▼ Vector Database The ingestion process turns unstructured documents into searchable pieces of information. The question-answering pipeline looks like this: text User Question │ ▼ Query Embedding │ ▼ Similarity Search │ ▼ Top-K Chunks │ ▼ Prompt + Context │ ▼ LLM │ ▼ Answer At first, I thought of RAG as: "An LLM connected to a vector database." After working on it, I realized that this description hides most of the engineering. A better description is: A retrieval system connected to a generation system. And if the retrieval system is bad, the LLM is starting with bad information. Step 1: Turning Documents Into Chunks The first real problem appeared before I even reached the LLM: documents aren't naturally optimized for semantic search. Suppose a document looks like this: text Authentication Introduction Authentication allows users to access the API. Token Lifetime Access tokens expire after 60 minutes. Refresh Tokens Refresh tokens can be used to obtain a new access token. I can't simply treat the entire document as one giant piece of knowledge — instead, I need to divide it into smaller chunks. Conceptually: text Document │ ├── Chunk 1 ├── Chunk 2 ├── Chunk 3 └── Chunk 4 This sounds like a simple preprocessing step. It isn't. Chunking Is a Retrieval Problem Imagine a user asks: How long does an access token remain valid? Now imagine the chunking process produces: text Chunk A: Access tokens are used to authenticate API requests... Chunk B: Access tokens expire after 60 minutes... If the retrieval system returns only Chunk A, the LLM has technically relevant information — but it doesn't have the answer. Now consider the opposite: suppose I create enormous chunks containing several thousand words. The answer might be somewhere inside them, but the retrieved context also contains a lot of irrelevant information. So chunking becomes a trade-off. text Small chunks │ ├── More precise retrieval └── Less surrounding context Large chunks │ ├── More context └── More irrelevant information There is no universal chunk size that works for every dataset. The right strategy depends on document structure, question types, content length, formatting, the retrieval model, the context window, and the downstream LLM. This was one of my first major lessons: RAG quality starts before the LLM sees the question. Step 2: Embeddings After splitting the documents into chunks, I needed a way to search them semantically — that's where embeddings come in. An embedding represents a piece of text as a vector. For example: text "How long does an access token last?" │ ▼ [0.12, -0.43, 0.81, ...] A semantically similar sentence should have a relatively similar representation, so these two questions can be related even though they use different words: text When does my access token expire? How long is the API token valid? A semantic search system can recognize that relationship — much more useful than simply searching for exact keywords. Step 3: Storing the Vectors The generated vectors and their source information need to be stored somewhere. A simplified record could look like this: python { "id": "chunk-42", "text": "Access tokens expire after 60 minutes.", "embedding": [...], "metadata": { "source": "authentication.md", "section": "Token Lifetime" } } The metadata turned out to be more useful than I initially expected. It helps with: debugging; filtering; displaying sources; tracing answers back to documents; evaluating retrieval; updating individual documents. A vector without useful metadata is much harder to reason about later. Step 4: Retrieving Relevant Context When the user asks a question, I create an embedding for it, then search the vector database for nearby vectors. Conceptually: text Question │ ▼ Query Vector │ ▼ Similarity Search │ ▼ Top-K Results For example: text 1. authentication.md — Token Lifetime 2. authentication.md — Refresh Tokens 3. authentication.md — Access Tokens 4. security.md — API Authentication 5. login.md — User Authentication Those results are then passed to the LLM — and this is where I discovered the biggest problem with my original mental model. The Retrieval Problem Nobody Warns You About I initially assumed that if the answer existed in my documents, the vector search would find it. That's not guaranteed. Suppose my knowledge base contains: text Password Reset Password Policy Authentication Account Recovery API Authentication The user asks: How can I reset my password? Several documents may contain words like password, account, authentication, security, or recovery. The retrieved chunks may therefore be related to the question without actually containing the best answer. That's the difference between: semantic similarity and useful retrieval. A chunk can be similar to the query and still be the wrong evidence. My Biggest RAG Mistake When my first answers weren't good enough, I did what many developers probably would: I changed the prompt. I tried things like: text You are an expert AI assistant. Then: text You are a highly accurate AI assistant. Then longer instructions. Then even longer instructions. Some changes helped, but they didn't solve the underlying problem: the problem was retrieval. The LLM can't reason about information it never received. That sounds obvious, but when you're looking at the final answer, it's easy to forget. I eventually started asking a different debugging question: Did the model give the wrong answer, or did I give the model the wrong information? That distinction changed everything. Debugging RAG Means Inspecting the Pipeline Instead of looking only at the final response, I started inspecting the intermediate steps. For every question, I wanted to see: text User Question │ ▼ Retrieved Chunks │ ▼ Similarity Scores │ ▼ Final Context │ ▼ LLM Prompt │ ▼ Generated Answer Now an incorrect answer could be classified. Failure #1: Retrieval failure The correct information wasn't retrieved. Failure #2: Ranking failure The correct chunk was retrieved but ranked too low. Failure #3: Context failure The relevant chunk was retrieved, but too much irrelevant context surrounded it. Failure #4: Generation failure The correct information was provided, but the model misunderstood or ignored it. Without this separation, all four problems look identical: "The AI gave me a bad answer." They aren't identical, and they require different solutions. RAG Doesn't Eliminate Hallucinations This was another assumption I had to abandon. I initially thought: If I provide the correct documents, the model will produce the correct answer. Not necessarily. The model can still: misinterpret the context; combine unrelated facts; infer information that isn't explicitly present; ignore an important detail; answer despite insufficient evidence. That's why I added a simple but important instruction to the generation stage: text Answer using only the provided context. If the context does not contain enough information to answer the question, say that you don't know. Do not invent missing facts. This doesn't make hallucinations impossible, but it establishes a much better default behavior. More Context Can Actually Make Things Worse One of the most counterintuitive things I learned was that retrieving more information isn't always better. Suppose the retrieval system returns 20 chunks: text Chunk 1 → highly relevant Chunk 2 → highly relevant Chunk 3 → relevant Chunk 4 → somewhat relevant Chunk 5 → unrelated ... Chunk 20 → unrelated The model now has more information, but it also has more noise, and the relevant information can become harder to identify. This means top-k isn't simply a performance parameter. It's a quality parameter. You need enough context to answer the question — but not so much that the answer gets buried. Where My RAG Application Broke Once I started testing more seriously, several failure modes became obvious. Failure What happened What I learned Wrong chunk Relevant information wasn't retrieved Improve retrieval Poor ranking Correct chunk appeared too low Tune ranking Broken context Answer was split across chunks Improve chunking Too much context Relevant information was buried Reduce noise Similar documents Several chunks looked equally relevant Add metadata/filtering Missing information Model tried to answer anyway Add refusal behavior Poor formatting Tables/headings lost meaning Improve ingestion This table summarizes something that took much longer to understand in practice: RAG is not one problem. It's a chain of problems. The Simple RAG Implementation The core retrieval logic doesn't have to be complicated. A simplified Python implementation looks something like this: python def retrieve_context(question, vector_store, embedder, top_k=5): query_embedding = embedder.embed(question) results = vector_store.search( query_embedding, top_k=top_k ) return [ { "text": result.text, "source": result.metadata.get("source"), "score": result.score } for result in results ] Then the retrieved context can be passed into the generation step: python def build_prompt(question, context): context_text = "\n\n".join( item["text"] for item in context ) return f""" Answer the question using only the context below. If the context does not contain the answer, say that you don't know. Context: {context_text} Question: {question} """ And then: python context = retrieve_context( question, vector_store, embedder, top_k=5 ) prompt = build_prompt(question, context) answer = llm.generate(prompt) This is the basic idea behind a RAG pipeline, but production quality doesn't come from these few lines — it comes from everything surrounding them. What I Would Do Differently Today If I rebuilt the application from scratch, I would change several things. 1. Build an evaluation dataset first This is probably my biggest change: instead of manually asking random questions, I would create a small test set containing a question, an expected answer, and the relevant document and chunk. For example: text Question: How long is an access token valid? Expected: 60 minutes Source: authentication.md Section: Token Lifetime Then every change to chunking, retrieval, ranking, or prompting could be evaluated against the same questions. Without this, it's easy to say: "This version feels better." That's not the same as proving that it is better. 2. Test Retrieval Before Testing Generation This is another major lesson. Before asking an LLM to generate an answer, I would test: Did the system retrieve the correct evidence? If the answer is no, changing the LLM won't fix the retrieval problem — and drawing that line makes the whole system much easier to debug. 3. Experiment With Chunking Earlier I spent too much time on the generation stage. Today, I'd test different chunking strategies much earlier: fixed-size chunks; overlapping chunks; section-based chunks; semantic chunks; metadata-aware chunks. Different documents require different strategies: a technical manual isn't structured the same way as a collection of support tickets. 4. Preserve Document Structure I would also preserve more information during ingestion. Instead of storing only: text text + embedding I'd keep: text document section heading page source document_type created_at This makes filtering and debugging much easier, and it also makes source attribution possible. 5. Consider Hybrid Search Vector search is excellent for semantic similarity. But semantic similarity isn't always enough. Imagine the user searches for: text ERR_CONNECTION_RESET or: text API v3 /users/{id} or: text Python 3.12 Exact keywords, identifiers, error codes, and version numbers can matter enormously — that's why I'd consider combining semantic retrieval with traditional keyword search. In many real-world systems, hybrid retrieval can be more useful than relying on vectors alone. How Long Did It Take? The first prototype was much faster to build than I expected — but that's where the distinction between a demo and a system became obvious. The initial version can be summarized as: text Documents ↓ Embeddings ↓ Vector Database ↓ LLM ↓ Answer The real work came afterward: text Prototype ↓ Testing ↓ Debugging ↓ Chunking experiments ↓ Retrieval experiments ↓ Prompt improvements ↓ Failure handling ↓ Evaluation So if you're planning your first RAG project, don't estimate the project based only on the time required to make the first demo work. The demo is the easy part. Reliability is the project. What Building RAG Taught Me About AI Engineering The biggest lesson wasn't about embeddings. It wasn't about vector databases. It wasn't even about LLMs. It was about systems thinking. Before building this application, I mostly thought about AI applications like this: text Question → Model → Answer After building RAG, I started seeing a much longer pipeline: text Raw Data ↓ Parsing ↓ Chunking ↓ Embeddings ↓ Indexing ↓ Retrieval ↓ Ranking ↓ Context Construction ↓ Prompt ↓ LLM ↓ Answer ↓ Evaluation Every stage can introduce errors, and when the final answer is wrong, the most important question isn't: "Why did the AI fail?" It's: "At which stage did the failure enter the system?" That's a much more useful engineering question. RAG Is Not Magic The more I worked with RAG, the less I saw it as an AI trick — it's an architecture, and the LLM is only one component. The quality of the final application depends on: source data quality; document parsing; chunking; embedding quality; retrieval; ranking; context construction; prompting; model behavior; evaluation. A powerful model cannot completely compensate for terrible retrieval, and perfect retrieval cannot solve every limitation of an LLM. Both parts have to work together. The Question I Would Ask Before Choosing an LLM When developers start an AI project, one of the first questions is usually: Which model should I use? That's important — but after building a RAG application, I would start somewhere else. I'd ask: What information does my application need to answer correctly? Then: Where does that information live? Then: How reliably can I retrieve it? Only after answering all three would I start optimizing the model. That change in thinking saved me from spending too much time optimizing the wrong layer. Would I Build RAG Again? Yes. But I wouldn't use RAG simply because it's popular — I'd use it when an application genuinely needs access to: private knowledge; changing information; domain-specific documentation; large document collections; external knowledge that shouldn't be baked into the model. And I would start much smaller. Not: "Let's build an intelligent enterprise knowledge platform." Instead: "Let's make the system answer these 20 questions correctly." If it can't reliably answer those 20 questions, adding another 100,000 documents probably won't solve the problem. Neither will blindly switching to a larger model. The Most Important Lesson Building my first RAG application changed the way I think about AI. The impressive part isn't making an LLM answer a question. The difficult part is making sure it receives the right information, uses that information correctly, and knows when the information isn't enough. That's what makes RAG both powerful and frustrating. A basic RAG demo can be built quickly. A reliable RAG application requires engineering. If you're building your first one, here's what I would recommend: Start small. Create an evaluation dataset early. Inspect your retrieval results. Separate retrieval problems from generation problems. Don't assume more context means better answers. Preserve document metadata. Don't spend three hours rewriting a prompt when the real problem is your retrieval pipeline. The first RAG demo might take an afternoon. Building one you can actually trust is the real challenge — and that's the part nobody tells you about.