Chunking Strategies for Structured Data in RAG Systems

While developing RAG pipelines across various enterprise use cases, a stakeholder asked me a question that stopped me mid conversation: "Can our database become a source for our knowledge base, just like our SharePoint documents? Can RAG give us answers by querying the database directly?" The honest answer is yes, but only if you rethink how you chunk the data before it enters the knowledge base. Most RAG chunking strategies are optimized for prose, i.e. documentation, articles, support tickets, web content, etc. The moment you ingest a CSV or metadata catalog export, they break down. Here's why: a table with 50 rows becomes a single chunk whose embedding captures a blurred average of all rows. When a user asks "What is the city with ID=5?", the retriever can't isolate that specific row because the chunk represents everything and nothing at once. This article covers six chunking strategies for structured data, with honest trade offs for each and guidance on when to use which. Strategy 6 touches on the agentic SQL routing approach but does not cover full agentic SQL implementation in depth. I will write a separate article for the same. Why Default Chunking Fails on Tables Fixed-size (512 tokens) and sentence-based chunking assume contiguous text carries contextual meaning. That a paragraph depends on surrounding paragraphs. Tables violate every one of these assumptions: Rows are independent facts — there's no relational "flow" between row 4 and row 5 Each row is self-contained — it doesn't need neighboring rows for interpretation A table embedding averages N unrelated facts — it becomes a weak match for any single fact query Fixed-size splits cut mid-row — producing chunks where half a record is orphaned from its values Overlap creates duplicate retrievals, not helpful continuity The result: your knowledge base confidently returns "I don't have enough information" for data that's sitting right there. Strategy 1: Row-Level Chunking Each row becomes its own chunk, serialized with schema context so it's self-explanatory: Table: prod_db.us_cities Columns: id, city_name, state, population, region --- Record: id=5, city_name=Seattle, state=WA, population=737015, region=Pacific Northwest In table us_cities, for this record: the id is 5; the city_name is Seattle; The state is WA, the population is 737015, the region is Pacific Northwest. The dual representation maximizes retrieval across different query phrasings. structured key=value + natural language prose Pros: Highest retrieval precision for point queries Simple to implement Pairs naturally with metadata filtering (.metadata.json sidecars for table/record filtering) Cons: Chunk explosion at scale. 100K rows means 100K chunks with 100K embedding calls Cannot answer aggregation queries (COUNT, SUM, AVG) since retrieval fetches top-k, not all qualifying rows Redundant schema headers in every chunk add ~30% storage overhead Suitable for: I have seen it getting used for lookup queries on small to medium tables ( 1M?" "Describe the Seattle record" "Average population by region" Specific entity lookups COUNT, SUM, AVG, GROUP BY, TOP-N, JOINs The Agent's instructions define routing logic, and the model distinguishes between "give me a specific record" from "compute something across records" without a separate classifier. Pros: Best of both worlds retrieval precision of RAG + computational power of SQL Aggregation works natively (the database computes, not the LLM) No chunk explosion on the SQL path Scales to millions of rows Cons: Two systems to build and maintain (KB + Athena + routing Agent) Athena latency adds 2–10s per query LLM-generated SQL can be syntactically valid but semantically wrong Ambiguous queries may route to the wrong engine Must keep schema context in the agent prompt synced with actual Glue Catalog Suitable for: Production systems with mixed query patterns, large datasets (>100K rows), enterprise use cases requiring accurate numerical answers, data platforms built on Glue + Athena Best Practices Always include schema context (table name, columns) in every chunk Use metadata filtering to narrow retrieval scope by table or partition Include both structured (key=value) and prose representations Don't fight RAG's limitations, route analytical queries to SQL Test with your actual user queries, not hypothetical ones Practical Recommendations After designing many RAG applications both at POC and Production scale, my recommendation is to start small to understand user query requirements at POC/MVP scale. Then move towards Agentic Approach. Start here: Row-Level Chunking (Strategy 1) with schema headers. It solves the most common failure mode (point lookups returning nothing) and takes an afternoon to implement. Graduate to Hybrid RAG + SQL (Strategy 6) when users start asking aggregation questions. RAG fundamentally cannot COUNT or SUM across all rows. It retrieves the top-k most similar, not all qualifying. Layer Schema Aware chunks (Strategy 3) on top of whatever base strategy you choose they're essentially free and dramatically improve the LLM's understanding of your data. Structured data is where most RAG implementations quietly fail, not because the technology is wrong, but because the chunking strategy was designed for prose, and not for structured data. The six approaches here aren't a menu to pick one from. They are a layered toolkit. Start simple, measure where retrieval breaks down, and add complexity only where your data and query patterns demand it.