Why Legal, Finance and Healthcare Documents Break Naive RAG — and How Structure-Aware Retrieval Fixes It

The tutorial RAG pipeline and where it breaks

The default recipe in most tutorials is simple: extract text, split it into fixed windows of a few hundred tokens with some overlap, embed each window, and fetch the top-k nearest vectors for every question. It is a good way to learn the concepts. It is also the reason so many pilots that looked brilliant on a handful of web pages disappoint on real enterprise documents.

The problem is that fixed windows are blind to what a document is. A boundary lands wherever the token counter says it should — mid-sentence, mid-clause, mid-row. The retrieved fragment looks relevant, the model dutifully answers from it, and the answer is confidently incomplete.

Legal documents: hierarchy, cross-references and defined terms

A contract is a tree. A clause such as "14.2 Termination for cause" has sub-clauses (a), (b), (c), refers to defined terms like "Material Breach" that are explained pages earlier, and points to other clauses by number. Split it every 500 tokens and one of three things happens: the sub-clauses are separated from the clause that gives them meaning, the defined term never appears in the same chunk as the sentence that uses it, or a cross-reference like "subject to Clause 9.4" is retrieved without the clause it points to.

What helps:

• Chunk on the hierarchy. Keep a clause together with its sub-clauses and prefix each chunk with its section path (for example "Master Services Agreement › 14 Termination › 14.2").
• Keyword search for citations. Lawyers search for "section 138" or "Clause 14.2". Semantic embeddings are weak at exact identifiers; keyword search is strong.
• Related-passage expansion. After finding the best chunk, bring in its neighbours, its whole section, or other passages mentioning the same defined term.
• Cite precisely. Every answer should point to the clause or paragraph it came from so a reviewer can verify it in seconds.

Financial documents: tables, totals and units

Bank statements, invoices, rate cards, fund factsheets and annual reports are dominated by tables. Flatten a table into a paragraph and the header is separated from its values; chunk it by token count and half the rows end up in another chunk. Then the model is asked to add up numbers it has seen only in fragments — a task language models are unreliable at.

What helps:

• Keep small tables whole; index one entry per row for large ones, with the header and the table caption attached to every row.
• Compute, don't estimate. When the question is "total spent with this vendor", retrieve all matching rows and calculate the total from them, returning the rows as citations.
• Preserve units and periods. A figure without "USD thousands" or "FY2025" is a trap.
• Protect account numbers before they are embedded (see our article on private-by-design RAG).

Healthcare documents: lists that must come back complete

Discharge summaries, drug monographs and clinical guidelines are full of lists: medications, contraindications, eligibility criteria, dosage steps. A fragment containing four of six medications is worse than no answer, because it looks complete. Chunking must treat a list as one unit, and the retrieval layer should favour completeness over brevity for list-type questions.

It is equally important to be honest about scope. A retrieval system supports clinical and administrative staff in finding information; it is not a diagnostic tool and should never present itself as one. troveGEN, for example, states explicitly that it is not a medical device and does not give clinical advice.

Government and public-sector documents: reference numbers and languages

Circulars, notifications, tenders and orders are identified by reference number, date and department. Citizens and officials alike search by those identifiers — "Circular 24/2025 of the Finance Department" — and by meaning in several languages. This is a textbook case for hybrid search: keyword matching handles the identifier and semantic search handles the paraphrase. Multilingual quality needs to be verified end to end, from OCR through retrieval to answers, rather than assumed.

Hybrid search and reciprocal rank fusion

Hybrid search runs two retrievers over the same corpus: a keyword retriever (typically BM25) and a dense vector retriever. Their scores are not comparable — BM25 scores are unbounded while cosine similarity lives in a fixed range — so the lists are usually fused by rank. The most widely used method is reciprocal rank fusion (RRF), introduced by Cormack, Clarke and Büttcher at SIGIR 2009. Each list contributes 1 / (k + rank) for every document it contains, and the contributions are summed; the original paper used k = 60, which remains a common default in search engines. A passage that ranks well in both lists rises to the top, and a passage missing from one list is simply not boosted by it.

Reranking and context: finding the needle, then placing it well

First-stage retrieval is built for speed and recall. A reranker — typically a cross-encoder or an LLM that reads the question and each candidate passage together — spends more compute on a short list of perhaps 20–50 candidates and returns the best few. The combined effect of layered techniques can be large. Anthropic's Contextual Retrieval write-up, which prepends a short explanatory context to each chunk before embedding and before BM25 indexing, reported that contextual embeddings plus contextual BM25 reduced the number of failed top-20 retrievals by 49%, and that adding reranking brought the reduction to 67%. These are the vendor's own benchmark figures and will vary with your corpus, but the direction is consistent with what practitioners see.

What you place into the prompt matters too. Liu et al. found in Lost in the Middle that models are less reliable at using information buried in the middle of a long context. So after reranking, pass fewer, better passages, and put the strongest evidence first.

A comparison of chunking strategies

Strategy	Works well for	Breaks on
Fixed-size windows	Prose such as blog posts and FAQs	Contracts, tables, lists, code
Recursive text splitting	Mixed prose with paragraphs	Tables and deep hierarchies
Structure-aware (headings, sections, rows)	Legal, financial, clinical, government records	Needs good parsing and OCR first
Structure-aware + contextual prefixes	Large corpora with ambiguous chunks	Higher indexing cost

How troveGEN approaches industry-specific retrieval

troveGEN is built around the idea that retrieval is where RAG fails, so it concentrates there:

• Structure that survives. Documents are split on headings; tables stay whole or become one entry per row; code and formulas are never cut.
• Search by words and by meaning. Keyword and semantic search run side by side, are fused and reranked, with filters on any attached field applying to both.
• Related passages together. Pull in neighbouring passages, the whole section, or everything mentioning the same entity.
• OCR that refuses to guess. Scans are read in the declared language, and unreadable pages are flagged rather than indexed as empty.
• Industry settings. Finance, healthcare, legal, hospitality, government and education profiles, with an Auto mode that classifies documents at ingestion.
• Prove it on your documents. Build test questions, run scored evaluations and compare runs side by side before you ship.

The practical outcome is that "a clause stays with its sub-clauses", "clinical lists come back complete, never a fragment" and "statements you can search and add up" are properties of the pipeline rather than hopes about the model. Explore the industry pages at trovegen.com/industries, or contact Intellara to design a pipeline for your own corpus.

A quick audit for your own RAG pipeline

1. Pick ten real questions and print the exact chunks retrieved for each. Are they complete?
2. Search for an exact identifier — a clause number, invoice number or circular reference. Is it found?
3. Ask a question whose answer is in a table. Is the total computed from rows or guessed?
4. Ask something the documents do not contain. Does the system refuse?
5. Change one chunking or retrieval setting and re-run your golden set. Can you see the score move?

Further reading

• Anthropic, Introducing Contextual Retrieval.
• Cormack, Clarke and Büttcher, Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods, SIGIR 2009.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts.

Frequently asked questions

Why does naive fixed-size chunking fail on legal and financial documents?

Fixed-size chunking splits text every N tokens regardless of meaning. In a contract it can separate a clause from its sub-clauses or defined terms; in a financial statement it can separate table rows from their headers. The retrieved fragment then looks plausible but is incomplete, and the model answers from the fragment.

What is hybrid search in RAG?

Hybrid search runs keyword search (such as BM25) and semantic vector search side by side, then fuses the two ranked lists, commonly with reciprocal rank fusion. Keyword search finds exact identifiers and reference numbers; semantic search finds passages that mean the same thing in different words.

Does reranking really improve RAG accuracy?

Yes. First-stage retrieval is fast but approximate. A reranker reads the question and each candidate passage together and reorders a short list. Anthropic's published Contextual Retrieval results, for example, report that adding reranking on top of contextual embeddings and BM25 cut failed retrievals by 67% in their benchmark.

How should tables be handled in RAG?

Keep tables whole when they are small, or index one entry per row with the header attached when they are large, and compute totals from the retrieved rows instead of asking the model to add numbers it has only seen in fragments.

How does troveGEN handle industry-specific documents?

troveGEN splits documents on their natural structure, keeps tables and code intact, runs keyword and semantic search together with optional reranking, and can be tuned for finance, healthcare, legal, hospitality, government and education. See trovegen.com/industries for details.