RAG is a retrieval problem, not a prompting problem

Teams reach for a bigger model when their RAG system gives bad answers. In our measurements that was almost never the binding constraint.

The measurement#

We built a 600-question evaluation set against a 2.4M-chunk corpus, with human-labelled correct answers and the specific chunks that support them. Then we changed one thing at a time.

Change	Answer accuracy	Delta
Baseline, 400-token fixed chunks	61.2%	—
Semantic chunking on section boundaries	68.9%	+7.7
Hybrid BM25 + dense retrieval	74.5%	+5.6
Cross-encoder reranking, top 50 → 8	79.1%	+4.6
Query decomposition for multi-hop	88.3%	+9.2
Citation verification pass	91.4%	+3.1

Swapping the generator model for a larger one, at any point in that sequence, was worth between 1 and 2 points.

Fix chunking first#

Fixed-size chunking cuts tables in half and separates a claim from the sentence that qualifies it. Splitting on document structure — headings, list boundaries, table units — and carrying the heading path into the chunk text was the single largest win we measured.

def semantic_chunks(doc: Document, max_tokens: int = 512) -> list[Chunk]:
    """Split on structure, never mid-table, and prepend the heading path."""
    chunks = []
    for section in doc.sections:
        prefix = " > ".join(section.heading_path)
        for block in pack(section.blocks, max_tokens - len(tokenize(prefix))):
            chunks.append(Chunk(text=f"{prefix}\n\n{block.text}", meta=block.meta))
    return chunks

Then measure retrieval separately#

Recall@k tells you whether the answer was even available to the model. If recall@8 is 70%, your ceiling is 70% — no prompt fixes that. Instrument the two stages independently or you will spend weeks tuning the wrong one.

When a bigger model does help#

Multi-hop synthesis, and long-context reasoning over many retrieved passages. Both are generation-side problems, and both only appear once retrieval is good enough to surface the right evidence.