What breaks in a RAG pipeline before the model does

Every RAG demo works. The demo has eight documents, one language, and a question the author already knew the answer to. Production has ten thousand documents, three languages, and a user asking something the knowledge base never covered.

Almost none of the failures are the model's fault.

Chunking is a retrieval decision, not a preprocessing step

Fixed-size chunks with a fixed overlap are the default because they're easy, not because they're good. A 512-token window slices tables in half and severs a heading from the paragraph it governs.

def chunk(doc: Document, target: int = 512, overlap: int = 64) -> list[Chunk]:
    """Split on structure first, size second."""
    sections = split_on_headings(doc)
    out: list[Chunk] = []
    for section in sections:
        if token_count(section) <= target:
            out.append(Chunk(text=section.text, heading=section.heading))
            continue
        out.extend(sliding_window(section, target, overlap))
    return out

Carrying the heading onto every chunk derived from a section costs a few tokens and buys a large accuracy win: the embedding now contains the context a human would have used.

Tenant isolation belongs in the vector store

For multi-tenant products, filtering results after retrieval is not isolation — it's a bug waiting for a bad k value. Isolation has to happen inside the search.

A payload filter is usually the right trade: one collection, tenant id enforced in the query itself, no cross-tenant path that depends on application code remembering to filter.

You need an eval set before you need a better prompt

Thirty real questions with known-good answers, run on every change. Without it, "improving the prompt" is superstition — you cannot tell a real gain from noise.

Track two things separately:

1. Retrieval hit rate — was the answer in the retrieved context at all?
2. Answer quality — given correct context, was the response right?

Conflating them sends you tuning the prompt when the retriever never surfaced the document.