Better retrieval beat better prompting

For months, the fastest way to make one of our AI agents better was to rewrite its prompt. Sharper instructions, better examples, a cleaner system message. It worked, up to a point. Then we hit a wall that no amount of prompting could climb: the agent was reasoning beautifully over the wrong information.

The problem was not the model. It was retrieval, the layer that decides which pieces of our company knowledge get handed to the model in the first place. We had a search system underneath everything, and it was quietly failing in ways that were easy to miss and expensive to keep.

So we rebuilt retrieval. Here is what was broken, what changed, and why the result mattered more than any prompt tweak we have shipped.

The old system, and why it lied to us

Every AI touchpoint we run, meeting prep, client health summaries, sales research, and the Slack assistant people ask questions in all day, depends on searching the same underlying corpus: meeting transcripts, Slack history, deal records, and our internal knowledge vault of playbooks, SOWs, and standards.

That search was keyword-only. Under the hood it was BM25, the same family of ranking that has powered full-text search for decades. BM25 is fast and good at what it does, but it has one fatal limitation for an AI system: it only matches words that are actually present.

Failure mode: Ask who is frustrated with their reporting? and if nobody had typed the literal word reporting. If they said the dashboards are confusing or I cannot tell what is going on with the numbers, the keyword system returned nothing. The meaning was in the transcript. The words were not.

It got worse under the hood. A single meeting row was only the first 3,900 bytes of the transcript. For a long client call, that meant the first few minutes were indexed and everything after that was invisible to search. A knowledge-vault document was stored as one enormous row, so long documents diluted their own keyword scores and the excerpt you got back was the top of the file, not the passage that answered your question.

The index itself was drifting from reality. Dates could reflect when CI checked out the file, not when the source material was written. Deleted files could live in the index forever. Two Slack channels for the same client could silently overwrite each other if their IDs collided. A capped daily re-index meant older changed records could fall through the gap.

None of this threw errors. It just gave us slightly wrong answers, all day, everywhere. That is the dangerous kind of broken.

Hybrid search: matching meaning, not just words

The core fix was to stop choosing between keyword search and semantic search and run both.

For the hybrid namespaces, each row carries the keyword-searchable text it always had plus a semantic embedding: a vector produced by text-embedding-3-large that captures what the text means rather than only which words it contains. When a query comes in, we run two searches in parallel. One is the BM25 keyword leg. The other is a nearest-neighbor search over embedding space, which can find the dashboards are confusing when you ask about reporting because those ideas live near each other in meaning.

Some ingestion paths still write keyword-only rows, and the search layer is designed to degrade gracefully when embeddings are missing, so a partial rollout never makes the product worse.

Then we combine two ranked lists. That is trickier than it sounds, because a BM25 score and a vector distance are not on the same scale. BM25 returns unbounded scores where higher is better; the vector leg returns cosine distances where lower is better. You cannot just add them.

Implementation note: Our fusion code uses Reciprocal Rank Fusion. Each leg votes based on rank position using 1 / (k + rank), with k = 60. It needs no score normalization, no per-corpus tuning, and degrades cleanly when one ranking is missing.

That last property mattered more than we expected, because it let us ship safely. If the vector leg fails, a namespace without embeddings yet, a rate-limited embedding service, or missing configuration, search can fall back to keyword behavior instead of breaking the product. Embedding failures should make a result less capable, not unavailable.

Chunking: return the passage, not the file

Semantic search does not help if you embed a 1.5MB document as one vector. The meaning smears into mush. So alongside hybrid search we broke documents into passages.

Meetings and vault docs are split into roughly 800-token chunks with a 100-token overlap so ideas that straddle a boundary are not lost. Each chunk becomes its own searchable row, but chunks retain their parent document key. At query time we over-fetch, then collapse results back down to one hit per document while keeping the best-matching chunk. You do not see the same meeting five times, and the excerpt you get is the passage that actually answered the question.

Implementation note: The shared chunker uses an intentionally simple four-characters-per-token heuristic: 800 tokens becomes 3,200 characters, and the 100-token overlap becomes 400 characters. It is dependency-free, easy to test, and consistent across Turbopuffer indexers.

The scale of this is what makes the difference concrete: our knowledge vault went from a few thousand fat rows to more than 100,000 precise chunks. A single large file can produce 500 or more of them. That is 500 places search can now land inside one document, instead of one.

We also gave every row real metadata while we were in there: the client it belongs to, the document type, and the actual date. Now search can filter to a client, a document family, or a time window instead of relying on keyword luck.

Making the index converge to truth

The last piece was making the index converge to the truth instead of drifting from it.

Instead of blindly re-indexing the newest records every night, a sync should ask the index itself for the most recent timestamp it has per source, back up to cover clock skew, and pull everything newer from the database oldest first. Deletions and renames should propagate, but only behind safety gates that refuse to wipe a namespace if the source suddenly looks empty or if too much would be deleted at once. In indexing systems, "the source returned zero rows" is far more often a bug than a real mass deletion.

Resilience principle: We designed indexing so partial failure is visible and recoverable. Embedding calls batch 64 inputs at a time, retry on rate limits and transient 5xx responses, honor Retry-After, and still allow keyword-only rows when semantic enrichment cannot complete.

One search core for every agent

Fixing storage was half the win. The other half was noticing that four different parts of our platform had each grown their own copy of the search logic, subtly different, separately maintained, and drifting apart.

We collapsed them into a shared search core and exposed it through one interface: Global Search MCP. MCP, the Model Context Protocol, is becoming the standard way to connect tools to AI agents. Standing search up behind one interface means agents in Claude Code, Cursor, Slack, and automated jobs can search the same corpus through the same door. Add a capability once, and every agent gets it.

What changed, measured

We ran the old keyword-only system head-to-head against the new hybrid one on real queries against real documents. Relevance went from 59% to 91%. A separate evaluation, scoring whether the right document showed up in the top five results for 30 grounded questions, came back at 93%.

But the number that tells the story is the one we cannot put a clean percentage on: the questions that used to return nothing now return the right thing. Who is unhappy with reporting? finds the client who never said the word. That is not a 30-point improvement on an existing answer. It is the difference between an answer and a blank.

The lesson we keep relearning

Every few weeks something reminds us of the same thing: most "make the AI smarter" problems are really "help the AI find the right thing" problems in disguise. The model was never the bottleneck. The context we fed it was.

Better retrieval beat better prompting, and it was not close. If your AI feels dumber than it should, before you touch the prompt, go look at what it is actually reading.

If you are rebuilding retrieval for agents, hybrid search, chunking, evals, or a shared tool interface, reach out. We are actively shipping this pattern with clients now.