In a RAG (Retrieval-Augmented Generation) system, before the LLM answers, you fetch a handful of relevant chunks from your knowledge base and hand them over as context. That fetch is retrieval — and it decides everything: the right chunks and the model answers well; the wrong ones and it just hallucinates.
So "retrieval accuracy" comes down to one question: did the right chunk reach the LLM? Everything here is a different way to enhance it — each fixing a different reason it didn't. I'm skipping chunking (that's a separate post) to focus purely on how you search — starting with why the obvious approach falls short.
Embed the query, grab the top-k by cosine similarity, done — that's the starter kit, and on any serious corpus it leaves a lot of right answers on the floor.
The floor in 2026 is higher than that: hybrid search plus a reranker. That's the baseline you should assume, and it's cheap. Everything beyond it — the query tricks, the structured sources, the agent loops — is worth adding only when you can name the failure it fixes.
Let's start by seeing exactly where plain vector search breaks.
Vector search is simple: embed every chunk into a vector once, and at query time embed the question and return the chunks whose vectors sit closest. It's great at meaning — "how do I keep my data safe?" finds a chunk about "encryption and access controls" even with no shared words.
That semantic matching is the whole appeal — but it's also where the blind spots come from. Four failure modes show up again and again, and each one names a camp of fixes:
1 · The question is worded badly. Vague, conversational, full of pronouns ("is it down again?"), or just phrased differently from the answer. The search is only as good as the query you feed it.
2 · It misses exact terms. Product codes, error IDs, names, acronyms — "X-200" or "ERR_507" — get blurred into the vector. Semantics can't do exact match, and keyword search would've nailed it.
3 · Close isn't the same as best. The top-k are topically near the query, but the one truly correct chunk is sitting at rank 8, buried under lookalikes.
4 · Some questions aren't a lookup at all. "Total revenue across all regions?" needs a sum from a database; "what are the main themes?" needs the whole corpus. No single chunk holds the answer.
Keep those four in mind — every technique below is aimed at one of them.
Here's the mental model that ties it all together. The top row is the path a query actually takes; each camp below sits under the step it fixes.
The cheapest wins often come before you search at all. The user's raw question is frequently the weakest link — so clean it up, or turn it into something the index can match.
The simplest move: have an LLM rewrite the raw query into a cleaner, self-contained one — resolve pronouns, pull in context from the chat history, fix typos, spell out what the user actually means. In a conversational assistant it's basically mandatory, since follow-up questions are full of "it" and "that."
Cheap, low-risk, high-ROI — one small LLM call, and it's basically table stakes for chat-based RAG. The only real downside: on a query that was already clean, the rewrite is wasted effort, and a careless rewrite can drift from what the user meant.
Here's a neat insight: a short question and the passage that answers it often don't look much alike in embedding space. A question and an answer are different kinds of text. So HyDE (Hypothetical Document Embeddings) flips it — ask the LLM to write a fake answer first, then embed that and search with it. A plausible answer sits much closer to the real answer-docs than the question ever did.
1The problem: the question is short and vague, the answer is long and technical. As vectors, they're not all that close — so the right chunk can rank lower than it should.
2Ask the LLM to write a hypothetical answer — no retrieval yet. It may get details wrong, and that's fine; we only need it to look like a real answer.
3Search with the fake answer's vector. It looks like a real answer, so it lands right on the real one — the security chunk that the bare question left at rank 6 now sits at rank 1.
Often the single highest-ROI query trick, especially on knowledge-dense corpora. The cost is one extra generation per query (latency), and the sharp edge is that a confidently wrong hypothetical can steer retrieval astray — so it helps most when a good answer is guessable, and least on rare, novel entities the model can't imagine. Introduced in Gao et al., 2023.
One phrasing of a question only reaches so far — synonyms and alternate wordings pull in different chunks. So generate several versions of the query, search each, and merge. The trick is how you merge: a naive union just dumps everything together. RAG-Fusion merges intelligently with Reciprocal Rank Fusion (RRF) — chunks that rank well across several variants float to the top.
1Ask the LLM for a few alternate phrasings of the same question. Each casts a slightly different net over the index.
2Search each variant separately — three ranked lists. Notice the "Cancel a plan" chunk shows up in all three, but never clearly on top; different distractors clutter each list.
3Reciprocal Rank Fusion. Every chunk earns points from each list based on its rank — higher rank, more points — and the points add up. Appearing across multiple lists beats ranking #1 in just one. No score-scaling or tuning needed.
4The chunk that was consistently relevant — but never obviously #1 — rises to the top, while one-list distractors sink. That's the win over a naive union.
Solid recall boost, and RRF is a genuinely useful little primitive you'll see again in a second. The cost is real though: N× the retrieval calls plus a generation call, so more latency. Reach for it when phrasing sensitivity is hurting recall, not by default. Coined in "Forget RAG, the Future is RAG-Fusion".
No amount of query polish helps if the engine underneath is half-blind. Pure vector search can't do exact matches — and that's exactly what this camp fixes. This is the highest-leverage camp, and where your baseline should live.
The core idea: run two searches and merge them. A dense (vector) search for meaning, and a sparse keyword search — classic BM25 — for exact terms. Each covers the other's blind spot: dense misses "X-200", BM25 misses "won't boot" ≈ "fails to power on". Merge the two ranked lists (with RRF again) and you get the best of both.
1This query has both a meaning part ("won't boot") and an exact-term part ("X-200"). Send it to two engines at once — a semantic one and a keyword one.
2Each engine half-solves it. Dense understands "won't boot" but treats "X-200" as noise; BM25 locks onto "X-200" but can't tell a boot fix from a spec sheet. The right chunk is stuck mid-list in both.
3RRF merges the two lists by rank — no need to reconcile their incompatible score scales. The one chunk that ranked decently in both wins outright and jumps to #1.
This is the production default — every major vector database ships it, RRF needs no tuning, and it's the single biggest robustness upgrade over vector-only. The only real cost is running and maintaining two indexes instead of one.
The dense side is only as good as the embedding model behind it — so a stronger model is often the single highest-ROI change you can make, lifting every query at once. Worth a look: OpenAI text-embedding-3, Cohere Embed v4, Voyage voyage-3-large, Google Gemini Embedding, or open-weight Qwen3-Embedding, BGE, and NV-Embed — the MTEB leaderboard ranks them, but always test on your own corpus.
You can also fine-tune one on your domain, but treat it as a last resort: labeling data and running a training pipeline is far more effort than anything else here. Reach for a stronger off-the-shelf model first.
Your first-stage search is built for speed over a huge index, so it's a little coarse — the truly best chunk often lands in the top 50 but not the top 5. This camp does a second, slower, sharper pass over just those candidates.
Here's the key difference. First-stage search is a bi-encoder: it embeds the query and each chunk separately, so it never actually compares them word-for-word — fast, but coarse. A reranker is a cross-encoder: it reads the query and one chunk together in a single pass and scores how well they truly match. Too slow to run over millions of chunks, perfect over the top ~50.
1First-stage search returns ~8 candidates fast. They're all about billing, so the truly correct one — the refund policy — sits buried at rank 7, lost among topical lookalikes.
2The reranker takes each (query, chunk) pair and reads them together. Because it sees both at once, it catches that only the refund chunk actually answers "refund if I cancel early" — and scores it far higher.
3Re-sort by the new scores. The right chunk vaults from rank 7 to rank 1 — so when you keep only the top 3, it makes the cut. This is often the biggest single accuracy jump you can add.
Pound for pound the highest-ROI upgrade after hybrid search — drop-in, no index change, big precision gain. The costs: ~100–500ms of extra latency and per-call cost, and it can't rescue what the first stage never retrieved (so keep first-stage recall healthy). Managed options like Cohere Rerank or self-hosted bge-reranker make it a few lines of code.
Same goal, different route: instead of a purpose-built reranker, hand the shortlist to an LLM and let it refine the results directly — usually one of three moves:
Because the LLM genuinely understands the query, quality can edge out a cross-encoder — but each move is an extra LLM call, so it's slower and pricier. For plain ranking, cross-encoder rerankers stay the default; this approach earns its place when you can spare the latency.
Some questions aren't a fuzzy text lookup at all. When the answer is a number to compute, or the user wants only a specific slice of the data, similarity search is the wrong tool. This is where Text2SQL — letting an LLM turn the question into a database query — earns its keep, and it shows up in two very different modes.
"What was our total revenue in Q2?" needs arithmetic over records. Vector search can fetch a few rows that mention revenue, but it can't sum them. The fix is an intent router up front: a lightweight classifier reads the query, spots that it needs a calculation, and switches the whole system off the vector-search path and onto pure Text2SQL.
1A classifier reads the query and sees it needs a calculation. Rather than run vector search — which can't add numbers — it routes the query onto the Text2SQL path instead. The whole system switches.
2Feed the LLM both the user question and the table schema (with a short system prompt), and it writes the matching SQL. The database, not the vector index, does the retrieval.
3Run it for an exact, verifiable number — computed, not guessed, then phrased into an answer. This is exactly what vector search couldn't produce.
Exact and auditable, and shipped as a product by Snowflake, Databricks and others — but accuracy drops on messy real-world schemas, so production systems add schema hints and validate or repair the SQL before running it. Benchmarks like BIRD track the frontier.
The second mode doesn't replace vector search — it constrains it. "What did EU customers say about outages?" still needs semantic search over text, but only within a slice of the data. So a structured filter (region = 'EU') runs alongside the vector search: the filter narrows the candidates, and similarity ranks by meaning within that slice.
1The LLM splits the question into two parts: the meaning to search for ("outages"), and the hard filter the query implies (region = EU).
2The filter runs first and drops every non-EU chunk — US and APAC are gone before ranking even starts. Only the EU slice survives.
3Now vector search ranks within the EU slice by relevance to "outages". Filter for the constraint, similarity for the meaning — the two work together, not one instead of the other.
This is exactly what metadata filtering (self-query) does: an LLM peels the constraint out of the question ("invoices after 2025 over $500" → date > 2025-01-01 AND amount > 500) — or it comes straight from your app (the user's tenant, region, access rights) with no LLM at all. It's commoditized in every vector DB. It's the right default when the filter is simple — equality, ranges, a few tags. Dedicated vector DBs are built to keep filtering fast even at high scale and QPS.
For complex filters — joins, aggregations, or complex business logic filters, which is where Text2SQL comes in. Text2SQL asks the LLM to write a whole query instead of filling a few filter slots.
And the vector-DB-vs-relational line has blurred: Postgres with pgvector now does vector search too, combining a rich SQL WHERE with vector ranking in one query.
Rough rule of thumb: simple filter or high scale → metadata filtering on a vector DB; complex filter, or data already in SQL → Text2SQL over a store like pgvector.
Notice mode 1 already leaned on a router to pick the path. Deciding, per query, which tool to reach for — search, SQL, or nothing at all — is exactly what the last camp does at full strength.
Everything so far runs once, in a fixed order. But hard questions need more than one shot — decompose, search, notice a gap, search again. The modern move is to hand retrieval to an agent: give the LLM search, SQL, and web as tools, and let it loop — reason, act, observe, repeat — until it has enough to answer.
1The query goes to an agent holding a set of tools — search(), text2sql(), web(). It doesn't retrieve blindly: first it plans, splitting the question into two facts to chase down.
2For the first fact, the agent activates search() — the other tools stay idle — runs it, and reads the result. SLA met. One down.
3For the second fact it switches to text2sql() — churn lives in a database, not the docs. Same loop, different tool.
4Now the agent judges it has enough — no tool fires — so it stops looping and answers, grounded in what it gathered. Had a search come back empty, it would just loop again or fall back to web().
This is the dominant frontier pattern, and it quietly absorbs half of this blog: query decomposition, routing, re-searching on a bad result, web fallback — all just things the agent decides to do. The cost is the flip side of the flexibility: unbounded latency and token spend, harder to debug and evaluate, and a real risk of loops running away — so guard it with step caps. It's now native to essentially every agent framework.
search() tool the agent calls is still your Camp 1–4 stack underneath: hybrid search, a reranker, metadata filter etc. The agent decides when and what to search; those techniques decide how well each search lands. And cheapest wins still come from getting that stack right — reach for the agent only when questions are genuinely multi-step.Around 2023–2024 a wave of research papers tackled "adaptive" retrieval — teaching the system to decide when and whether to retrieve, and to critique its own results. Great ideas, and worth knowing by name. But here's the reality: their specific machinery (custom-trained tokens, bespoke classifier models) rarely ships. Instead, the agent loop from Camp 5 recreates each behavior with a plain prompt and a tool call. They live on as ideas, not as code you deploy.
Trains the model to emit "reflection tokens" that critique its own output segment by segment — is this relevant? is it supported?
// now: an LLM-judge grades in-loop, no special training
Classifies each query's difficulty and routes it to the cheapest strategy that'll work — skip, single search, or full multi-hop.
// now: routing, done by a prompted classifier
The pattern is the same in both: each is something an agent now decides on the fly. That's the honest verdict — study them for the ideas, but reach for a general agent loop to actually build it.
Retrieval isn't one algorithm — it's a toolbox. So instead of going camp by camp, here's every technique we covered ranked by how widely it's used today (★ = niche, ★★★★★ = nearly everyone runs it), each with a one-line take and the camp it comes from.
| Technique | Adoption | Camp | The gist — gain · cost · when |
|---|---|---|---|
| Hybrid search + RRF | ★★★★★ | 2 · search | The default baseline; fixes vector search's exact-match blind spot at the cost of a second index — run it always. |
| Reranking | ★★★★★ | 3 · refine | The biggest single precision jump, and drop-in; costs ~100–500ms and can't rescue bad recall. Use whenever the right chunk lands just outside the top few. |
| Stronger embeddings | ★★★★☆ | 2 · search | A better model lifts every query at once; the cost is re-embedding the whole corpus. Start here — fine-tune only as a last resort. |
| Query rewriting | ★★★★☆ | 1 · query | Cheap cleanup of messy, conversational queries; wasted on already-clean ones. Near-mandatory for chat. |
| Metadata filtering | ★★★★☆ | 4 · structured | Enforces hard constraints (dates, tiers, region) that similarity ignores; needs clean metadata. The default for simple filters. |
| Text2SQL | ★★★★☆ | 4 · structured | Exact, aggregatable answers a vector index can't compute; brittle on messy schemas. When the answer is numeric or the filter is complex. |
| Agentic RAG | ★★★★☆ | 5 · loop | Handles genuinely multi-step questions and absorbs the other tricks; unbounded cost/latency and harder to debug. Only when one shot won't do. |
| LLM refinement | ★★★☆☆ | 3 · refine | An LLM reranks, filters, or compresses the shortlist; flexible but slower and pricier than a cross-encoder. Handy when an LLM's already in the loop. |
| RAG-Fusion | ★★★☆☆ | 1 · query | Multi-query + RRF for phrasing robustness; N× the retrieval calls. Worth it when wording sensitivity is hurting recall. |
| HyDE | ★★☆☆☆ | 1 · query | Embeds a hypothetical answer to close the question–answer gap; an extra LLM call, and a wrong guess can mislead. Good on dense knowledge corpora. |
If I had to compress it to one path: get hybrid search + a reranker working first — that's most of the win, cheaply. Then, and only then, add a query trick, structured data, or an agent loop when you can point to the exact query it's failing on. These aren't rivals; the strongest systems stack several, each earning its place.
Thanks for reading, and hope you got something out of it.