Re-ranking: The Retrieval-Quality Layer Most RAG Apps Skip

Most RAG systems retrieve with a vector search and stop there: embed the question, find the nearest chunks, stuff the top five into the prompt. It works well enough to demo, and then quietly disappoints in production — the model keeps answering from chunks that are merely related to the question instead of the ones that actually contain the answer.

The fix is re-ranking: a second, smarter pass that re-orders your retrieved chunks by true relevance before any of them reach the model. It is the highest return-on-effort change you can make to a RAG pipeline, and it is strange how often it gets skipped.

Why vector search is not enough

Vector search uses a bi-encoder: it embeds the query and every document separately, ahead of time, then compares them by distance. That is what makes it fast and scalable — but also what makes it coarse. The query and the document never actually look at each other; you are comparing two summaries made in isolation.

• A bi-encoder (your embedding model) is great at recall: it reliably pulls the right chunk into the top 50. It is mediocre at precision: it often does not put it in the top 5.
• A cross-encoder reads the query and one document together and outputs a single relevance score, so it can judge subtle fit — but it cannot be precomputed and is far slower.
• Re-ranking combines them: cheap recall first, expensive precision second, on only a handful of candidates.
• The win is ordering. Same chunks, better order — and the model reads the best evidence first instead of burying it at position nine.

The retrieve-then-rerank pattern

The whole technique is two stages. First, retrieve broadly and cheaply — pull the top 50 or 100 candidates with vector search, or better, a hybrid of vector plus keyword search. Then re-rank narrowly and expensively — score each candidate against the query with a cross-encoder and keep only the best 3 to 5.

Because the slow cross-encoder only ever sees a few dozen candidates instead of your whole corpus, you get most of its accuracy at a tiny fraction of the cost. You are spending compute exactly where it changes the answer: the final shortlist.

What it looks like in code

Whether you use a hosted re-ranker or a local cross-encoder, the shape is identical — over-retrieve, then trim:

# Stage 1: cheap recall. Stage 2: precise re-ranking on the shortlist.
from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

def retrieve(query, k=5):
    candidates = vector_search(query, top_n=50)          # broad, fast, coarse
    pairs = [(query, c.text) for c in candidates]
    scores = reranker.predict(pairs)                     # query + doc seen together
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:k]]                    # keep only the best k

That is the entire change. One model, one sort, applied to 50 candidates — and the chunks your LLM reads go from roughly right to actually relevant.

Embeddings decide which documents are in the room; the re-ranker decides which ones get to speak. Skipping it means your best evidence is often in the room but never heard.

Costs, latency, and when to use it

• It adds latency: a cross-encoder pass over 50 candidates is real work, usually tens to a couple hundred milliseconds depending on model and hardware.
• It is cheap to adopt: hosted re-rankers (Cohere, Jina, Voyage) are a single API call; strong open ones (bge-reranker, mxbai) run on a modest GPU or even CPU for small batches.
• Tune the funnel: retrieving 50 and keeping 5 is a sane default — widen stage one if recall is the problem, narrow stage two if the prompt is too long.
• Pair it with hybrid search. Vector plus keyword recall feeds the re-ranker better candidates than either alone, especially for names, codes, and exact terms.
• Skip it when latency is sacred and your corpus is tiny — but measure first; the quality jump usually justifies the milliseconds.

The bottom line

If your RAG app retrieves but does not re-rank, you are leaving the easiest quality win unclaimed. Over-retrieve with your existing vector search, drop a cross-encoder in front of the prompt, and keep the top few. It is a few lines of code and one extra model, and it consistently does more for answer quality than swapping embedding models or fancier chunking.

Good retrieval is two jobs, not one: find the candidates, then order them. Vector search is built for the first and bad at the second. Give the second job to a re-ranker, and your model finally reads the right thing first.