A8gent
GlossaryAgent concepts

What is reranking?

Also called: reranker, cross-encoder reranking

Reranking is a second retrieval pass that takes the candidates returned by fast search and reorders them with a slower, more accurate model that reads the question and each passage together.

Written by A8gent
Contents
  1. At a glance
  2. What reranking is
  3. Where it goes in the pipeline
  4. Cost and latency
  5. What you should know
  6. FAQ
  7. Related terms

At a glance

Category
Agent concept - retrieval
Stage one
Fast approximate search optimised for recall - retrieve dozens of candidates
Stage two
Cross-encoder reranking optimised for precision - keep the top three to five
Why it is more accurate
It reads question and passage together instead of encoding them separately
Latency
Typically low hundreds of milliseconds for a few dozen candidates
Often pays for itself
Fewer, better passages means fewer input tokens sent to the model

What reranking is

Reranking is the second stage of a two-stage retrieval pipeline. The first stage - vector search, keyword search, or both - is fast and approximate, and its job is to get from millions of passages down to a few dozen plausible candidates. The reranker then scores each of those candidates against the query properly and reorders them, so the three passages you actually put in the prompt are the three best ones.

The technical reason it works is that the two stages use different model architectures. Embeddings are bi-encoders: the question and the passage are encoded separately and never meet, which is what makes pre-computing an index possible. A reranker is a cross-encoder: it reads the question and one passage together in a single pass, so it can judge whether this passage answers this question rather than whether the two are broadly about the same topic. That is far more accurate and far too slow to run over a whole corpus.

Where it goes in the pipeline

Retrieve generously, rerank, then select tightly. A typical shape is: retrieve fifty candidates by hybrid search, rerank them, keep the top three to five for the prompt. The first stage optimises recall - do not miss the right passage - and the second optimises precision - put the right passage first.

This is also what makes hybrid retrieval workable. Keyword and vector search return differently ranked lists with no common score, and merging them is fiddly. A reranker sidesteps the problem: throw both lists at it, let it score everything on one scale, and take the winners.

Cost and latency

Reranking adds one API call per query and a corresponding cost, priced per search or per document scored depending on the provider. It also adds latency, typically in the low hundreds of milliseconds for a few dozen candidates, which matters for anything conversational - see latency budget.

Against that, it usually lets you send fewer passages to the model, because the ones you send are better. That reduces input tokens and often pays for the reranker outright, while also improving answers - a rare case where the cheaper option is also the better one. Open-weight cross-encoders can be self-hosted if you would rather not send passages to another vendor.

What you should know

Reranking is the highest-return single addition to a mediocre RAG system, and it is usually a few lines of code. If your retrieval returns the right passage somewhere in the top twenty but not in the top three, a reranker fixes that directly. If the right passage is not in the top fifty at all, reranking cannot help you - that is a chunking or indexing problem, and you should fix it first.

Measure both stages separately. Track recall at fifty for stage one and precision at three after reranking. Teams that track only end-to-end answer quality spend weeks unable to tell which half of the pipeline is failing.

Reranking FAQ

What is a reranker in RAG?

A second-pass model that takes the candidates from fast search and scores each one against the query directly, then reorders them. You keep only the top few for the prompt. It is more accurate than vector search because it reads the question and the passage together.

Why is reranking more accurate than vector search?

Embeddings encode question and passage separately and compare the results, which is what allows a pre-built index. A reranker is a cross-encoder that processes both together in one pass, so it can judge whether this specific passage answers this specific question rather than topical similarity.

How many candidates should I rerank?

Enough that the right passage is almost certainly among them - a few dozen is common - then keep three to five after reranking. Retrieve for recall, rerank for precision. Reranking two candidates achieves nothing; reranking five hundred is slow and rarely better than fifty.

Does reranking cost much?

One extra API call per query, priced per search or per document scored. It often nets out cheaper, because better ordering lets you send three passages to the model instead of ten, cutting input tokens on the far more expensive generation call.

Will reranking fix my bad RAG results?

If the right passage is somewhere in your top twenty but not the top three, yes, and quickly. If it is not in the top fifty at all, no - that is a chunking, extraction or indexing failure and reranking cannot recover a passage that was never retrieved.

Can I self-host a reranker?

Yes. Open-weight cross-encoder models are widely available and run on modest hardware for the volumes most businesses see. That is the route to take if sending customer document passages to a third-party API conflicts with your data residency or DPA commitments.

Primary source

Cohere - Rerank overview

Checked August 2026. Anything priced or rate-limited moves, so verify against the source before you budget.

Want this built rather than explained?

Knowing what reranking means is one thing; having it running in your business is another. We scope one workflow at a time, quote a fixed price in writing after a free consult, build it, and hand it over with documentation you own.