The Reranking article covered the general two-stage strategy. This article covers the specific model that actually does the more careful, second-stage evaluation: a reranker.
The simple definition
A reranker is a model specifically trained to score how relevant a chunk is to a query, taking both the query and the chunk together as input, rather than comparing pre-computed embeddings the way initial retrieval does. Recall from the Retriever article’s description of dense retrieval comparing a query embedding against pre-computed chunk embeddings, calculated separately.
A reranker works differently — it processes the query and a specific candidate chunk together, in a single pass, letting it directly model how the two relate to each other rather than comparing two independently-computed vectors.
Why processing the query and chunk together produces a more accurate score
This is the real, technical reason rerankers achieve better accuracy than initial retrieval, and it’s worth being precise about it. Recall from the Attention article’s core mechanism — letting every token attend to every other token.
A reranker, commonly built using an architecture called a cross-encoder, feeds the query and chunk into a Transformer together, letting the model’s attention mechanism directly compare specific words and phrases between the two, capturing subtle relationships that comparing two separately-computed embeddings can miss entirely.
This is a genuinely more thorough evaluation, but it can only be done one query-chunk pair at a time, which is exactly why it’s too slow for searching millions of chunks directly, as covered in the Reranking article.
flowchart LR
A[Query] --> C[Cross-Encoder: processes both together]
B[Candidate Chunk] --> C
C --> D[Single relevance score for this specific pair]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of the difference between comparing two people’s resumes side by side from a distance (initial retrieval, comparing pre-computed summaries) versus sitting both people down together for a direct, detailed conversation about how their specific experience relates to a specific job (a reranker, directly comparing the query and chunk together).
The direct conversation reveals a more nuanced, accurate match, but it takes real time and can’t be done with thousands of candidates simultaneously.
Where this breaks down: A hiring conversation involves genuine human judgment developing in real time. A cross-encoder reranker is a fixed, trained neural network — the same Transformer architecture covered throughout the Transformers phase — calculating a single relevance score through a forward pass, not an evolving conversation, just a more thorough, single-pass calculation than comparing two independently-computed vectors allows.
Real, published rerankers used in production today
This is worth grounding concretely, since real, named products exist specifically for this purpose.
Cohere offers a dedicated Rerank product, a real, commercially available API specifically built to take a query and a list of candidate documents and return them re-scored by relevance — widely referenced in published RAG system architectures and directly integrated into cloud platforms like AWS.
The underlying cross-encoder approach itself traces back to published research by Nogueira and Cho (2019), demonstrating that this direct, joint-processing approach to relevance scoring meaningfully outperformed the separate-embedding-comparison approach used in initial retrieval.
A concrete example, layered
For a simple beginner example: given the query “best exercises for lower back pain” and 20 initially retrieved chunks, a reranker processes the query alongside each individual chunk, one at a time, and might correctly promote a chunk specifically discussing “lumbar strengthening exercises” above a more generically similar chunk about “general fitness tips,” a distinction that comparing pre-computed embeddings alone might have scored more closely together.
For a production example: companies building serious RAG products commonly add Cohere’s Rerank API, or a similar dedicated reranking model, as a distinct second stage after initial retrieval from a vector database like Pinecone or Weaviate — a real, standard architectural pattern across the industry, not a rare or experimental addition.
Why rerankers have a genuine, real limitation of their own
It’s worth being honest about a real, practical constraint here, not presenting rerankers as a strictly superior replacement for initial retrieval.
Because a reranker must process the query and each candidate together, one pair at a time, it fundamentally cannot scale to searching an entire knowledge base directly — recall from the Reranking article that this is precisely why rerankers only ever operate on an already-narrowed shortlist, never as a replacement for fast initial retrieval across the full knowledge base.
A cross-encoder-style example
Query: “Can contractors access parental leave?”
Candidate 1: “Parental leave is available to permanent employees...”
Candidate 2: “Contractors receive unpaid personal leave under...”A reranker reads the query together with each candidate and can notice that Candidate 2 directly addresses contractors, even if Candidate 1 contains more repeated query words.
The retriever and reranker therefore have different jobs. The retriever quickly finds a broad candidate pool, while the reranker spends more computation estimating fine-grained relevance within that small pool.
A real result from Google’s reranker documentation
Google’s published example asks “Why is the sky blue?” The science passage receives a relevance score of 0.98, while a poem about a blue sky receives 0.64. Both contain similar words, but the science passage actually answers the question.
The current Ranking API supports up to 1,024 tokens per record with model version 004, while older models up to version 003 support 512. Longer content is truncated, which is why chunk size still matters before reranking. Source: Google’s Ranking API guide.
Common misconception
A frequent beginner assumption: that a reranker is just a bigger, more powerful embedding model, doing fundamentally the same kind of comparison as initial retrieval, just with more parameters.
As this article has explained, the real difference is architectural, not just scale — a reranker (typically a cross-encoder) processes the query and chunk jointly in one pass, while an embedding model used for retrieval processes them entirely separately, producing independent vectors compared afterward; this structural difference, not raw model size, is what gives rerankers their genuine accuracy advantage.
Where this fits in what comes next
You now understand the specific model that performs reranking, and why its joint-processing approach is both more accurate and less scalable than initial retrieval. The next article, Metadata Filtering, covers a different, complementary way to narrow down search results — using structured attributes rather than relevance scoring at all.
In one sentence
A reranker is typically a cross-encoder model that processes a query and a candidate chunk jointly, letting it capture subtler, more accurate relevance signals than comparing pre-computed embeddings alone allows — a genuinely more thorough evaluation, real and commercially available through products like Cohere’s Rerank, but one that only scales to the small, already-narrowed shortlist reranking operates on.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed