Two-Stage Retrieval & Reranking

Begin with the problem

Fast retrieval finds candidates; a reranker spends more effort ordering the best few. This two-stage design concentrates expensive judgment where it matters.

query → sparse + dense retrieval → merge → rerank

What you will learn

Explain Two-Stage Retrieval & Reranking in simple language before using its technical details.
Follow the mechanism step by step through a small RAG example.
Connect this topic to the modules before and after it.
Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: Pinecone’s search documentation documents dense, sparse, hybrid, metadata-filtered, and reranked retrieval patterns used in production search systems.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Module 17 closed with a robust hybrid retrieval system. This module closes Level 4 by asking a really important follow-up question: is the first retrieval stage’s ranking actually the best possible ranking, or can a second, more careful pass really improve it? The answer — reranking — is standard practice in real production RAG systems.

2. The Problem — Initial Retrieval Is Necessarily a Compromise

Recall Module 13’s core trade-off: fast retrieval (whether dense, sparse, or hybrid) really sacrifices some precision for speed. A top-20 result set from initial retrieval likely contains a real mix:

1. Really, strongly relevant chunks
2. Somewhat relevant chunks
3. Chunks that scored well by coincidence, but are really NOT
   what the user actually needs

The initial retrieval’s ranking, while fast, isn’t necessarily the most accurate possible ranking — it was optimized for speed across potentially millions of candidates, not maximum precision on a small, already-narrowed set.

3. The Two-Stage Solution

Query
   ↓
STAGE 1 -- Fast RETRIEVAL (hybrid search, Module 17): narrow millions
          of chunks down to a top-20 candidate set, QUICKLY
   ↓
Top 20 candidates
   ↓
STAGE 2 -- Slower, more ACCURATE RERANKING: carefully re-score just
          these 20 candidates, using a more thorough (but more
          computationally expensive) method
   ↓
Top 5, re-ranked with really higher precision
   ↓
LLM

This is the core mental model this module builds toward: RECALL FIRST, PRECISION SECOND. Stage 1’s job is to make sure the really relevant chunks are SOMEWHERE in the candidate set — even if not perfectly ordered. Stage 2’s job is to get the ORDER right, working with a small enough set that a more expensive, careful method becomes really affordable.

4. Why Not Just Use the Reranker on Everything?

If the reranker is MORE accurate, why not use it on ALL millions of
chunks directly, skipping Stage 1 entirely?

Because rerankers are really MORE COMPUTATIONALLY EXPENSIVE per
comparison than the embedding-based similarity search from Modules
10-14 -- running it against MILLIONS of chunks for every single
query would be far too SLOW (exactly Module 13's speed concern,
revisited).

The two-stage design exists precisely to get the best of both worlds: Stage 1’s speed narrows the field to a manageable size; Stage 2’s accuracy then operates on that small, already-relevant-ish set, where its higher cost is really affordable.

5. Cross-Encoders — Why Rerankers Are Really More Accurate

This is worth understanding mechanically, not just accepting as a given:

EMBEDDING retrieval (Modules 10-14):

Query -> embed SEPARATELY -> query vector
Document -> embed SEPARATELY -> document vector
Compare the two vectors (Module 11)

The query and document NEVER actually interact with each other
DURING embedding -- each is embedded in isolation, then compared
afterward.


CROSS-ENCODER reranking:

Query + Document -> fed TOGETHER into ONE model -> a single
                    relevance score

The model can really examine how SPECIFIC WORDS in the query
relate to SPECIFIC WORDS in the document, DIRECTLY, since they're
processed together -- not compressed into two separate, isolated
vectors beforehand.

Why this really produces higher accuracy: a cross-encoder can capture subtle, direct interactions between query and document text that get lost when each is compressed into a single vector in isolation. This is also exactly why cross-encoders are more expensive — they require a full model pass for every single query-document PAIR, rather than reusing pre-computed document embeddings (Module 10) across many different queries.

6. A Real Developer Example

TechCorp's hybrid retrieval (Module 17) returns 20 candidate chunks
for the query "What's TechCorp's London hotel reimbursement
exception?"

Among the top 20, TWO chunks are both plausible:

Chunk A: "London and Tokyo have a raised hotel limit of $250 per
         night." (really THE answer)

Chunk B: "Singapore also has various travel-related exceptions,
         though hotel limits follow the standard $200/night rate."
         (mentions "exceptions" and "hotel," superficially similar,
         but really NOT the answer to THIS question)

Initial hybrid retrieval might rank these CLOSE together, or even
B ABOVE A, since both share real surface-level similarity to the
query.

A CROSS-ENCODER reranker, examining the FULL query and EACH chunk
TOGETHER, can recognize that Chunk A directly answers "London...
exception" while Chunk B really does NOT -- correctly promoting A
to the top, even if initial retrieval had them closer together or
reversed.

7. A Simple Agentic AI Connection

An agent performing a knowledge-base search benefits directly from reranking when the initial candidate set is really ambiguous — an agent that retrieves 20 candidates and then reranks before deciding what to actually cite in its final answer produces more reliably accurate, well-grounded responses than one that trusts the initial retrieval ranking blindly.

8. How Is This Used in AI?

🤖 How Is This Used in AI?

Two-stage retrieval with reranking is standard practice in production RAG systems handling really large knowledge bases — fast initial retrieval (dense, sparse, or hybrid) followed by a cross-encoder reranking pass on a small candidate set is a well- established pattern precisely because it combines the speed necessary for real-time applications with the precision needed for reliable, accurate final results.

9. Real-World Applications

Any production RAG system operating on a really large or ambiguous knowledge base
Search engines generally (web search, e-commerce search) use this exact recall-then-precision pattern
Question-answering systems where getting the SINGLE most relevant passage really matters

10. Common Mistakes

Incorrect idea: Running a cross-encoder reranker against an entire large knowledge base directly.

Why it is incorrect: As shown directly in Section 4, this is really too slow — rerankers exist specifically for small, pre-narrowed candidate sets.

Incorrect idea: Skipping reranking entirely and trusting initial retrieval’s ranking as final.

Why it is incorrect: As shown directly in Section 6, initial retrieval can really misorder subtly different but superficially similar candidates.

Incorrect idea: Not narrowing the candidate set enough before reranking.

Why it is incorrect: Reranking a really large candidate set (like 500 chunks) defeats the purpose of the two-stage design’s efficiency — Stage 1 should narrow meaningfully before Stage 2 begins.

11. Limitations

Reranking adds real, real latency (Module 25 of the Generative AI course) and cost to the overall retrieval pipeline — a real trade-off against the accuracy gain
A reranker can only improve the ORDER of candidates already present in the initial retrieval set — if Stage 1 really missed a relevant chunk entirely, reranking cannot recover it (this directly connects back to Module 2’s principle: “a good LLM can’t compensate for bad retrieval” — the same applies to reranking and Stage 1)

12. Quick Reference — The Whole Idea in One Diagram

Millions of chunks
   ↓
STAGE 1: fast retrieval (dense/sparse/hybrid, Modules 10-17) --
        RECALL first
   ↓
Top-20 candidates
   ↓
STAGE 2: cross-encoder RERANKING -- PRECISION second, query + doc
        processed TOGETHER for higher accuracy
   ↓
Top-5, re-ordered by real relevance
   ↓
LLM

13. Code — Implementing a Simplified Cross-Encoder Reranking Stage

🎯 Target of this example: implement Section 5-6’s real developer example directly — demonstrating a case where embedding-based initial retrieval ranks two candidates closely (or incorrectly), and a simplified cross-encoder-style reranker correctly reorders them by directly examining query-document interaction.

Example 1 — Simple

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def simplified_cross_encoder_score(query: str, document: str) -> float:
    """A SIMPLIFIED, illustrative stand-in for a real cross-encoder
    -- scores based on DIRECT word overlap with the FULL query
    examined against the FULL document TOGETHER, rather than two
    separately-embedded vectors compared afterward (Section 5). A
    real cross-encoder uses a trained neural model; this demonstrates
    the ALGORITHM's core idea: joint examination, not isolated
    embedding."""
    query_words = set(query.lower().split())
    doc_words = set(document.lower().split())
    overlap = len(query_words & doc_words)
    # Reward EXACT phrase containment even more strongly -- a cross-
    # encoder can recognize this kind of direct, specific match.
    phrase_bonus = 2.0 if query.lower() in document.lower() else 0.0
    return overlap + phrase_bonus

query = "London hotel reimbursement exception"
chunk_a = "London and Tokyo have a raised hotel limit of $250 per night."
chunk_b = "Singapore also has various travel exceptions, though hotel limits follow the standard $200/night rate."

score_a = simplified_cross_encoder_score(query, chunk_a)
score_b = simplified_cross_encoder_score(query, chunk_b)

print(f"Chunk A score: {score_a} -- {chunk_a}")
print(f"Chunk B score: {score_b} -- {chunk_b}")

Expected Output:

Chunk A score: 2.0 -- London and Tokyo have a raised hotel limit of
$250 per night.
Chunk B score: 1.0 -- Singapore also has various travel exceptions,
though hotel limits follow the standard $200/night rate.

What we conclude from this example: even this simplified, word- overlap-based scorer correctly favors Chunk A (score 2.0, matching “London” and “hotel”) over Chunk B (score 1.0, matching only “hotel” — “exceptions” doesn’t exactly match the query’s “exception”). This illustrates the core idea a real cross-encoder implements at far greater sophistication: examining the query and document TOGETHER, directly, rather than relying purely on pre-computed embedding similarity. Example 2 shows a case where embedding-based Stage 1 retrieval really gets the order wrong, and this kind of direct term-matching signal correctly fixes it.

Example 2 — Intermediate

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def initial_retrieval_rank(query_embedding, chunks_with_embeddings):
    """STAGE 1 -- fast, embedding-based initial retrieval (Modules
    10-14), which may rank superficially similar chunks closely
    together or even incorrectly."""
    scored = [(name, cosine_similarity(query_embedding, emb))
              for name, emb in chunks_with_embeddings.items()]
    scored.sort(key=lambda x: x[1], reverse=True)
    return scored

def rerank_with_relevance_signal(query: str, ranked_chunks: list, chunk_texts: dict) -> list:
    """STAGE 2 -- reranking based on a MORE CAREFUL relevance signal
    that directly examines specific query terms against document
    content -- illustrating what a real cross-encoder does at a
    conceptual level, without requiring an actual trained model."""
    query_terms = set(query.lower().replace("?", "").split())

    def relevance_signal(chunk_name):
        text = chunk_texts[chunk_name].lower()
        # A real cross-encoder learns this kind of relevance signal
        # from training data; here we approximate it with a
        # deliberately MORE DISCRIMINATING term-match count.
        matches = sum(1 for term in query_terms if term in text)
        return matches

    reranked = sorted(ranked_chunks, key=lambda x: relevance_signal(x[0]), reverse=True)
    return reranked

chunk_texts = {
    "chunk_a": "London and Tokyo have a raised hotel limit of $250 per night due to higher costs.",
    "chunk_b": "Singapore also has various travel exceptions, though hotel limits follow standard rates.",
}
chunk_embeddings = {
    "chunk_a": np.array([0.72, 0.60, 0.30]),
    "chunk_b": np.array([0.71, 0.61, 0.29]),  # deliberately VERY close in embedding space
}

query = "What is the London hotel reimbursement exception?"
query_embedding = np.array([0.70, 0.62, 0.28])

stage1_ranking = initial_retrieval_rank(query_embedding, chunk_embeddings)
print("Stage 1 (initial retrieval) ranking:")
for name, score in stage1_ranking:
    print(f"  [{score:.4f}] {name}")

stage2_ranking = rerank_with_relevance_signal(query, stage1_ranking, chunk_texts)
print("\nStage 2 (reranked) ranking:")
for name, score in stage2_ranking:
    print(f"  {name}")

Expected Output:

Stage 1 (initial retrieval) ranking:
  [0.9999] chunk_b
  [0.9994] chunk_a

Stage 2 (reranked) ranking:
  chunk_a
  chunk_b

What we conclude from this example: Stage 1’s embedding-based retrieval ranks chunk_b marginally ABOVE chunk_a — an incorrect order, since chunk_a is the really relevant one (it directly mentions “London” and “hotel,” matching the query). Stage 2’s reranking correctly SWAPS the order, promoting chunk_a to the top by directly counting real query-term matches. This is exactly Section 6’s real developer example, made concrete: reranking can correct subtle ordering mistakes initial retrieval makes.

Example 3 — Production Grade

import numpy as np
from dataclasses import dataclass

@dataclass
class RerankedResult:
    chunk_id: str
    stage1_rank: int
    stage2_rank: int
    rank_changed: bool

class TwoStageRetriever:
    """A production-style two-stage retriever making Section 3's
    recall-first-precision-second pipeline STRUCTURAL -- Stage 1
    ALWAYS runs before Stage 2, and the class tracks whether
    reranking actually CHANGED the order, directly useful for
    monitoring how much value reranking is really adding."""

    def __init__(self, chunk_embeddings: dict, chunk_texts: dict):
        self.chunk_embeddings = chunk_embeddings
        self.chunk_texts = chunk_texts

    def _stage1_retrieve(self, query_embedding: np.ndarray, top_k: int = 10) -> list:
        def cosine_similarity(a, b):
            return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
        scored = [(name, cosine_similarity(query_embedding, emb))
                  for name, emb in self.chunk_embeddings.items()]
        scored.sort(key=lambda x: x[1], reverse=True)
        return [name for name, score in scored[:top_k]]

    def _stage2_rerank(self, query: str, candidates: list) -> list:
        query_terms = set(query.lower().replace("?", "").split())

        def relevance_signal(chunk_name):
            text = self.chunk_texts[chunk_name].lower()
            return sum(1 for term in query_terms if term in text)

        return sorted(candidates, key=relevance_signal, reverse=True)

    def search(self, query: str, query_embedding: np.ndarray, top_k: int = 10) -> list:
        stage1_results = self._stage1_retrieve(query_embedding, top_k)
        stage2_results = self._stage2_rerank(query, stage1_results)

        results = []
        for new_rank, chunk_id in enumerate(stage2_results, start=1):
            old_rank = stage1_results.index(chunk_id) + 1
            results.append(RerankedResult(
                chunk_id=chunk_id, stage1_rank=old_rank, stage2_rank=new_rank,
                rank_changed=old_rank != new_rank,
            ))
        return results

chunk_texts = {
    "chunk_a": "London and Tokyo have a raised hotel limit of $250 per night due to higher costs.",
    "chunk_b": "Singapore also has various travel exceptions, though hotel limits follow standard rates.",
    "chunk_c": "General company holiday schedule for the upcoming year.",
}
chunk_embeddings = {
    "chunk_a": np.array([0.72, 0.60, 0.30]),
    "chunk_b": np.array([0.71, 0.61, 0.29]),
    "chunk_c": np.array([0.1, 0.1, 0.9]),
}

retriever = TwoStageRetriever(chunk_embeddings, chunk_texts)
query = "What is the London hotel reimbursement exception?"
query_embedding = np.array([0.70, 0.62, 0.28])

results = retriever.search(query, query_embedding, top_k=3)
for r in results:
    changed_flag = " <- REORDERED" if r.rank_changed else ""
    print(f"{r.chunk_id}: stage1_rank={r.stage1_rank}, stage2_rank={r.stage2_rank}{changed_flag}")

Expected Output:

chunk_a: stage1_rank=2, stage2_rank=1 <- REORDERED
chunk_b: stage1_rank=1, stage2_rank=2 <- REORDERED
chunk_c: stage1_rank=3, stage2_rank=3

What we conclude from this example: the rank_changed flag immediately and clearly shows that reranking really altered the top-2 order (chunk_a and chunk_b swapped), while chunk_c — clearly irrelevant to both systems — stayed correctly in last place regardless of stage. This kind of explicit, auditable tracking is exactly what a real team would want to monitor how much real value their reranking stage is adding over initial retrieval alone, directly connecting to Module 32’s evaluation practices.

14. Interview Questions

Q: Explain the “recall first, precision second” mental model for two-stage retrieval.

Ans: Stage 1 (fast retrieval — dense, sparse, or hybrid) is responsible for recall: making sure the really relevant chunks are SOMEWHERE within a reasonably-sized candidate set, even if not perfectly ordered. Stage 2 (reranking) is responsible for precision: carefully re-scoring just that small candidate set with a more accurate but more expensive method, to get the final order really right. This two- stage split gets the benefits of both speed (from Stage 1, operating across a potentially huge dataset) and accuracy (from Stage 2, operating on a much smaller, already-narrowed set).

Q: Why are cross-encoder rerankers generally more accurate than embedding-based retrieval, and why does that same property make them more computationally expensive?

Ans: Embedding-based retrieval embeds the query and each document separately, in isolation, then compares the resulting vectors afterward — the query and document never directly interact during embedding. A cross-encoder feeds the query and document together into one model, letting it directly examine how specific words in the query relate to specific words in the document.

This joint examination captures subtler relevance signals, but it also means a cross-encoder needs a full model pass for every single query-document pair, rather than reusing pre-computed document embeddings across many different queries — which is exactly why it’s too slow to run against an entire large knowledge base directly.

Q: If Stage 1 retrieval completely fails to include a really relevant chunk in its top-20 candidates, can Stage 2 reranking recover that chunk? Explain.

Ans: No — reranking can only reorder candidates that Stage 1 already retrieved. If a really relevant chunk was never included in the initial candidate set at all, reranking has no opportunity to promote it, since it was never among the candidates being reranked in the first place. This directly echoes the broader principle from earlier in this course: a good downstream step (whether an LLM’s generation or a reranker’s precision) can’t compensate for information that was never retrieved in the first place — Stage 1’s recall really bounds what Stage 2 can ever recover.

Q: Design a way to measure how much value a reranking stage is actually adding to a production RAG system.

Ans: I’d track, for every search, whether reranking actually changed the relative order of the top results compared to Stage 1’s initial ranking — explicitly recording each result’s rank before and after reranking, and flagging when they differ. Aggregating this over many real queries would show how often reranking meaningfully alters results versus simply confirming Stage 1’s ranking was already correct, which directly informs whether the added latency and cost of running a reranking stage is really paying off for this specific application, or whether Stage 1 alone is already sufficient.

15. What You Should Remember

Two-stage retrieval follows recall first (Stage 1), precision second (Stage 2) — fast, broad retrieval narrows the field; slower, more accurate reranking gets the final order right.
Cross-encoders examine query and document together, capturing interactions embedding-based retrieval misses — but at really higher per-comparison cost, which is exactly why they’re only used on Stage 1’s small, pre-narrowed candidate set.
Reranking can only reorder what Stage 1 already retrieved — verified directly through a production-style retriever that tracks and reveals exactly which results reranking actually changed.

16. Quick Practice

Design a two-stage retrieval configuration (roughly how many candidates Stage 1 should retrieve, and how many Stage 2 should finally return) for a legal research tool where missing a really relevant precedent would be a serious problem — justify your specific numbers using this module’s recall-first-precision-second principle.

17. Next Step

Next: Module 19 — Query Transformation — Level 5 begins here: user questions aren’t always ideal search queries, and this module covers rewriting, expanding, and decomposing them before retrieval even happens.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026