TechByteByByte

Context Construction & Lost-in-the-Middle

Retrieval gives you chunks, but you can't blindly hand them all to the LLM — organizing, deduplicating, and ordering retrieved context effectively, and why more context isn't always better.

#RAG#AI#Context Construction#Level 5

Begin with the problem

Retrieving good passages is not enough if the prompt buries or truncates them. Context construction decides what the model actually gets to read and in what order.

user question → transform/retrieve → construct context → grounded answer + citations

What you will learn

  • Explain Context Construction & Lost-in-the-Middle in simple language before using its technical details.
  • Follow the mechanism step by step through a small RAG example.
  • Connect this topic to the modules before and after it.
  • Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: Google documents returned grounding information and citations in Gemini File Search. A citation exposes a source; your application still must verify that the source supports the claim.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Modules 15-20 covered finding and ranking the right chunks. This module addresses a really important next step that’s easy to skip past: retrieval gives you a list of chunks, but you can’t just paste that list straight into a prompt and expect optimal results. Context construction is the deliberate work between retrieval and generation.

2. The Problem — Raw Retrieved Chunks Aren’t Prompt-Ready

Retrieved chunks, straight from search (Modules 15-18):

- Chunk 7 (score 0.95): "London hotel limit is $250/night."
- Chunk 12 (score 0.88): "Standard hotel limit is $200/night."
- Chunk 7 AGAIN (score 0.95): "London hotel limit is $250/night."
  (duplicate -- maybe retrieved twice via hybrid search's TWO
  underlying methods, Module 17)
- Chunk 31 (score 0.31): "Office parking closes at 10pm."
  (really low relevance, barely cleared the top-k cutoff)

Handing this raw list directly to the LLM includes duplication and really marginal content — context construction exists to clean this up deliberately before generation ever happens.

3. What Context Construction Actually Does

Retrieved Chunks
   ↓
FILTER (drop really low-relevance results, even if they cleared
       the raw top-k cutoff)
   ↓
DEDUPLICATE (remove exact or near-exact repeats)
   ↓
ORGANIZE (with metadata, Module 9 -- source, section -- for eventual
         citation, Module 23)
   ↓
ORDER (decide the SEQUENCE chunks appear in the final prompt --
      Section 4 explains why this really matters)
   ↓
Final, clean Context

4. Lost in the Middle — Why Ordering Really Matters

This is a really important, somewhat counter-intuitive finding worth understanding directly:

Research on how LLMs use long context has found that models don't
use information EQUALLY WELL regardless of WHERE it sits in the
context window.

Information placed at the VERY BEGINNING or VERY END of a long
context tends to be used MORE reliably than information buried
somewhere in the MIDDLE.

This is precisely why “just retrieve more chunks and stuff them all in” isn’t automatically better — even really relevant information can be effectively “lost” if it lands in the middle of a long, undifferentiated context block. Context construction should deliberately consider ORDER, not just WHICH chunks to include.

Practical implication:      place your MOST relevant chunk (Module
                           18's reranked top result) at the
                           BEGINNING or END of the context, not
                           buried in the middle of several
                           less-relevant chunks

5. Why More Context Isn’t Always Better

Retrieved context should be:

- RELEVANT (really related to the actual question)
- CONCISE (not diluted with marginal, low-value content)
- WELL ORGANIZED (deduplicated, properly ordered)
- PROPERLY ORDERED (Section 4's lost-in-the-middle consideration)

This directly connects back to Module 15’s top-k trade-off: retrieving MORE chunks doesn’t guarantee a BETTER answer — it can really dilute the prompt and increase the risk that the model’s attention gets spread too thin across too much marginal content.

6. A Real Developer Example

TechCorp's HR assistant retrieves 8 chunks for a question about
London hotel reimbursement (Module 18's reranked, top-8 result).

WITHOUT context construction:      all 8 chunks pasted in RAW
                                  retrieval order, INCLUDING one
                                  near-duplicate and two really
                                  marginal, low-relevance chunks --
                                  the SINGLE most relevant chunk
                                  happens to land in POSITION 5,
                                  right in the really riskier
                                  "middle" of the context

WITH context construction:            duplicates removed, marginal
                                    chunks filtered out entirely,
                                    remaining chunks REORDERED so
                                    the MOST relevant chunk appears
                                    FIRST -- directly applying
                                    Section 4's lost-in-the-middle
                                    finding

7. A Simple Agentic AI Connection

An agent accumulating context across multiple tool calls (search results, prior reasoning, intermediate findings) within one task faces this exact context construction challenge at a larger scale — careful organization and ordering of accumulated context really matters more as an agent’s task grows longer and its context window fills with more and more accumulated information.

8. How Is This Used in AI?

🤖 How Is This Used in AI?

Every production RAG system implements deliberate context construction between retrieval and generation — deduplication, relevance filtering, and thoughtful ordering are standard, really necessary steps, precisely because raw retrieval output isn’t automatically optimized for how an LLM actually uses long context.

9. Real-World Applications

  • Any RAG system combining results from multiple retrieval methods (Module 17’s hybrid search, Module 20’s multi-query) that can produce real duplicates
  • Long-context RAG applications where lost-in-the-middle effects really matter
  • Systems needing to balance context completeness against real token cost (Module 27 of the Generative AI course)

10. Common Mistakes

Incorrect idea: Pasting raw retrieval results directly into a prompt without any processing.

Why it is incorrect: As shown directly in Section 2, this really includes duplication and marginal content.

Incorrect idea: Ignoring chunk ORDER within the final context.

Why it is incorrect: As shown directly in Section 4, this can cause really relevant information to be effectively underused, purely due to its position.

Incorrect idea: Assuming more retrieved chunks always means a better answer.

Why it is incorrect: As shown directly in Section 5, this directly contradicts the lost-in-the-middle finding and can really dilute prompt quality.

11. Limitations

  • The exact severity of lost-in-the-middle effects really varies by model and context length — worth verifying empirically (Module 32) for your specific model and use case, rather than assuming a fixed rule
  • Context construction adds real processing steps between retrieval and generation — a real, worthwhile trade-off against simply passing raw results through

12. Quick Reference — The Whole Idea in One Diagram

Retrieved chunks
   ↓
FILTER (drop marginal, low-relevance results)
   ↓
DEDUPLICATE (remove exact/near-exact repeats)
   ↓
ORDER (most relevant chunks at BEGINNING or END, not buried in the
       middle -- Section 4's lost-in-the-middle finding)
   ↓
Clean, well-organized context -> ready for generation (Module 22)

13. Code — Implementing Deliberate Context Construction

🎯 Target of this example: implement Section 6’s real developer example directly — filtering marginal results, deduplicating, and reordering so the most relevant chunk lands at the beginning of the final context, exactly applying Section 4’s lost-in-the-middle principle.

Example 1 — Simple

def construct_context(retrieved_chunks: list, dedupe=True, max_chunks=5, min_score=0.5) -> str:
    """Implements Section 3's full pipeline: FILTER marginal
    results, DEDUPLICATE, then ORDER by relevance -- most relevant
    FIRST, directly applying Section 4's lost-in-the-middle
    principle."""
    # FILTER: drop really marginal, low-relevance results
    filtered = [c for c in retrieved_chunks if c["score"] >= min_score]

    # DEDUPLICATE: remove exact repeats
    if dedupe:
        seen = set()
        unique_chunks = []
        for chunk in filtered:
            if chunk["text"] not in seen:
                seen.add(chunk["text"])
                unique_chunks.append(chunk)
        filtered = unique_chunks

    # ORDER: most relevant FIRST (Section 4's lost-in-the-middle fix)
    sorted_chunks = sorted(filtered, key=lambda c: c["score"], reverse=True)[:max_chunks]

    context_parts = [f"[Source: {c['source']}]\n{c['text']}" for c in sorted_chunks]
    return "\n\n".join(context_parts)

retrieved = [
    {"text": "London hotel limit is $250/night.", "source": "travel_policy_2026", "score": 0.95},
    {"text": "Standard hotel limit is $200/night.", "source": "travel_policy_2026", "score": 0.88},
    {"text": "London hotel limit is $250/night.", "source": "travel_policy_2026", "score": 0.95},
    {"text": "Office parking closes at 10pm.", "source": "facilities_faq", "score": 0.31},
]

context = construct_context(retrieved, max_chunks=3, min_score=0.5)
print(context)
print(f"\n--- Included {context.count('[Source:')} chunks (marginal parking chunk correctly excluded) ---")

Expected Output:

[Source: travel_policy_2026]
London hotel limit is $250/night.

[Source: travel_policy_2026]
Standard hotel limit is $200/night.

--- Included 2 chunks (marginal parking chunk correctly excluded) ---

What we conclude from this example: the duplicate London chunk was correctly deduplicated, the really marginal parking chunk (score 0.31) was correctly filtered out by the min_score threshold, and the remaining two chunks are ordered with the highest-scoring one first — exactly Section 6’s real developer example, implemented directly.

Example 2 — Intermediate

def demonstrate_ordering_effect(chunks: list, strategy: str) -> str:
    """Compares TWO ordering strategies -- RAW retrieval order
    (relevant chunk possibly buried in the middle) vs. RELEVANCE-
    SORTED order (most relevant FIRST) -- directly illustrating
    Section 4's lost-in-the-middle concern."""
    if strategy == "raw_order":
        ordered = chunks  # exactly as retrieved, unordered by relevance
    elif strategy == "relevance_sorted":
        ordered = sorted(chunks, key=lambda c: c["score"], reverse=True)
    else:
        raise ValueError(f"Unknown strategy: {strategy}")

    context_parts = [f"[position {i+1}, score={c['score']}] {c['text']}" for i, c in enumerate(ordered)]
    return "\n".join(context_parts)

# The MOST relevant chunk (score 0.95) is deliberately placed in the
# MIDDLE of the raw retrieval order -- exactly the risky scenario
# Section 4 describes.
chunks = [
    {"text": "General travel guidelines apply to all employees.", "score": 0.42},
    {"text": "London hotel limit is $250/night.", "score": 0.95},  # the KEY answer, buried in raw order
    {"text": "Submit receipts within 30 days.", "score": 0.55},
]

print("RAW retrieval order (key answer buried in the middle):")
print(demonstrate_ordering_effect(chunks, "raw_order"))

print("\nRELEVANCE-SORTED order (key answer moved to the front):")
print(demonstrate_ordering_effect(chunks, "relevance_sorted"))

Expected Output:

RAW retrieval order (key answer buried in the middle):
[position 1, score=0.42] General travel guidelines apply to all
employees.
[position 2, score=0.95] London hotel limit is $250/night.
[position 3, score=0.55] Submit receipts within 30 days.

RELEVANCE-SORTED order (key answer moved to the front):
[position 1, score=0.95] London hotel limit is $250/night.
[position 2, score=0.55] Submit receipts within 30 days.
[position 3, score=0.42] General travel guidelines apply to all
employees.

What we conclude from this example: in raw retrieval order, the highest-scoring chunk (0.95, the actual answer) sits at position 2 — the literal middle of this 3-chunk context, exactly the risky position Section 4 warns about. After relevance-sorting, that same chunk moves to position 1, directly applying the lost-in-the-middle mitigation: placing the most important content at the beginning rather than leaving its position to retrieval-order chance.

Example 3 — Production Grade

from dataclasses import dataclass

@dataclass
class ConstructedContext:
    final_text: str
    chunks_included: int
    chunks_filtered_out: int
    duplicates_removed: int
    top_chunk_position: int

class ContextConstructor:
    """A production-style context constructor tracking EXACTLY what
    happened at each stage -- filtering, deduplication, ordering --
    making the construction process really auditable, directly
    connecting to Module 33's observability practices from the
    Generative AI course."""

    def __init__(self, min_score: float = 0.5, max_chunks: int = 5):
        self.min_score = min_score
        self.max_chunks = max_chunks

    def construct(self, retrieved_chunks: list) -> ConstructedContext:
        original_count = len(retrieved_chunks)

        # FILTER
        filtered = [c for c in retrieved_chunks if c["score"] >= self.min_score]
        filtered_out_count = original_count - len(filtered)

        # DEDUPLICATE
        seen = set()
        unique_chunks = []
        for chunk in filtered:
            if chunk["text"] not in seen:
                seen.add(chunk["text"])
                unique_chunks.append(chunk)
        duplicates_removed = len(filtered) - len(unique_chunks)

        # ORDER: most relevant FIRST
        sorted_chunks = sorted(unique_chunks, key=lambda c: c["score"], reverse=True)[:self.max_chunks]

        final_text = "\n\n".join(f"[Source: {c['source']}]\n{c['text']}" for c in sorted_chunks)
        top_position = 1 if sorted_chunks else 0

        return ConstructedContext(
            final_text=final_text, chunks_included=len(sorted_chunks),
            chunks_filtered_out=filtered_out_count, duplicates_removed=duplicates_removed,
            top_chunk_position=top_position,
        )

retrieved = [
    {"text": "London hotel limit is $250/night.", "source": "travel_policy_2026", "score": 0.95},
    {"text": "Standard hotel limit is $200/night.", "source": "travel_policy_2026", "score": 0.88},
    {"text": "London hotel limit is $250/night.", "source": "travel_policy_2026", "score": 0.95},
    {"text": "Office parking closes at 10pm.", "source": "facilities_faq", "score": 0.31},
]

constructor = ContextConstructor(min_score=0.5, max_chunks=3)
result = constructor.construct(retrieved)

print(f"Chunks included: {result.chunks_included}")
print(f"Chunks filtered out (below threshold): {result.chunks_filtered_out}")
print(f"Duplicates removed: {result.duplicates_removed}")
print(f"Top chunk position in final context: {result.top_chunk_position}")

Expected Output:

Chunks included: 2
Chunks filtered out (below threshold): 1
Duplicates removed: 1
Top chunk position in final context: 1

What we conclude from this example: tracking chunks_filtered_out and duplicates_removed explicitly makes the entire construction process auditable — a real team could log these values across many production queries to monitor whether their filtering thresholds and deduplication are really working as intended, rather than trusting the process silently and without any visibility into what was actually removed or reordered.

14. Interview Questions

Q: Why can’t retrieved chunks be passed directly into a generation prompt without any additional processing?

Ans: Raw retrieval results can really include near-duplicate content (especially when combining multiple retrieval methods like hybrid search), and results that technically cleared a top-k cutoff but are still only marginally relevant. Passing this raw list directly wastes context space on redundant or low-value content, and doesn’t account for how the LLM actually uses information based on its position within a long context — deliberate filtering, deduplication, and ordering are really necessary steps between retrieval and generation.

Q: Explain the “lost in the middle” phenomenon and its practical implication for context construction.

Ans: Research on how LLMs use long context has found that models don’t use information equally well regardless of where it appears — content placed at the very beginning or very end of a long context tends to be used more reliably than content buried in the middle. The practical implication is that context construction should deliberately place the most relevant, important chunks at the beginning (or end) of the final context, rather than leaving their position to arbitrary retrieval order, since a really important chunk landing in the middle risks being underused by the model.

Q: Why isn’t “retrieve more chunks” automatically a good strategy for improving RAG answer quality?

Ans: More retrieved chunks means more content competing for the model’s attention, really diluting the prompt with potentially marginal or irrelevant information, and increasing the risk that truly important content ends up in a poorly-attended middle position due to the lost-in-the-middle effect. It also increases token cost. Retrieved context should be relevant, concise, and well-organized — quality and thoughtful construction matter more than raw quantity.

Q: Design a context construction pipeline that tracks its own effectiveness — what specific metrics would you log, and why?

Ans: I’d track the number of chunks filtered out for falling below a relevance threshold, the number of duplicates removed, and the final position of the highest-scoring chunk in the constructed context. These metrics make the construction process auditable — filtered and duplicate counts reveal whether retrieval itself is producing a lot of noise that construction has to clean up, while confirming the top chunk’s position verifies that the lost-in-the-middle mitigation (placing the most relevant content first) is actually being applied correctly and consistently across real production queries.

15. What You Should Remember

  • Context construction — filter, deduplicate, order — is really necessary between retrieval and generation, not an optional afterthought.
  • Lost in the middle means content position within the context really matters — verified directly by observing the key chunk moved from a risky middle position to a safer first position.
  • More retrieved chunks isn’t automatically better — quality, organization, and ordering matter more than raw quantity, verified directly through an auditable construction pipeline tracking exactly what was filtered, deduplicated, and reordered.

16. Quick Practice

Design a context construction strategy for a RAG system where the single most important piece of information is really likely to be the LAST chunk retrieved (lowest similarity score, but still above your relevance threshold) — how would you order the final context to account for both relevance AND the lost-in-the-middle effect?

17. Next Step

Next: Module 22 — Prompt Construction for RAG — closing Level 5: assembling the final prompt from constructed context, system instructions, and the user’s question, directly connecting to your Prompt Engineering course.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed