Embeddings in the Context of RAG

Begin with the problem

Embedding text is not the goal; retrieving useful evidence is. In RAG, embeddings are measuring tools whose quality depends on the documents, queries, and evaluation set.

query → vector/filters → index search → top candidates

What you will learn

Explain Embeddings in the Context of RAG in simple language before using its technical details.
Follow the mechanism step by step through a small RAG example.
Connect this topic to the modules before and after it.
Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: OpenAI’s vector store API and Google’s File Search guide are current examples of managed vector retrieval. Exact indexes and tuning controls vary by product.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Level 2 covered turning documents into well-structured, metadata-rich chunks. Level 3 begins here: how those chunks actually become searchable by meaning. You already know embeddings from your LLM and NLP courses — this module doesn’t re-teach that foundation, it applies it specifically to the retrieval problem RAG needs solved.

2. The Problem — How Do You Search for Meaning, Not Exact Words?

Employee's question:      "How much can I claim for a hotel?"

Actual chunk text:            "Employees are eligible for
                             accommodation reimbursement up to $200
                             per night."

Notice: not a single word matches exactly. “claim” vs. “reimbursement.” “hotel” vs. “accommodation.” A simple keyword search comparing exact words would really miss this chunk entirely — even though it’s precisely the right answer.

3. Why Embeddings Solve This

"car"        vs.        "automobile"        vs.        "vehicle"

These are different words — but really related meaning.

Text
   ↓
Embedding Model
   ↓
Vector (a list of numbers)

An embedding model converts text into a vector such that texts with similar MEANING produce vectors that are mathematically CLOSE to each other — regardless of whether they share any exact words at all.

This is exactly why “How much can I claim for a hotel?” and “accommodation reimbursement up to $200 per night” can be recognized as really related, despite sharing zero literal words.

4. Embeddings for Retrieval, Specifically

Your LLM course covered embeddings broadly. In RAG, embeddings serve one very specific, narrow purpose:

Query embedding
   +
Document (chunk) embeddings
   ↓
SIMILARITY (Module 11)
   ↓
Relevant chunks

Both the user’s question and every stored chunk get embedded using the same embedding model, into the same vector space — this is what makes them directly comparable to each other at all. If you embedded queries with one model and chunks with a different, unrelated model, their vectors would live in really incompatible spaces, and comparing them would be meaningless.

5. A Real Developer Example

TechCorp embeds ALL 8 chunks of its travel policy document (Module 9)
using the SAME embedding model.

An employee asks: "How much can I claim for a hotel?"

This question is embedded using the SAME model.

The resulting query vector is compared against all 8 chunk vectors
(Module 11's similarity, Module 13's search) -- and the chunk about
"$200 per night accommodation reimbursement" is correctly identified
as the closest match, DESPITE having no words in common with the
actual question.

This is the ENTIRE reason semantic search outperforms simple keyword
matching for a huge range of real, natural-language questions.

6. A Simple Agentic AI Connection

An agent’s “search knowledge base” tool relies entirely on this mechanism — when the agent formulates a search query (which may be really different phrasing than the user’s original question, Module 19’s query transformation), that query still needs to be embedded into the same vector space as the stored chunks for retrieval to work at all.

7. How Is This Used in AI?

🤖 How Is This Used in AI?

Embedding models are the foundational technology behind every semantic search and RAG system in production — chosen deliberately based on their language coverage, embedding dimensionality, and real retrieval quality on representative evaluation data (Module 32), not simply defaulted to without consideration.

8. Real-World Applications

Semantic search across enterprise knowledge bases
Natural-language question answering over documentation
Recommendation systems finding “similar” content by meaning

9. Common Mistakes

Incorrect idea: Embedding queries and documents with different, incompatible models.

Why it is incorrect: As shown directly in Section 4, this really breaks comparability — both need the SAME embedding model and space.

Incorrect idea: Assuming embeddings guarantee perfect semantic understanding.

Why it is incorrect: Embedding quality really varies, and even good embeddings can miss nuanced or highly technical distinctions — Module 32’s evaluation exists precisely because this can’t just be assumed to work.

Incorrect idea: Re-deriving embedding fundamentals from scratch instead of building on your LLM course.

Why it is incorrect: This module deliberately connects to, rather than repeats, that foundation — the mechanism is identical.

10. Limitations

Embedding quality really depends on the specific model and how well it was trained on content similar to your actual domain
Some really important distinctions (exact IDs, specific numbers, rare technical terms) can be poorly captured by semantic embeddings alone — Module 16’s discussion of hybrid search addresses this directly

11. Quick Reference — The Whole Idea in One Diagram

Query text                              Chunk text
   ↓                                       ↓
SAME Embedding Model                    SAME Embedding Model
   ↓                                       ↓
Query Vector          <-- compare -->   Chunk Vector

Similar MEANING -> vectors end up CLOSE, even with different words

12. Code — Demonstrating Semantic Match Without Word Overlap

🎯 Target of this example: directly reproduce Section 2’s real problem and Section 3’s solution — showing that a really zero-word-overlap query and chunk can still be correctly matched through embedding similarity.

Example 1 — Simple

import numpy as np

def embed_text(text: str) -> np.ndarray:
    """A SIMPLIFIED, illustrative embedding using shared CONCEPT
    words (not exact matches) -- meant to demonstrate the ALGORITHM;
    a real embedding model (like those covered in your LLM course)
    captures FAR richer semantic relationships automatically, without
    needing hand-picked concept groups like this."""
    concept_groups = {
        "money_claim": ["claim", "reimbursement", "eligible", "cost", "pay", "money"],
        "lodging": ["hotel", "accommodation", "stay", "room", "lodging"],
        "travel": ["travel", "trip", "international", "night"],
    }
    text_lower = text.lower()
    return np.array([
        sum(1 for word in words if word in text_lower) for words in concept_groups.values()
    ], dtype=float)

def cosine_similarity(a, b):
    norm_a, norm_b = np.linalg.norm(a), np.linalg.norm(b)
    return 0.0 if norm_a == 0 or norm_b == 0 else np.dot(a, b) / (norm_a * norm_b)

query = "How much can I claim for a hotel?"
chunk = "Employees are eligible for accommodation reimbursement up to $200 per night."

query_vector = embed_text(query)
chunk_vector = embed_text(chunk)

print(f"Query vector: {query_vector}")
print(f"Chunk vector: {chunk_vector}")
print(f"Similarity: {cosine_similarity(query_vector, chunk_vector):.3f}")
print(f"\nShared exact words: {set(query.lower().split()) & set(chunk.lower().split())}")

Expected Output:

Query vector: [1. 1. 0.]
Chunk vector: [2. 1. 1.]
Similarity: 0.866

Shared exact words: {'for'}

What we conclude from this example: despite really sharing almost no meaningful exact words (only the function word “for”), the query and chunk score a high 0.866 similarity — because both activate the same underlying CONCEPTS (money/claiming, lodging). This directly demonstrates Section 3’s core claim: semantic embeddings capture meaning that plain keyword overlap would completely miss.

Example 2 — Intermediate

import numpy as np

def embed_text(text: str) -> np.ndarray:
    concept_groups = {
        "money_claim": ["claim", "reimbursement", "eligible", "cost", "pay", "money"],
        "lodging": ["hotel", "accommodation", "stay", "room", "lodging"],
        "travel": ["travel", "trip", "international", "night"],
    }
    text_lower = text.lower()
    return np.array([
        sum(1 for word in words if word in text_lower) for words in concept_groups.values()
    ], dtype=float)

def cosine_similarity(a, b):
    norm_a, norm_b = np.linalg.norm(a), np.linalg.norm(b)
    return 0.0 if norm_a == 0 or norm_b == 0 else np.dot(a, b) / (norm_a * norm_b)

def embed_all_chunks(chunks: list) -> dict:
    """Embeds a FULL knowledge base of chunks using the SAME model
    that will later embed the query -- Section 4's requirement that
    query and chunks share a vector space."""
    return {chunk: embed_text(chunk) for chunk in chunks}

knowledge_base = [
    "Employees are eligible for accommodation reimbursement up to $200 per night.",
    "The office parking garage closes at 10pm on weekdays.",
    "London and Tokyo hotel stays have a raised limit of $250 per night.",
]

chunk_embeddings = embed_all_chunks(knowledge_base)
query = "How much can I claim for a hotel?"
query_vector = embed_text(query)

print(f"Query: {query}\n")
for chunk, chunk_vector in chunk_embeddings.items():
    similarity = cosine_similarity(query_vector, chunk_vector)
    print(f"  similarity={similarity:.3f}  {chunk}")

Expected Output:

Query: How much can I claim for a hotel?

  similarity=0.866  Employees are eligible for accommodation
  reimbursement up to $200 per night.
  similarity=0.000  The office parking garage closes at 10pm on
  weekdays.
  similarity=0.632  London and Tokyo hotel stays have a raised limit
  of $250 per night.

What we conclude from this example: the really irrelevant parking-garage chunk correctly scores 0.000 similarity, while both really relevant hotel-related chunks score meaningfully higher — this is exactly what a retrieval system needs: a way to RANK chunks by real relevance to a query, using nothing but the meaning captured in their embeddings. Notice the first chunk scores higher (0.866) than the second (0.632) — it more directly addresses “eligible … reimbursement,” while the second chunk is about the exception amount specifically, a really real, useful distinction the embedding captures.

Example 3 — Production Grade

import numpy as np
from dataclasses import dataclass

@dataclass
class EmbeddedChunk:
    text: str
    embedding: np.ndarray
    document_id: str  # carried forward from Module 9's metadata

def embed_text(text: str) -> np.ndarray:
    concept_groups = {
        "money_claim": ["claim", "reimbursement", "eligible", "cost", "pay", "money"],
        "lodging": ["hotel", "accommodation", "stay", "room", "lodging"],
        "travel": ["travel", "trip", "international", "night"],
    }
    text_lower = text.lower()
    return np.array([
        sum(1 for word in words if word in text_lower) for words in concept_groups.values()
    ], dtype=float)

def cosine_similarity(a, b):
    norm_a, norm_b = np.linalg.norm(a), np.linalg.norm(b)
    return 0.0 if norm_a == 0 or norm_b == 0 else np.dot(a, b) / (norm_a * norm_b)

class EmbeddingPipeline:
    """A production-style pipeline enforcing that the SAME embedding
    function is used for BOTH indexing chunks and embedding queries --
    structurally guaranteeing Section 4's requirement, rather than
    leaving it to convention."""

    def __init__(self):
        self._embedded_chunks: list[EmbeddedChunk] = []

    def index_chunks(self, chunks_with_ids: list) -> None:
        for text, document_id in chunks_with_ids:
            self._embedded_chunks.append(
                EmbeddedChunk(text=text, embedding=embed_text(text), document_id=document_id)
            )

    def search(self, query: str, top_n: int = 2) -> list:
        query_vector = embed_text(query)  # SAME embed_text function, guaranteed
        scored = [
            (chunk, cosine_similarity(query_vector, chunk.embedding))
            for chunk in self._embedded_chunks
        ]
        scored.sort(key=lambda x: x[1], reverse=True)
        return scored[:top_n]

pipeline = EmbeddingPipeline()
pipeline.index_chunks([
    ("Employees are eligible for accommodation reimbursement up to $200 per night.", "travel_policy_2026"),
    ("The office parking garage closes at 10pm on weekdays.", "facilities_faq"),
    ("London and Tokyo hotel stays have a raised limit of $250 per night.", "travel_policy_2026"),
])

results = pipeline.search("How much can I claim for a hotel?", top_n=2)
for chunk, score in results:
    print(f"[{score:.3f}] ({chunk.document_id}) {chunk.text}")

Expected Output:

[0.866] (travel_policy_2026) Employees are eligible for accommodation
reimbursement up to $200 per night.
[0.632] (travel_policy_2026) London and Tokyo hotel stays have a
raised limit of $250 per night.

What we conclude from this example: the EmbeddingPipeline class structurally guarantees the same embed_text function embeds both chunks and queries — an architectural safeguard against Section 9’s “different, incompatible models” mistake — and correctly retrieves both really relevant chunks, both carrying their document_id metadata forward from Module 9, ready to support citation in the next stage of the pipeline.

13. Interview Questions

Q: Why can’t simple keyword matching reliably answer a question like “How much can I claim for a hotel?” against a document that says “accommodation reimbursement”?

Ans: Keyword matching compares exact words, and this query and document share essentially no meaningful words in common — “claim” versus “reimbursement,” “hotel” versus “accommodation.” Even though these phrases mean really the same thing to a human reader, keyword matching has no way to recognize that relationship, since it only compares literal text, not underlying meaning.

Q: Explain, at a mechanical level, why embeddings solve the problem keyword search cannot.

Ans: An embedding model converts text into a vector of numbers such that texts with similar meaning produce vectors that are mathematically close to each other, regardless of whether they share exact words. Since “claim for a hotel” and “accommodation reimbursement” express related underlying concepts, a well-trained embedding model produces vectors for both that end up close together in the vector space, even though the literal words are completely different.

Q: Why must the query and the stored chunks be embedded using the same embedding model?

Ans: Different embedding models can produce vectors in entirely different, incompatible vector spaces, even if trained for a similar general purpose — a vector from one model has no guaranteed mathematical relationship to a vector from a different model. For similarity comparison between a query and stored chunks to be meaningful at all, both need to be embedded using the identical model, so their vectors really live in the same, comparable space.

Q: What’s a real limitation of relying purely on semantic embeddings for retrieval, even when they work well?

Ans: Semantic embeddings can struggle with content where exact precision really matters more than general meaning — specific IDs, exact numbers, or rare technical terms may not be well captured by an embedding model that’s optimized for broader semantic similarity. This is precisely why hybrid search approaches combining semantic embeddings with exact keyword matching (covered later in this course) exist — to cover cases where pure semantic similarity alone isn’t sufficient.

14. What You Should Remember

Embeddings let retrieval find content by meaning, not exact words — verified directly by matching a query and a chunk sharing essentially zero literal words.
Query and chunks must be embedded with the same model into the same vector space, or comparison becomes meaningless — verified directly through a pipeline structurally enforcing this guarantee.
This module builds on, rather than repeats, your LLM course’s embedding foundation — applied specifically to the retrieval problem.

15. Quick Practice

Think of a question you might ask that shares almost no exact words with its correct answer (like this module’s hotel/accommodation example). Explain why embedding-based retrieval would find it while keyword search likely wouldn’t.

16. Next Step

Next: Module 11 — Vector Space & Similarity — the mathematical comparison tools (cosine similarity, dot product, Euclidean distance) that turn “close vectors” into an actual, computable ranking.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026