Embeddings and Representation Learning

Begin with the central question

How can meaning, similarity, and relationships be stored as lists of numbers?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

item → neural network → dense vector → distance or downstream model

Before you continue: three tools for this module

Vector: an ordered list of coordinates.
Dense vector: a vector in which most positions contain meaningful nonzero numbers.
Cosine similarity: a comparison of vector directions; values closer to 1 usually indicate greater similarity.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

What a learned representation actually is, what an embedding concretely is, and — critically — the precise, non-interchangeable distinction between an embedding, a hidden state, an activation, and a parameter. You’ll compute real cosine similarities between word vectors and run the classic “king − man + woman ≈ queen” demonstration yourself.

An embedding is a learned coordinate vector:

item → embedding model → [0.18, -0.42, 0.73, ...]
                              ↓ compare in one model's space
                     useful similarity signal

The coordinates are not universal facts about meaning. Their usefulness depends on the training objective, model, data, distance function, and evaluation task. Vectors from different embedding models should not be compared directly.

Why Neural Networks Learn Coordinates for Meaning

Module 1 introduced the idea that Deep Learning learns its own features instead of relying on hand-engineered ones. This module makes that idea completely concrete for the specific case of representing discrete things — words, sentences, documents — as vectors, in a way that captures genuine semantic relationships.

This is the single concept that bridges everything you’ve learned so far directly into RAG, semantic search, and vector databases.

an embedding is a learned map where distance and direction carry meaning. Words used in similar contexts end up as nearby vectors; words used in very different contexts end up far apart. Nobody tells the model “king and queen are related” — this relationship emerges purely from how the model learns to represent words that behave similarly in real text.

Analogy: The Multi-Dimensional Library Classification System Imagine trying to organize every book in a massive library using a system more sophisticated than alphabetical order:

One-Hot Encoding (Alphabetical Catalog): You assign each book a unique sequential number. Book #4321 is “The Hobbit”. While this ID is unique, the number tells you absolutely nothing about the content — you don’t know if #4321 is similar to #4322 or completely different.

Embeddings (The Dewey Decimal Space): Instead of one number, you assign each book a coordinate position in a multi-dimensional room based on its thematic attributes:

Axis X (Fictionality): $1.0$ for extreme fantasy, $-1.0$ for hard history.

Axis Y (Target Age): $1.0$ for toddlers, $-1.0$ for academics.

Axis Z (Action/Pace): $1.0$ for high-thrill chase scenes, $-1.0$ for slow philosophy.

In this library space, The Hobbit might land at [0.9, 0.2, 0.7], while The Lord of the Rings lands at [0.95, -0.1, 0.85]. Because their coordinate vectors point in similar directions, their cosine similarity is high (while cosine distance is low), reflecting their shared genres.

Vector Math in Coordinate Space: The king − man + woman ≈ queen example became famous because some historical embedding models showed approximately similar relationship directions. It is an illustrative pattern, not an exact equation or a universal property of embeddings.

📊 Visual Chart: Embeddings Space Vector Operations

Here is how semantic vector subtraction and addition map relationships geometrically in 3D coordinate space:

graph TD
    subgraph VectorSpace ["Semantic Embedding Coordinate Space"]
        King["King [0.9, 0.9, -0.8]<br>(Royalty, Male, Non-edible)"]
        Queen["Queen [0.9, -0.9, -0.8]<br>(Royalty, Female, Non-edible)"]
        Man["Man [-0.9, 0.9, -0.9]<br>(Commoner, Male, Non-edible)"]
        Woman["Woman [-0.9, -0.9, -0.9]<br>(Commoner, Female, Non-edible)"]
        Apple["Apple [-0.9, 0.0, 0.95]<br>(Commoner, Neutral, Edible)"]

King -->|Minus Man| Temp["[1.8, 0.0, 0.1]"]
        Temp -->|Plus Woman| Queen
    end

4. Core Concept

Term	Definition
Learned representation	Any internal numerical encoding a network produces, discovered through training rather than hand-specified
Dense vector	A vector where most/all values carry information (as opposed to a sparse, mostly-zero one-hot vector)
Embedding	A learned, dense vector specifically representing an entity (a token, sentence, document, image) in a way that captures meaningful relationships
Embedding layer	A network layer that maps discrete tokens to their embedding vectors — essentially a lookup table
Token embedding	An embedding specifically representing one token (e.g., one word or sub-word piece)

Embedding vs. hidden state vs. activation vs. parameter — precisely

⚠️ Do not oversimplify this relationship. These terms are related but genuinely distinct:

Parameter:        a LEARNED weight or bias -- part of the network
                    ITSELF, fixed after training (until the next
                    training step)

Activation:        a TEMPORARY value produced during a forward
                    pass -- recomputed every time new input flows
                    through

Hidden state:       an internal representation produced by the
                    model DURING PROCESSING, at some specific layer
                    and position -- a specific kind of activation

Embedding:          a learned representation/vector used to
                    represent an ENTITY (a token, sentence,
                    document). DEPENDING ON THE MODEL AND METHOD,
                    an embedding MAY BE derived from a hidden
                    state -- but they are not simply the same
                    thing by definition.

⚠️ Do not imply that all embeddings are simply copied from one hidden layer. In a simple word-embedding model, the embedding is literally a learned lookup-table row (itself a set of parameters). In a large language model, a “sentence embedding” is often derived from one or more hidden states (e.g., averaged, or taken from a specific layer/ position) — a related but genuinely more involved relationship, not a strict, universal equivalence.

5. How It Works — Step by Step

1. Each token in a vocabulary is assigned a vector (initially
   random) in an EMBEDDING LAYER -- essentially a big lookup table,
   one row per token
2. During training, as the model learns its main task (e.g.,
   next-word prediction, Module 6), gradients flow back (Module 7)
   into these embedding vectors too, since they're just more
   PARAMETERS being optimized
3. Tokens that behave similarly in the training data -- appearing
   in similar contexts -- gradually end up with similar vectors,
   PURELY as a side effect of optimizing the main training
   objective
4. After training, these vectors can be used directly for
   similarity comparisons, clustering, or retrieval -- without
   needing the original training task at all

6. Mathematical Intuition

First, use only small numbers

If two short vectors point in nearly the same direction, their cosine similarity is close to 1. If they point at right angles, it is near 0. An embedding model learns coordinates so useful relationships can often be detected through comparisons like this.

Read the mathematics as a story

An embedding is a learned coordinate. Training moves items used in similar ways toward useful regions of vector space, so geometric comparisons can support search, recommendation, and language understanding.

item → neural network → dense vector → distance or downstream model

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. Cosine similarity, the standard way to compare embeddings:

cosine_similarity(A, B) = (A · B) / (|A| × |B|)

A · B is the dot product; |A|, |B| are each vector’s magnitude (length). Dividing by the magnitudes means only the angle between vectors matters, not their length — two vectors pointing the same direction score 1.0; perpendicular vectors score 0.0; opposite directions score −1.0.

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible.

If “king” and “queen” tend to appear in similar sentence contexts across a huge training corpus (both often followed by words like “reign,” “crown,” “throne”), a model trained on next-word prediction will gradually push their embedding vectors closer together — not because anyone told it these words are related, but because doing so helps the model predict surrounding words more accurately for both.

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of Embeddings and Representation Learning.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

# In a REAL model these vectors are LEARNED (Module 7-8). Here they're
# hand-set to deliberately reflect plausible semantic structure, purely
# for illustration -- exactly what training would discover on its own.
embeddings = {
    "king":   np.array([0.90, 0.85, 0.10, 0.05]),
    "queen":  np.array([0.88, 0.80, 0.12, 0.55]),
    "man":    np.array([0.40, 0.30, 0.15, 0.05]),
    "woman":  np.array([0.38, 0.28, 0.18, 0.55]),
    "apple":  np.array([0.05, 0.10, 0.90, 0.20]),
    "banana": np.array([0.08, 0.12, 0.85, 0.15]),
}

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

print("Cosine similarity between word pairs:")
pairs = [("king", "queen"), ("king", "man"), ("king", "apple"), ("apple", "banana"), ("man", "woman")]
for w1, w2 in pairs:
    sim = cosine_similarity(embeddings[w1], embeddings[w2])
    print(f"  {w1:8s} vs {w2:8s}: {sim:.4f}")

# The classic embedding-arithmetic demonstration
result_vector = embeddings["king"] - embeddings["man"] + embeddings["woman"]
print("\n'king - man + woman' vector:", np.round(result_vector, 3))

best_match, best_sim = None, -1
for word, vec in embeddings.items():
    if word == "king":
        continue
    sim = cosine_similarity(result_vector, vec)
    print(f"  similarity to '{word}': {sim:.4f}")
    if sim > best_sim:
        best_sim, best_match = sim, word

print(f"\nClosest match to 'king - man + woman': '{best_match}'")

Expected Output:

Cosine similarity between word pairs:
  king     vs queen   : 0.9242
  king     vs man     : 0.9704
  king     vs apple   : 0.1992
  apple    vs banana  : 0.9979
  man      vs woman   : 0.7418

'king - man + woman' vector: [0.88 0.83 0.13 0.55]
  similarity to 'queen': 0.9998
  similarity to 'man': 0.9255
  similarity to 'woman': 0.8953
  similarity to 'apple': 0.2855
  similarity to 'banana': 0.3107

Closest match to 'king - man + woman': 'queen'

9. How It Works

Fruit words (apple, banana) score very high similarity to each other (0.9979) and low similarity to royal words (king vs. apple: 0.1992) — the embedding space genuinely separates unrelated concepts.
The vector arithmetic result is genuinely striking: computing king − man + woman and comparing it against every other word’s embedding finds queen as the closest match by a wide margin (0.9998, versus 0.93 or lower for everything else) — the famous “embedding arithmetic” phenomenon, reproduced here numerically, not just described. This works precisely because “royalty” and “gender” turned out to be roughly consistent directions across this (admittedly hand-constructed) embedding space — real, trained embeddings exhibit this behavior along many such directions, discovered automatically.

⚠️ Caution on interpretation: these particular numbers were hand-constructed to illustrate a real, well-documented phenomenon genuinely observed in trained embedding models — not proof that every embedding space cleanly represents every human concept as a linear direction. Real embeddings are messier and less perfectly interpretable than this deliberately clean example.

10. Real-World Example

A RAG system embeds every document chunk in a knowledge base using an embedding model, storing the resulting vectors in a vector database.

When a user asks a question, the question itself is embedded using the same model, and the documents whose embeddings have the highest cosine similarity to the question’s embedding are retrieved as likely relevant context — mechanically identical to Section 8’s cosine_similarity function, just applied to sentence/paragraph-level embeddings instead of single words, and at a scale of potentially millions of documents.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

Embedding models turn queries and document chunks into vectors for semantic search and RAG. All compared vectors must come from compatible model versions and preprocessing; an LLM then reads retrieved text, not the database’s raw vector coordinates.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

Every LLM has a token embedding layer as its very first component (Module 17 traces this precisely) — converting each input token into a dense vector before any Transformer processing happens at all. Sentence or document embeddings, used throughout RAG and semantic search systems, are typically derived from a trained language model’s internal representations — either a dedicated embedding model, or hidden states extracted from a general-purpose LLM.

Concept	AI application
Token embeddings	The very first layer of every LLM
Cosine similarity	The standard metric for comparing embeddings in vector databases
Sentence/document embeddings	The foundation of RAG retrieval and semantic search
Embedding vector space	What a vector database actually stores and searches over

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Very High. This is one of the most directly load-bearing concepts for building agents.

Agent
  ↓
Memory / RAG
  ↓
Embedding (query and stored content, embedded with the SAME model)
  ↓
Vector Search (cosine similarity, computed exactly as above, at scale)
  ↓
Retrieved context, fed back into the agent's reasoning

An agent’s retrieval-augmented memory — recalling relevant past interactions, or grounding a response in a document store — works through exactly this mechanism, every single time.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: an embedding is just a hidden state, full stop.

Why it is incorrect: As Section 4 states explicitly: related, but not strictly the same — an embedding may be derived from a hidden state depending on the model and method, but isn’t universally defined as one.

⚠️ Mistake

Incorrect idea: comparing embeddings from two different embedding models works fine.

Why it is incorrect: It doesn’t — different models learn different, incompatible vector spaces. An embedding from Model A and one from Model B occupy unrelated coordinate systems; comparing them directly produces meaningless results. Always embed everything you intend to compare using the same model.

⚠️ Mistake

Incorrect idea: embeddings perfectly capture every human concept as a clean linear direction.

Why it is incorrect: Section 9’s caution applies here — this approximate pattern was observed in some influential historical embedding models, but embedding spaces are messier than the clean illustration. It is not guaranteed across models, words, languages, or relationships.

14. Important Distinctions

Embedding	Hidden State
A learned representation of an ENTITY (token, sentence, document)	An internal representation produced DURING PROCESSING, at a specific layer/position
May be derived from a hidden state, depending on model/method	A more general term for any internal activation at any layer

Activation	Parameter
A temporary value from a forward pass	A learned weight or bias, part of the model itself
Recomputed every time	Fixed until the next training update

15. When to Use

Use embeddings (via cosine similarity or a vector database) whenever you need to measure semantic similarity between pieces of content — the standard, modern solution for RAG, semantic search, deduplication, and clustering unstructured content.

16. When Not to Use

Don’t compare embeddings produced by different models. Don’t use embeddings where exact keyword/identifier matching is genuinely what’s needed (e.g., a specific product SKU or error code) — many production systems combine embedding-based semantic search with traditional keyword search for exactly this reason.

17. Interview Questions

Beginner

Q: What is an embedding?

Ans: A learned, dense vector representing an entity — a token, sentence, document, or image — structured so that distance and direction in the vector space reflect meaningful relationships: similar entities end up with similar vectors.

Intermediate

Q: What’s the difference between an embedding and a hidden state?

Ans: A hidden state is a general term for any internal representation a model produces during processing, at a specific layer and position.

An embedding specifically represents an entity (like a token or a sentence) — and depending on the model and method, it may be derived directly from one or more hidden states, but the two terms aren’t strictly interchangeable; an embedding is a more specific, purpose-built concept.

Advanced

Q: Why can’t you compare an embedding from one model against an embedding from a different model?

Ans: Each embedding model is trained independently and develops its own internal vector space, with its own arbitrary geometry shaped by that specific model’s training process. There’s no guarantee that “pointing in a similar direction” means the same thing across two different models’ spaces — comparing them is comparing coordinates from two unrelated maps.

All embeddings intended for comparison must come from the same model.

Scenario

Q: You compute cosine similarity between a user’s query embedding and your document embeddings, and retrieval quality is poor — semantically related documents aren’t being retrieved. What would you investigate?

Ans: First, confirm the query and documents were embedded using the exact same model — a mismatch here silently produces meaningless similarity scores.

I’d also check whether the embedding model is well-suited to the domain (a general-purpose model may underperform on highly specialized text), and whether documents were chunked sensibly before embedding — overly long or poorly-segmented chunks can dilute the embedding’s semantic focus.

AI Engineering

Q: Trace, at a high level, how a RAG system uses this module’s concepts end to end.

Ans: Documents are split into chunks and each chunk is converted into an embedding using a chosen embedding model, then stored in a vector database.

At query time, the user’s question is embedded using the same model, and cosine similarity is computed between the query embedding and every stored document embedding — the highest-scoring chunks are retrieved as relevant context, then passed to the LLM alongside the original question to generate a grounded response.

18. What You Should Remember

An embedding is a learned, dense vector representing an entity, structured so distance/direction reflect semantic similarity.
Embedding ≠ hidden state ≠ activation ≠ parameter — related, but precisely distinct concepts (Section 4).
Cosine similarity is the standard metric for comparing embeddings — verified numerically here, including the classic “king − man + woman ≈ queen” demonstration.
Never compare embeddings from different models — they occupy incompatible vector spaces.

19. How This Helps Me Build AI Systems

This module is the direct mechanical foundation of every RAG system and every agent memory/retrieval component you’ll build. You’ve now computed real cosine similarities and reproduced the famous embedding-arithmetic result yourself — the exact mechanism vector databases run, at massive scale, every time a RAG system retrieves relevant context.

Next: Module 13 — CNNs and Computer Vision — how convolution and pooling let networks learn hierarchical visual features directly from pixels.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026