TechByteByByte

Word Embeddings

Understand Word2Vec, CBOW, Skip-gram, and GloVe intuition — the direct answer to Module 7's proven TF-IDF failure — with the classic 'king - man + woman ≈ queen' result reproduced and verified numerically.

#NLP#AI#Word Embeddings#Word2Vec#GloVe

Begin with the central question

Can a model learn that car and automobile are related despite different spellings?

Essential words

An embedding is a learned vector. A dense vector stores learned decimal values. Cosine similarity compares vector directions to estimate relatedness.

What You Will Understand

Word embeddings — dense, learned vector representations where semantic relationships genuinely emerge — as the direct fix for Module 7’s proven TF-IDF failure. You’ll cover Word2Vec’s CBOW and Skip-gram intuition, GloVe conceptually, and reproduce the famous “king − man + woman ≈ queen” result with real, verified numbers.

word -> learned dense vector -> nearby meanings

Why Words Need Learned Coordinates

Module 7 proved, with real numbers, that TF-IDF’s “one word = one fixed vocabulary position” representation cannot capture meaning — synonyms are unrelated, word senses are conflated, and similarity can even come out backwards. Word embeddings exist to fix this at the representational level: instead of an arbitrary vocabulary index, every word gets a dense, learned vector positioned so that semantically related words end up genuinely close together.

Meaning as a Position in Space

you already covered this exact concept in your Deep Learning course (DL Module 12) — an embedding is a learned map where distance and direction carry meaning. This module applies that same mechanism specifically to solve NLP’s word-representation problem, tracing precisely how training on raw text produces this structure.

Analogy: The Semantic Coordinate Map (The Word GPS) Imagine mapping cities on a physical globe using Latitude and Longitude coordinates:

  • The Representation: You don’t just assign cities arbitrary ID numbers. You place them in a 2D space where distance represents physical distance. Dallas and Fort Worth have very similar coordinate vectors, while Dallas and Tokyo are very far apart.
  • ** Emergent Directions:** Furthermore, directional movements represent physical vectors:
    • Going from London to Paris requires traveling South-East.
    • Going from Edinburgh to London requires traveling South.
  • Word Embeddings: This is exactly what a word embedding does to vocabulary. It maps words into a multi-dimensional space where:
    • Synonyms like “car” and “automobile” land at nearly identical coordinates.
    • Semantic transitions (like gender, tense, or capital cities) become consistent directional steps. The vector direction from “Man” to “Woman” is identical to the vector step from “King” to “Queen”. This is why King - Man + Woman yields a vector right next to Queen.

📊 Visual Chart: Word Embeddings Coordinate Space & Semantic Math

Here is how semantic concepts (like Gender and Royalty axes) align words in vector space, making vector arithmetic possible:

graph TD
    subgraph Space ["2D Word Embedding Space Projection"]
        King["King<br>[Royalty: 0.9, Gender: 0.8]"]
        Queen["Queen<br>[Royalty: 0.9, Gender: -0.8]"]
        Man["Man<br>[Royalty: -0.9, Gender: 0.8]"]
        Woman["Woman<br>[Royalty: -0.9, Gender: -0.8]"]

Man -->|South: Gender shift (-1.6)| Woman
        King -->|South: Gender shift (-1.6)| Queen
    end

4. Core Concept

TermDefinition
Word embeddingA dense, learned vector representing a word’s meaning
Word2VecA family of methods for learning word embeddings from raw text, via CBOW or Skip-gram
CBOW (Continuous Bag of Words)Predicts a target word FROM its surrounding context words
Skip-gramPredicts surrounding CONTEXT words FROM a target word (the reverse of CBOW)
GloVeAn alternative embedding method using global word co-occurrence statistics across the whole corpus, rather than local context windows

How training produces meaningful structure

CBOW:       context words  ->  predict the TARGET word
            ("the ___ rules the kingdom" -> predict "king")

Skip-gram:   target word  ->  predict the CONTEXT words
            ("king" -> predict "the", "rules", "kingdom", ...)

🧠 Words that tend to appear in similar contexts (“king” and “queen” both frequently appear near “rules,” “throne,” “crown”) get pushed, through this training process, toward similar embedding vectors — purely as an emergent side effect of the model getting better at this prediction task. Nobody manually tells the model “king and queen are related.”

5. How It Works — Step by Step (Skip-gram)

1. Slide a WINDOW across the training corpus
2. For each target word, generate (target, context) TRAINING
   PAIRS with every word within the window
3. Train a small neural network (DL course Modules 2, 5) to
   predict context words GIVEN the target word
4. During training, the network's INTERNAL representation of
   each word (an embedding, DL course Module 12) gradually
   shifts so that words used in similar contexts end up with
   similar embeddings -- a direct consequence of the prediction
   objective, not a separately imposed rule
5. After training, DISCARD the prediction task itself -- the
   learned embeddings are the actual useful output

6. Mathematical Intuition

Cosine similarity (already covered in DL course Module 12, and Module 7 of this course) is the standard way to measure how related two embeddings are:

cosine_similarity(A, B) = (A · B) / (|A| × |B|)

The key property this module verifies: embeddings trained to solve a prediction task end up with a genuinely useful, emergent side effect — directions in the embedding space can correspond to consistent semantic relationships (like “royalty” or “gender”), verified concretely below.

7. Simple Example

If “king” and “queen” tend to co-occur with similar surrounding words across a large training corpus (both near “throne,” “reign,” “royal”), Skip-gram training will push their embeddings closer together — purely because doing so helps the model predict shared context words more accurately. This is precisely how “king” and “queen” end up with high cosine similarity, without anyone hand-labeling them as related.

8. Build It in Python

What the code will demonstrate

The vectors below are small teaching vectors chosen to make semantic relationships visible; they are not downloaded Word2Vec or GloVe values. Real embedding values are learned from large corpora and usually contain hundreds or thousands of dimensions.

Follow the important operation: subtract one relationship from a vector and add another, then use cosine similarity to find which candidate points in the most similar direction.

import numpy as np

# In a REAL Word2Vec/GloVe model these vectors are LEARNED from massive
# text corpora. Here they're hand-constructed to illustrate a real,
# well-documented phenomenon genuinely observed in trained embeddings.
embeddings = {
    "king":   np.array([0.90, 0.85, 0.10, 0.05]),
    "queen":  np.array([0.88, 0.80, 0.12, 0.55]),
    "man":    np.array([0.40, 0.30, 0.15, 0.05]),
    "woman":  np.array([0.38, 0.28, 0.18, 0.55]),
    "apple":  np.array([0.05, 0.10, 0.90, 0.20]),
    "banana": np.array([0.08, 0.12, 0.85, 0.15]),
}

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

print("Cosine similarity between word pairs:")
pairs = [("king", "queen"), ("king", "man"), ("king", "apple"), ("apple", "banana")]
for w1, w2 in pairs:
    sim = cosine_similarity(embeddings[w1], embeddings[w2])
    print(f"  {w1:8s} vs {w2:8s}: {sim:.4f}")

# The classic embedding arithmetic
result_vector = embeddings["king"] - embeddings["man"] + embeddings["woman"]
print("\n'king - man + woman' vector:", np.round(result_vector, 3))

best_match, best_sim = None, -1
for word, vec in embeddings.items():
    if word == "king":
        continue
    sim = cosine_similarity(result_vector, vec)
    print(f"  similarity to '{word}': {sim:.4f}")
    if sim > best_sim:
        best_sim, best_match = sim, word

print(f"\nClosest match to 'king - man + woman': '{best_match}'")

# --- Skip-gram: (target, context) training pair generation ---
sentence = "the king rules the kingdom wisely".split()
window = 2
print("\nSkip-gram style (target, context) training pairs (window=2):")
for i, target in enumerate(sentence):
    context_indices = [j for j in range(max(0, i-window), min(len(sentence), i+window+1)) if j != i]
    for j in context_indices:
        print(f"  ({target}, {sentence[j]})")

Expected Output:

Cosine similarity between word pairs:
  king     vs queen   : 0.9242
  king     vs man     : 0.9704
  king     vs apple   : 0.1992
  apple    vs banana  : 0.9979

'king - man + woman' vector: [0.88 0.83 0.13 0.55]
  similarity to 'queen': 0.9998
  similarity to 'man': 0.9255
  similarity to 'woman': 0.8953
  similarity to 'apple': 0.2855
  similarity to 'banana': 0.3107

Closest match to 'king - man + woman': 'queen'

Skip-gram style (target, context) training pairs (window=2):
  (the, king)
  (the, rules)
  (king, the)
  (king, rules)
  (king, the)
  (rules, the)
  (rules, king)
  (rules, the)
  (rules, kingdom)
  (the, king)
  (the, rules)
  (the, kingdom)
  (the, wisely)
  (kingdom, rules)
  (kingdom, the)
  (kingdom, wisely)
  (wisely, the)
  (wisely, kingdom)

9. How It Works

  • Fruit words score high similarity to each other (apple vs. banana: 0.9979) and low similarity to royal words (king vs. apple: 0.1992) — embeddings genuinely separate unrelated concepts, directly unlike Module 7’s TF-IDF, which had no mechanism for this at all.
  • The vector arithmetic result is genuinely striking: computing king − man + woman and comparing against every other word finds queen as the closest match by a wide margin (0.9998, versus ~0.93 or lower for everything else) — the famous embedding-arithmetic phenomenon, reproduced numerically, not just described.
  • Skip-gram’s training pairs show exactly how a single sentence generates many (target, context) examples — “king” gets paired with both “the” and “rules” (its window-2 neighbors), and this exact pairing process, repeated across a massive corpus, is what shapes the embedding space’s structure.

⚠️ Caution on interpretation: these particular numbers were hand-constructed to illustrate a real, well-documented phenomenon observed in actually-trained embedding models — not proof that every embedding space cleanly represents every human concept as a linear direction. Real embeddings are messier and less perfectly interpretable than this deliberately clean example (a caution worth repeating from the Deep Learning course’s own embeddings module).

10. Strengths and Remaining Limitations

Strengths, directly addressing Module 7’s proven failure

  • Synonyms end up genuinely related, not arbitrary unrelated vocabulary entries.
  • Dense, fixed-size vectors regardless of vocabulary size (unlike Bag of Words/TF-IDF’s sparse, vocabulary-sized vectors).
  • Semantic relationships emerge from training, without manual labeling.

The limitation that remains (setting up Module 9)

Word embeddings assign ONE fixed vector PER WORD, regardless of context.

"bank" gets exactly ONE embedding -- averaging together its
financial-institution sense and its riverbank sense, since
training saw BOTH uses mixed together in the corpus.

This is a genuine, real limitation — Module 9 demonstrates it directly and traces the path toward the fix.

11. How Is This Used in Modern AI?

🤖 How Is This Used in Modern AI?

Word embeddings were the direct historical and conceptual predecessor to modern embedding models used in RAG and semantic search today — the core mechanism (learn dense vectors where meaning determines closeness) is unchanged; what evolved is how those vectors are produced (Module 13’s contextual representations, and eventually Transformer-based embedding models).

ConceptAI application
Cosine similarity between embeddingsThe standard metric in every vector database (DL course Module 12)
Embedding arithmeticA genuinely real, verified phenomenon in trained embedding spaces
Skip-gram/CBOW trainingThe historical foundation; modern embedding models use more sophisticated (often Transformer-based) training, but the “learn from context” principle persists

Real systems you can recognize

OpenAI’s embedding documentation connects text-to-vector conversion with search, while Hugging Face exposes feature-extraction and sentence-similarity model tasks. See OpenAI embeddings and Hugging Face tasks.

Modern retrieval embeddings usually represent a sentence or passage with context, rather than using the average of old Word2Vec vectors. Static word embeddings remain valuable for learning the core geometry and for small legacy systems.

12. How Is This Used in Agentic AI?

Direct relevance to Agentic AI: Very High. This is the exact mechanism underlying every RAG retrieval system and agent memory component you’ll build — embed a query, embed stored content, retrieve by cosine similarity. You already built and verified this precise computation in the Deep Learning course (DL Module 12); this module adds the specific NLP-historical context of why embeddings were originally developed and how CBOW/Skip-gram training produces this useful structure.

13. Common Mistakes / Misunderstandings

⚠️ Mistake: assuming word embeddings fully solve the ambiguity problem from Module 7. They significantly improve on TF-IDF’s total lack of semantic structure, but as Section 10 states directly, they still assign one fixed vector per word — the word-sense problem isn’t fully solved until Module 13’s contextual representations.

⚠️ Mistake: confusing CBOW and Skip-gram’s direction. CBOW predicts a target word FROM context; Skip-gram predicts context words FROM a target — opposite directions of the same underlying co-occurrence signal.

⚠️ Mistake: believing embedding arithmetic works perfectly for every possible relationship. As cautioned directly, this module’s example was constructed to cleanly illustrate a real phenomenon — actual trained embeddings are messier, and not every conceptual relationship is this cleanly linear.

14. Important Distinctions

CBOWSkip-gram
Context words → predict target wordTarget word → predict context words
Generally faster, works well with frequent wordsGenerally works better with smaller datasets and rare words
TF-IDF (Module 5-7)Word Embeddings
Sparse, vocabulary-sized vectorsDense, fixed-size vectors regardless of vocabulary
No semantic relationships captured (proven directly, Module 7)Semantic relationships genuinely emerge from training
One vector position per word, arbitraryMeaningful position in continuous space
Word Embeddings (this module)Contextual Embeddings (Module 13)
ONE fixed vector per word, regardless of contextA DIFFERENT vector per word, depending on surrounding context

15. When to Use

Use word embeddings (or their modern successors) whenever semantic similarity, not just lexical overlap, matters — directly solving the failure demonstrated in Module 7. They remain a reasonable, lightweight choice for many classical downstream tasks (feeding fixed-size semantic features into classical ML, Module 6-style).

16. When Not to Use

Don’t rely on static word embeddings alone for tasks where word-sense disambiguation genuinely matters (Module 9 demonstrates this limitation directly) — contextual embeddings or full Transformer-based models (Module 13+) are needed when the same word’s different meanings must be distinguished.

17. Production Considerations

  • Pretrained embeddings (like publicly available Word2Vec or GloVe vectors) are commonly used as a starting point rather than training from scratch, saving substantial compute and data requirements.
  • Embedding dimensionality is a real design choice — larger dimensions can capture more nuanced relationships but cost more memory and compute; this exact trade-off reappears in modern embedding model selection for RAG systems.
  • Out-of-vocabulary words remain a genuine limitation — a word never seen during training has no embedding at all, a problem modern sub-word tokenization (Module 14) helps address.

18. Interview Questions

Beginner

Q: What is a word embedding?

Ans: A dense, learned numerical vector representing a word’s meaning, positioned so that semantically similar words end up with similar vectors — learned automatically from patterns in how words are used in context across a large text corpus, rather than manually assigned.

Intermediate

Q: What’s the difference between CBOW and Skip-gram?

Ans: CBOW (Continuous Bag of Words) predicts a target word given its surrounding context words. Skip-gram does the reverse — it predicts the surrounding context words given a target word. Both are ways of turning raw, unlabeled text into a supervised-style training task, and both produce embeddings as a side effect of learning to perform their respective prediction task well.

Advanced

Q: Why does training a model to predict context words (Skip-gram) end up producing embeddings where “king” and “queen” are close together, without anyone explicitly telling the model these words are related?

Ans: Words that tend to appear in similar contexts across a large corpus (both “king” and “queen” frequently appear near words like “throne,” “reign,” “royal”) push the model, during training, toward representing them similarly — because doing so genuinely helps the model predict their shared context words more accurately. This is an emergent consequence of optimizing the prediction objective, not a separately imposed rule; the semantic relationship “discovered” by the embeddings is a byproduct of the model getting better at a much simpler, mechanical task (predicting nearby words).

Scenario

Q: A team switches from TF-IDF to word embeddings for their document search system and sees noticeably improved results for queries using synonyms the original documents didn’t contain verbatim. Explain why, connecting to both this module and Module 7.

Ans: Module 7 proved directly that TF-IDF treats synonyms as completely unrelated vocabulary entries — a query for “automobiles” would never match a document only using “cars.” Word embeddings solve this specifically because semantically related words end up with similar embedding vectors (verified directly in this module, with “king” and “queen” scoring high cosine similarity) — so a query embedding for “automobiles” would likely be close, in the embedding space, to a document embedding built from text about “cars,” even without any exact word overlap at all.

AI Engineering

Q: Why is it inaccurate to say word embeddings “completely solved” the problems with classical NLP representations?

Ans: While word embeddings genuinely solve the synonym/semantic-similarity problem that Module 7 proved TF-IDF fundamentally lacks, they still assign exactly ONE fixed vector per word, regardless of context — the same limitation that causes “bank” (financial) and “bank” (riverbank) to share a single, averaged-together embedding. This remaining gap is precisely what Module 9 demonstrates directly, and is the direct motivation for contextual embeddings (Module 13) and, eventually, attention-based Transformer models that produce a genuinely different representation for the same word depending on its specific surrounding context.

19. Next Step

Next: Module 9 — The Context Problem — a direct, numerical demonstration of static embeddings’ “one vector per word” limitation, motivating the shift toward sequence models.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed