Embedding

The Vector article ended with a promise: vectors that capture meaning, positioned so that similar things land close together in space. That specific, learned kind of vector is called an embedding.

The simple definition

An embedding is a vector, produced by a trained model, positioned in space such that things with similar meaning end up close together, and things with different meaning end up far apart. The word “king” and the word “queen” would have embeddings positioned relatively near each other in this space, since they’re semantically related; “king” and “banana” would be positioned far apart. Crucially, this closeness isn’t programmed by hand — it’s learned automatically from data, exactly the way weights are learned through training, as covered throughout the Training Mechanics phase.

Why this is such a powerful idea

Recall from the Pattern article that raw data on its own doesn’t reveal meaning — a computer looking at the words “king” and “monarch” has no built-in sense that they’re related, unless something teaches it that relationship. Embeddings solve this by converting words (or sentences, images, or other data) into vectors where mathematical closeness — literally, distance in space — corresponds to real-world semantic similarity. This turns an abstract, fuzzy question like “are these two things related in meaning?” into a concrete, calculable one: “how close are these two vectors?” — a question ordinary math can actually answer.

flowchart LR
    A[Word: 'king'] --> D[Embedding Model]
    B[Word: 'queen'] --> D
    C[Word: 'banana'] --> D
    D --> E["'king' and 'queen' vectors land close together"]
    D --> F["'banana' vector lands far away"]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of organizing a massive library not alphabetically, but by topic similarity — books about cooking Italian food end up shelved near books about cooking French food, which end up somewhat near books about wine pairing, which end up further from, say, books about auto repair. Physical closeness on the shelf reflects conceptual closeness in subject matter.

Where this breaks down: A librarian organizes shelves using conscious judgment about topics and categories. An embedding model has no such conscious understanding — its positions emerge purely from statistical patterns in training data, discovered through the same gradient descent and backpropagation process covered in the Training Mechanics phase, with no human deciding in advance “king should be near queen.”

The famous, well-documented example: embedding arithmetic

This is worth knowing because it’s one of the most striking, widely cited illustrations of what embeddings actually capture. Word embedding research has repeatedly demonstrated that vector arithmetic on embeddings can reflect real relationships — the vector for “king,” minus the vector for “man,” plus the vector for “woman,” lands close to the vector for “queen.” This isn’t because anyone explicitly programmed that relationship — it emerged automatically from the embedding model noticing the statistical pattern of how these words are actually used in relation to each other across huge amounts of real text. It’s a striking, concrete demonstration that these vectors capture something genuinely meaningful about relationships between concepts, not just superficial word similarity.

How embeddings actually get produced

Embeddings aren’t hand-designed — they’re produced by an Embedding Model, the topic of the very next article, which is itself a trained neural network. During training, that network adjusts its weights, exactly as described in the Weights and Backpropagation articles, specifically to push related things closer together and unrelated things further apart, based on patterns observed across enormous amounts of training data.

A concrete example, layered

For a simple beginner example: an embedding model given the sentences “I love hiking in the mountains” and “I enjoy trekking through hills” would likely produce two vectors positioned quite close together, since the sentences express similar meaning using different words — capturing similarity that a simple, word-for-word text match would completely miss. For a production example: OpenAI’s text-embedding-3-small model converts any text into a 1,536-number vector specifically engineered so that semantically related pieces of text land near each other in that 1,536-dimensional space — a real, documented capability that underlies the search and retrieval systems covered later in this phase.

One document embedding versus many token embeddings

These are related but serve different purposes:

Representation	Shape idea	Typical use
Token embeddings	One vector for every token	Internal language-model processing and next-token prediction.
Document embedding	One vector representing a whole text chunk	Search, clustering, recommendations, and retrieval.

flowchart TB
    A[Sentence with 7 tokens] --> B[7 token vectors]
    A --> C[Embedding model]
    C --> D[1 sentence vector]

The document vector is not necessarily a plain average of token vectors. Its exact creation depends on how the embedding model was trained and how it pools information.

Real model example: OpenAI embedding dimensions

OpenAI introduced text-embedding-3-large with embeddings of up to 3,072 dimensions and support for shortening them through a dimensions parameter.

The trade-off is practical:

more dimensions → potentially more representational capacity
                → more storage and comparison work

fewer dimensions → smaller storage and faster operations
                 → possible retrieval-quality reduction

The model and chosen dimension must remain consistent for stored documents and incoming queries.

How GPT and Gemini use embeddings in two different ways

Embedding use	GPT/OpenAI example	Gemini/Google example
Inside generation	Token IDs select learned token representations processed by GPT Transformer layers.	Token and multimodal representations are processed by Gemini Transformer layers.
For retrieval	text-embedding-3-small or text-embedding-3-large can create vectors for search and RAG.	gemini-embedding-2 can create vectors for retrieval, similarity, and multimodal search.

Generation embedding: helps the language model process its current sequence
Retrieval embedding:   helps an external search system compare stored items

A chat model and an embedding model can belong to the same provider while remaining separate models with separate API operations.

Common misconception

A frequent beginner assumption: that an embedding is just a fancier way of counting which words two pieces of text have in common. This misses the whole point — embeddings are specifically valuable because they capture similarity in meaning even when the actual words used are completely different, as the hiking/trekking example above showed. A simple word-overlap comparison would see “I love hiking in the mountains” and “I enjoy trekking through hills” as almost entirely unrelated, since they share barely any words — an embedding-based comparison correctly recognizes them as closely related in meaning.

Where this fits in what comes next

You now understand what an embedding is and why closeness in this vector space corresponds to real similarity. The next article, Embedding Model, covers the actual trained system that produces these vectors in the first place — how it’s built, trained, and used in real applications.

In one sentence

An embedding is a learned vector positioned so that mathematical closeness reflects real semantic similarity, turning the abstract question of “how related are these two things in meaning” into a concrete, calculable distance — the foundational trick behind modern search, recommendation, and retrieval systems.

Related Terms

• Vector
• Embedding Model
• Pattern
• Weights
• Similarity Search