Dense Vector

The Embedding Model article described embeddings as vectors of, say, 1,536 numbers. It’s worth asking directly: what do those 1,536 numbers actually look like — mostly zeros, or mostly real, varied values? For a typical embedding, the answer is the latter — it’s a dense vector.

The simple definition

A dense vector is a vector where most or all of the positions hold non-zero values. If you printed out a 1,536-number embedding, you’d see a long list of ordinary-looking decimal numbers — 0.23, -0.41, 0.08, 0.67... — almost none of them exactly zero. Every position is doing real, meaningful work, contributing some small piece to the overall pattern the vector represents.

Why most modern embeddings end up dense

Recall from the Embedding article that an embedding’s meaning comes from the overall pattern across all its numbers together, not from any single number in isolation — echoing the distributed-meaning idea from the Weights article. A neural network producing this kind of vector, through the weighted-sum-plus-activation calculations described in the Node article, naturally tends to produce output where most positions carry some nonzero value, since each one is the result of combining many different weighted inputs. This is a direct, structural consequence of how embedding models are built and trained, not an arbitrary choice.

flowchart LR
    A[Dense Vector: 0.23, -0.41, 0.08, 0.67, -0.12, ...] --> B[Nearly every position holds real information]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a fully detailed weather report describing today’s conditions — temperature, humidity, wind speed, air pressure, UV index, pollen count — every single field filled in with a real, specific reading. Nothing is left blank; every measurement contributes some real information to the overall picture.

Where this breaks down: A weather report’s fields each have an obvious, human-readable label. A dense vector’s individual positions, as the Vector article explained, typically have no such clean, nameable meaning on their own — the “fully filled in” quality is genuine, but what each position actually represents is usually not directly interpretable, unlike a labeled weather field.

Why density matters practically: storage and computation

This has real, concrete consequences worth naming. A dense vector, having a real value in every position, takes a predictable, fixed amount of storage — a 1,536-dimensional dense vector always needs space for 1,536 numbers, regardless of what it represents. Comparing two dense vectors — measuring how similar they are, a topic covered in detail later in this phase — requires touching every single one of those numbers in the calculation, since none of them can be safely skipped as “empty.” This is a direct contrast with the article that follows, Sparse Vector, where most positions genuinely are empty, and calculations can skip over them entirely.

A concrete example, layered

For a simple beginner example: a tiny 4-number dense vector representing a fruit’s characteristics — [0.8, 0.3, 0.9, 0.1] for sweetness, tartness, juiciness, and crunchiness — has a real, specific value in every single position, none of them zero, each contributing to the overall description. For a production example: OpenAI’s text-embedding-3-small model, as documented in its API specification, produces a dense 1,536-number vector for any input text, with essentially every one of those 1,536 positions holding a genuine, non-zero decimal value — the standard, expected output format for this kind of modern neural embedding model.

Dense does not mean every value is large

Dense vector: [0.012, -0.403, 0.008, 0.771]

Several values may be small, but they are explicitly stored and participate in calculations. “Dense” contrasts with a sparse representation in which most coordinates are exactly zero and only non-zero entries need to be stored.

Estimate storage with real dimensions

If every component uses a 32-bit float, each value needs 4 bytes. For one 3,072-dimensional vector:

3,072 × 4 bytes = 12,288 bytes ≈ 12 KB

For one million vectors, the raw values alone require roughly:

12,288 × 1,000,000 bytes ≈ 12.3 GB

Metadata, IDs, and ANN indexes require additional space. This explains why dimension shortening, lower-precision storage, compression, and index selection matter in production.

How GPT and Gemini use dense vectors

GPT and Gemini Transformer layers represent every token position using a dense hidden vector. Dedicated OpenAI and Gemini embedding models also return dense vectors for whole inputs used in search.

Inside chat model: many dense token vectors change at every layer
Embedding API:     one or more dense output vectors returned to the application

Google’s Gemini embeddings documentation supports selected output sizes up to 3,072 dimensions. OpenAI’s text-embedding-3-large also supports up to 3,072 dimensions. Similar dimension counts do not create compatibility: vectors from different models occupy different learned spaces.

Common misconception

A frequent beginner assumption: that “dense” describes how much meaning a vector carries, as in “a densely meaningful representation.” That’s not quite the technical sense intended here — “dense” specifically describes the mathematical structure of the vector (most positions are non-zero), a distinct, more precise idea from how much semantic richness the vector happens to capture. A vector can be structurally dense while still being a poor, low-quality embedding if the model producing it wasn’t trained well.

Where this fits in what comes next

You now understand the typical shape of a modern embedding. The next article, Sparse Vector, covers the opposite structural pattern — a vector where the vast majority of positions genuinely are zero — and why that different structure exists and remains useful for certain tasks.

In one sentence

A dense vector is one where nearly every position holds a real, non-zero value, the natural output shape of most modern neural embedding models, and its “every position matters” structure is exactly what makes similarity comparisons between dense vectors computationally involve every single dimension.

Related Terms

• Vector
• Embedding
• Embedding Model
• Sparse Vector
• Cosine Similarity