Vector

The Sequence article established that order matters, but tokens still aren’t numbers a model can directly do math on in a meaningful way — recall from the Input article that raw input has to become numbers, but a single token ID like 15339 doesn’t carry any rich information on its own. The actual numeric form nearly everything in AI gets converted into is a vector.

The simple definition

A vector is an ordered list of numbers, treated together as a single point or direction in a mathematical space. [3, 7, 2] is a vector with three numbers, describing a point in three-dimensional space. A vector with 1,536 numbers describes a point in 1,536-dimensional space — impossible to picture directly, but mathematically just as valid and usable as the three-dimensional case.

Why AI relies on vectors so heavily

This connects directly to the “everything becomes numbers” principle first raised in the Input article. A single number can capture one simple fact — a temperature, a price. But most interesting things in AI — a word’s meaning, an image’s content, a customer’s preferences — are too rich and multi-faceted to capture in one number. A vector solves this by using many numbers together, each one capturing a different aspect or dimension of the thing being represented. Recall from the Feature article that a house is represented by several features (square footage, bedrooms, age) — that entire collection, arranged in order, is itself a vector: [2000, 3, 15].

flowchart LR
    A[Square footage: 2000] --> D["Vector: [2000, 3, 15]"]
    B[Bedrooms: 3] --> D
    C[Age: 15] --> D

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of describing a location using GPS coordinates: latitude and longitude together, as a pair of two numbers, pin down one exact point on Earth’s surface. Neither number alone tells you where you are — you need both, together, in the right order.

Where this breaks down: GPS coordinates describe a physical, three-dimensional location a person can visit. Most vectors in AI describe points in spaces with far more than three dimensions — sometimes thousands — that have no physical, visitable location at all. The “space” is a pure mathematical construct, useful for calculation, not a place with any real-world geography.

What the individual numbers in a vector actually mean

This varies depending on what the vector represents, and it’s worth distinguishing two very different cases. In a simple feature vector, like the house example above, each position has a clear, human-readable meaning — position 1 is always square footage, position 2 is always bedrooms. In an embedding — covered fully in the very next article — each individual number typically has no clean, human-readable meaning on its own; the vector’s usefulness comes from the overall pattern of all its numbers together, echoing the “distributed meaning” point made in the Weights article about how large neural networks represent information.

Why vectors let you do useful math on meaning

This is the real payoff, and it’s worth stating directly: once something is represented as a vector, you can apply real mathematical operations to it — measuring how far apart two vectors are, or how similar their direction is. Two vectors that are “close together” in this mathematical space can represent two things that are genuinely similar in the real world — a foundational idea the rest of this phase builds on directly, once Embedding makes this concrete for text and other data.

A concrete example, layered

For a simple beginner example: a simple 2-number vector [75, 30] might represent “75°F temperature, 30% humidity” — two numbers, together, describing one weather condition as a single point in a two-dimensional space. For a production example: OpenAI’s text-embedding-3-small model, as documented in OpenAI’s own published specifications, converts any piece of text into a vector of exactly 1,536 numbers — every single word, sentence, or paragraph you might feed it becomes one point in a 1,536-dimensional mathematical space, ready for the similarity comparisons covered later in this phase.

A vector has dimensions, not named boxes of meaning

For a simple house model, a feature vector may be human-readable:

[area, bedrooms, age] = [2000, 3, 15]

For an embedding, the dimensions are learned and usually not individually nameable:

password-reset article → [0.12, -0.44, 0.81, ..., 0.07]

The useful information is distributed across the pattern of values. Dimension 17 rarely means exactly “password.”

Vector shape must match

You can compare two vectors coordinate by coordinate only when they use the same embedding space and dimension.

Model A vector: 1,536 dimensions
Model B vector: 3,072 dimensions

They cannot be directly compared merely by trimming or padding one unless the model explicitly supports a compatible shortening method. Even two models that output the same number of dimensions can organize meaning differently.

How vectors appear inside GPT and Gemini

Vectors occur throughout both model families:

flowchart LR
    A[Token ID] --> B[Embedding vector]
    B --> C[Contextual vector after layer 1]
    C --> D[Contextual vector after later layers]
    D --> E[Output-token score vector]

For every token position, hidden layers maintain a vector of learned values. Attention and feed-forward operations repeatedly transform these vectors. The final vector is projected into scores across the output vocabulary.

These internal token vectors are not automatically the same as the single document vectors returned by a dedicated OpenAI or Gemini embedding API. The model, training objective, and intended use are different.

Common misconception

A frequent beginner assumption: that a vector’s individual numbers must always correspond to something specific and nameable, the way “position 1 = square footage” does in the simple house example. As the “what the individual numbers mean” section explained, this is only true for simple, hand-built feature vectors — for the far more common case of embeddings, produced by trained neural networks, no single number in the vector typically means anything on its own, and trying to interpret one number in isolation is usually a dead end.

Where this fits in what comes next

You now understand vectors as the general mathematical form — an ordered list of numbers describing a point in space. The next article, Embedding, covers the specific, learned kind of vector that captures meaning — turning words, sentences, or images into vectors positioned so that similar things end up close together in that space.

In one sentence

A vector is simply an ordered list of numbers describing a point in a mathematical space, and it’s the foundational numeric form that lets AI systems apply real, useful math — like measuring similarity — to things as abstract as a word’s meaning.

Related Terms

• Feature
• Input
• Embedding
• Weights
• Sequence