Sequence

The Tokenization article showed how “hello world” becomes the two tokens [15339, 1917]. But those two numbers aren’t just a pile — their order matters enormously. That ordered arrangement has a name: a sequence.

The simple definition

A sequence is an ordered list of tokens, where the position of each token relative to the others carries real, essential meaning. “The dog bit the man” and “the man bit the dog” use the exact same five tokens (roughly) — but in a completely different order, producing a completely different meaning. A sequence isn’t just “a bunch of tokens”; it’s tokens plus their specific arrangement, and that arrangement is often just as important as which tokens are present at all.

Why order can’t just be thrown away

This might seem obvious, but it’s worth being precise about why it matters technically. Recall from the Input article that a model processes numbers, and recall from the Prediction article that a language model predicts the next token based on everything that came before it in the conversation. If a model only knew which tokens were present, without knowing their order, it would have no way to distinguish “the dog bit the man” from “the man bit the dog” — both contain identical tokens. Preserving sequence order is what lets a model track grammar, cause and effect, and who’s doing what to whom — genuinely essential information that a simple, unordered collection of tokens would completely lose.

flowchart LR
    A["Sequence: [The, dog, bit, the, man]"] --> B[Meaning: dog is the attacker]
    C["Same tokens, different order: [The, man, bit, the, dog]"] --> D[Meaning: man is the attacker]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a recipe’s list of steps versus the same steps shuffled into random order. “Preheat oven, mix batter, pour into pan, bake 30 minutes” produces a cake. The exact same four steps, reordered as “bake 30 minutes, preheat oven, pour into pan, mix batter,” produces a mess — the individual steps didn’t change at all, but their order completely changes the outcome.

Where this breaks down: A recipe’s steps are followed by a person applying real-world understanding of cause and effect — you can’t bake batter that hasn’t been mixed yet. A model doesn’t understand causality in that sense; it processes sequence order purely because its architecture — specifically the attention mechanism referenced in the Algorithm article — was mathematically built to track relative token positions, not because it comprehends why order matters the way a baker does.

Sequence length and the context window

This connects directly to a concept referenced throughout earlier phases of this glossary: a model’s context window is simply the maximum sequence length it can process at once, measured in tokens. Recall from the Inference article’s discussion of prefill and decode: the entire conversation history, plus your latest message, forms one growing sequence that the model processes together. If that sequence grows longer than the model’s context window allows, older parts of the conversation typically have to be dropped or summarized, since the model physically cannot process a sequence longer than its architecture supports.

A concrete example, layered

For a simple beginner example: the sentence “Cats chase mice” tokenizes into a short sequence where the specific order tells you cats are doing the chasing; simply having the tokens “cats,” “chase,” and “mice” present, without their order, would leave it genuinely ambiguous which animal is chasing which. For a production example: GPT-4 is documented as supporting context windows of up to 128,000 tokens in some of its variants — meaning a single sequence fed into the model can be up to 128,000 tokens long, roughly equivalent to a very long book chapter, with the model tracking the order and relationship of every single one of those tokens relative to every other token in that same sequence.

A sequence is more than a bag of tokens

Compare:

Sequence A: [dog, bites, person]
Sequence B: [person, bites, dog]

Both contain the same three tokens, but their order changes who did what. A language model therefore needs token identity and position information.

flowchart LR
    A[Token ID] --> C[Token representation]
    B[Position 1, 2, 3...] --> C
    C --> D[Attention processes relationships across sequence]

Sequence length, dimensions, and batch shape

Suppose a batch contains 8 sequences. Each sequence is padded or truncated to 128 tokens, and each token has a 768-value hidden vector:

token IDs shape       = [8, 128]
hidden vectors shape  = [8, 128, 768]
                        batch, tokens, values per token

These three numbers describe different things. A sequence of 128 tokens does not mean each token vector has 128 dimensions.

How GPT and Gemini process sequences

GPT-style models process an ordered token sequence using causal attention: a token can use earlier context when predicting what comes next. The model generates one new token, appends it, and repeats.

The Gemini 1.5 report describes long-context processing across text, documents, audio, and video. Those inputs are represented as ordered sequences so the model can relate information appearing at different positions.

flowchart LR
    A[Existing sequence] --> B[Transformer blocks]
    B --> C[Next-token probabilities]
    C --> D[Append selected token]
    D --> A

A longer supported context does not mean every detail will always be used perfectly. Retrieval and reasoning quality across long sequences must still be evaluated.

Common misconception

A frequent beginner assumption: that a “sequence” is just a fancy word for “sentence” or “a bunch of text.” In practice, the term specifically emphasizes the ordering itself as the carrier of meaning — a sequence could be a single word, a full conversation, or a string of code, and what makes it a sequence, technically, isn’t its length or content, but the fact that its elements have a meaningful, fixed order that a model must track and use.

Where this fits in what comes next

You now understand tokens, how they’re produced, and how their order matters as a sequence. The next article, Vector, covers the actual numeric form each token — and eventually each whole sequence — gets converted into, before any of the weighted calculations from the Neural Networks phase can happen.

In one sentence

A sequence is an ordered run of tokens where position itself carries meaning, and a model’s ability to track that order — up to the limits of its context window — is what lets it distinguish genuinely different meanings built from the exact same underlying tokens.

Related Terms

• Token
• Tokenization
• Prediction
• Inference
• Vector