Tokenization

The Token article showed the result — “unbelievable” split into “un” + “believ” + “able.” This article covers the process that actually produces that split: tokenization.

The simple definition

Tokenization is the process of converting raw text into a sequence of tokens, using a fixed, predetermined vocabulary of possible pieces. It’s the very first step any text has to go through before it can become the kind of numeric Input a model can actually process — recall from the Input article that raw input always has to be converted into numbers before a model can use it, and for text, tokenization is that conversion’s first stage.

Why tokenization isn’t just “split on spaces and punctuation”

A naive approach — split text wherever there’s a space — seems simple, but breaks down fast. It would treat “run,” “runs,” and “running” as three completely unrelated tokens, missing the obvious shared root. It would create a separate token for every possible word form and misspelling, resulting in an enormous, unwieldy vocabulary. And it wouldn’t handle languages that don’t separate words with spaces, or code, or unusual text like URLs and hashtags. Real tokenizers instead use a data-driven, statistical approach that learns which chunks of text are common enough to deserve their own token, and which rarer combinations should be built from smaller, shared pieces.

How the actual algorithm works: Byte-Pair Encoding

Most modern language models, including OpenAI’s, use a technique called Byte-Pair Encoding (BPE) to build their tokenizer’s vocabulary. The process starts by treating text as individual characters, then repeatedly finds the most frequently occurring pair of adjacent pieces across a huge amount of training text, and merges that pair into a single new token. This merging step repeats over and over — each round creating slightly larger, more common chunks — until the vocabulary reaches a target size (roughly 100,000 tokens for OpenAI’s cl100k_base encoding, as covered in the Token article). The result is a vocabulary where extremely common sequences (whole common words, frequent word pieces) get their own single token, and everything else gets built by combining smaller, more frequent pieces.

flowchart LR
    A[Start: individual characters] --> B[Find most frequent adjacent pair]
    B --> C[Merge into one new token]
    C --> D{Vocabulary at target size?}
    D -->|No| B
    D -->|Yes| E[Final fixed vocabulary]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of building a shorthand system for court stenography by studying years of past transcripts and noticing which specific phrases come up constantly — “objection, your honor,” “the witness stated” — and assigning those common phrases their own single shorthand symbol, while rarer phrases still have to be spelled out using individual shorthand letters combined together.

Where this breaks down: A stenographer designs shorthand symbols using human judgment about what’s useful. BPE’s merging process is entirely mechanical and statistical — it merges whichever pair of pieces happens to appear most frequently in the training text, with no human judgment about meaning or usefulness involved in any individual merge decision, just raw frequency counting repeated many times.

Once built, the vocabulary is fixed

This is a genuinely important, practical detail: tokenization’s vocabulary-building process (the BPE merging described above) happens once, ahead of time, using a large sample of text — it isn’t redone for every new sentence a model processes. Once that fixed vocabulary is set, actually tokenizing new text becomes a much faster, simpler lookup process: break the new text down according to the existing vocabulary’s rules, no fresh statistical analysis needed. This is exactly why OpenAI’s tiktoken library is documented as running several times faster than comparable tokenizers — the hard statistical work was already done once, in advance.

Vocabulary creation versus runtime tokenization

These are two different moments:

Moment	What happens?
Tokenizer creation	The algorithm studies a corpus, chooses useful token pieces, and assigns every vocabulary entry a permanent ID.
Runtime tokenization	Saved rules split new text and look up the already-assigned IDs.

Runtime tokenization is deterministic under the same tokenizer configuration: identical input text normally produces the same token-ID sequence. A model can still generate different answers because token sampling happens later; generation randomness does not mean the input tokenizer changed its IDs.

Token ID and embedding row are connected

Suppose a vocabulary has 50,000 entries and each token embedding contains 768 values:

embedding table shape = [50,000, 768]

If tokenization returns ID 917, the embedding lookup selects row 917:

flowchart LR
    A[Text piece] --> B[Token ID 917]
    B --> C[Embedding-table row 917]
    C --> D[Vector with 768 learned values]

The token ID is not the embedding. It selects the embedding. Training changes the 768 learned values in that row; the row’s token ID stays fixed.

A concrete example, layered

For a simple beginner example: tokenizing “the cats ran” with a vocabulary that includes “the,” “cat,” “s,” and “ran” as existing tokens would likely produce four tokens: “the,” “cat,” “s,” “ran” — “cats” gets split because “cats” itself wasn’t common enough to earn its own dedicated token, even though “cat” was. For a production example: OpenAI’s real tiktoken tokenizer, using the cl100k_base encoding built through exactly this BPE process, converts “hello world” into the two tokens represented as [15339, 1917], as confirmed directly in OpenAI’s published tokenizer library — a real, verifiable, working example of the entire pipeline this article has described, from raw text to final token IDs.

Tokenization and embedding are separate steps

flowchart LR
    A[Raw text] --> B[Tokenizer]
    B --> C[Token IDs]
    C --> D[Embedding lookup or embedding model]
    D --> E[Vectors]

• Tokenization decides the pieces and produces IDs.
• Embedding produces numerical representations used by a neural network or search system.

A tokenizer does not calculate semantic similarity. An embedding model cannot directly process arbitrary text until its expected tokenizer has converted that text into tokens.

Why different models can count the same sentence differently

Tokenizers have different vocabularies and merge rules. Therefore:

• The same text can use a different number of tokens in GPT, Gemini, or another model.
• A common word may be one token while a rare name becomes several tokens.
• Languages and writing systems can have different token efficiency.
• Changing a tokenizer changes the token IDs, so model weights trained for one vocabulary cannot simply assume another vocabulary’s IDs mean the same thing.

How GPT and Gemini carry out tokenization

OpenAI’s published GPT-2 report describes a byte-level BPE vocabulary of 50,257 tokens. GPT-2 tokenization turns text into IDs from that vocabulary before its Transformer processes them.

Gemini also converts input into token sequences before Transformer processing, while multimodal Gemini models additionally encode images, audio, and video into numerical representations. The Gemini 1.0 report describes text interleaved with those modalities.

GPT text:    text → GPT tokenizer → GPT token IDs
Gemini text: text → Gemini tokenizer → Gemini token IDs
Gemini media: image/audio/video → modality encoder → model representations

Do not assume their vocabularies, token counts, or token IDs are interchangeable.

Common misconception

A frequent beginner assumption: that tokenization is a simple, universal process — that “hello world” would tokenize the same way no matter which AI system you’re using. This isn’t true. Different models use different tokenizers, built from different training text and different target vocabulary sizes, so the exact same sentence can split into a different number of tokens, in different places, depending on which model’s tokenizer is doing the splitting — GPT-4’s cl100k_base and GPT-4o’s o200k_base, both from the same company, don’t tokenize identically, as covered in the Token article.

Where this fits in what comes next

You now understand how text actually gets converted into the tokens a model processes. The next article, Sequence, covers what happens once you have a whole string of these tokens in order — how a model treats an ordered run of tokens as a single, connected unit of meaning, rather than a disconnected pile of individual pieces.

In one sentence

Tokenization is the process of splitting text into tokens using a fixed vocabulary built in advance through statistical analysis of real text — most commonly via Byte-Pair Encoding — and it’s the essential first conversion step that turns raw language into something a model can actually compute on.

Related Terms

• Token
• Sequence
• Input
• Vector
• Embedding