Tokenization for Modern NLP

Begin with the central question

How can a model process a word never seen as a complete word?

Essential words

A subword is a reusable word piece. BPE, WordPiece, and SentencePiece are tokenization approaches. A merge rule joins frequent neighboring pieces.

What You Will Understand

Modern sub-word tokenization — BPE, WordPiece, SentencePiece conceptually — deep enough to understand exactly what real LLMs process as input. You’ll build Byte Pair Encoding entirely from scratch and watch it correctly tokenize a word it never saw during training.

text -> subword pieces -> token IDs -> model

Why Modern Models Use Word Pieces

Module 2 deliberately used simple whole-word tokenization, flagging that real systems work differently. Module 9’s out-of-vocabulary problem — what happens to a word never seen during training? — remained unresolved. Sub-word tokenization exists to solve both: it dramatically reduces vocabulary size requirements while gracefully handling words never seen during training, by breaking them into smaller, previously- seen pieces.

Building Unseen Words from Familiar Pieces

instead of needing a dictionary entry for every possible word (impossible — new words are coined constantly), sub-word tokenization builds a vocabulary of common word pieces — prefixes, suffixes, common character sequences — learned from how frequently they appear together in a large corpus. Nearly any word, even one never seen before, can be assembled from these smaller, familiar pieces.

Analogy: The Lego Brick Box Imagine you are a toy manufacturer trying to ship plastic models of vehicles to children:

Whole-Word Tokenization (Ready-made Models): You try to manufacture a complete, pre-assembled plastic replica for every model of car, truck, ship, plane, and helicopter that will ever exist (a massive, infinite vocabulary). The moment a child requests a newly invented vehicle type, like a “hovercraft”, your factory breaks down with an Out-Of-Vocabulary (OOV) error because you don’t have a mold for it.

Character Tokenization (Raw Plastic Pellets): You only ship tiny plastic pellets. Children can build anything, but they have to put together 10,000 tiny pellets just to build a small car. The instruction manuals (sequence lengths) are too long to read.

Sub-word Tokenization (The Lego Box): You ship a box of standard Lego bricks containing wheels, axles, pegs, plates, and blocks. Children can assemble almost any vehicle:

A standard car is built from “chassis” + “wheel” blocks.

The newly invented “hovercraft” is assembled by connecting “hover” + “craft” bricks, which you already manufacture.

Sub-word tokenization creates a fixed, manageable set of common blocks (reusable parts), ensuring you can build any word without running out of warehouse space.

📊 Visual Flowchart: BPE Tokenizer training merge loop

Here is how sub-word tokens are learned iteratively by scanning adjacency patterns:

graph TD
    Start["1. Initialize vocabulary with single characters:<br>[a, c, e, h, o, r, t, v, w]"] --> Scan["2. Scan corpus and count adjacent symbol pairs"]
    Scan --> FindMax["3. Find most frequent pair:<br>'c' and 'h' appear together 500 times"]
    FindMax --> Merge["4. Merge 'c' + 'h' into new token: 'ch'"]
    Merge --> UpdateVocab["5. Update Vocab:<br>[a, c, e, h, o, r, t, v, w, ch]"]

UpdateVocab --> CheckStop{"Vocabulary target size<br>reached?"}
    CheckStop -->|No| Scan
    CheckStop -->|Yes| End["6. Output fitted tokenizer vocabulary"]

4. Core Concept

Term	Definition
Character tokenization	Splitting text into individual characters — tiny vocabulary, very long sequences
Word tokenization	Splitting text into whole words (Module 2’s simplification) — large vocabulary, out-of-vocabulary problem
Sub-word tokenization	Splitting text into pieces smaller than whole words but larger than characters — balances vocabulary size and sequence length
BPE (Byte Pair Encoding)	Iteratively merges the most frequently co-occurring symbol pairs to build a sub-word vocabulary
WordPiece	Similar to BPE, but merges based on likelihood improvement rather than raw frequency (used by BERT)
SentencePiece	A tokenization framework that treats text as a raw stream (including spaces), commonly using BPE or a similar algorithm underneath

5. How It Works — Step by Step (BPE)

1. Start with EVERY word split into INDIVIDUAL CHARACTERS,
   plus an end-of-word marker
2. Count all ADJACENT SYMBOL PAIRS across the entire corpus
3. Find the MOST FREQUENT pair, and MERGE it into one new symbol
4. Repeat: recount pairs (now including the newly merged symbol),
   merge the next most frequent pair, and so on
5. Stop after a chosen number of merges (or when reaching a
   target vocabulary size)
6. The result: a vocabulary of increasingly larger sub-word units,
   built up from the most statistically common patterns

6. Mathematical Intuition

No formulas — BPE is a purely iterative, frequency-driven procedure. The key property worth internalizing: each merge step greedily selects whichever adjacent pair appears most often across the whole corpus, progressively building up common sub-word units (like “-est” or “-ing”) without any linguistic rules being hand-specified.

7. Simple Example

Given a training corpus dominated by words like “low,” “lower,” “newest,” and “widest,” BPE will likely learn a merged unit for “est” (appearing in both “newest” and “widest”) relatively early, since that character pair combination is common across the corpus. Once “est” exists as a learned unit, a genuinely new word like “lowest” — never seen during training — can still be tokenized as “low” + “est,” both already-known pieces.

8. Build It in Python

What the code will demonstrate

This small BPE trainer begins with characters and repeatedly merges the most frequent adjacent pair. The learned merge rules are then reused to split a new word that did not appear as a complete training word.

Watch the distinction between training and use: training learns merge rules from word frequencies, while tokenization applies those fixed rules without inventing new merges for each input.

from collections import defaultdict

def get_pair_counts(word_freqs):
    pairs = defaultdict(int)
    for word, freq in word_freqs.items():
        symbols = word.split()
        for i in range(len(symbols) - 1):
            pairs[(symbols[i], symbols[i+1])] += freq
    return pairs

def merge_pair(pair, word_freqs):
    new_word_freqs = {}
    bigram = " ".join(pair)
    replacement = "".join(pair)
    for word, freq in word_freqs.items():
        new_word_freqs[word.replace(bigram, replacement)] = freq
    return new_word_freqs

# Start: every word as individual characters + end-of-word marker
corpus = {"low": 5, "lower": 2, "newest": 6, "widest": 3}
word_freqs = {" ".join(list(word)) + " </w>": freq for word, freq in corpus.items()}

print("Initial character-level representation:")
for word, freq in word_freqs.items():
    print(f"  '{word}' (freq={freq})")

num_merges = 6
merges = []
for i in range(num_merges):
    pairs = get_pair_counts(word_freqs)
    if not pairs:
        break
    best_pair = max(pairs, key=pairs.get)
    word_freqs = merge_pair(best_pair, word_freqs)
    merges.append(best_pair)
    print(f"\nMerge {i+1}: {best_pair} (count={pairs[best_pair]})")

print("\nFinal learned merges (in order):", merges)

# --- Tokenize a genuinely NEW, never-seen word ---
def apply_bpe(word, merges):
    symbols = list(word) + ["</w>"]
    word_str = " ".join(symbols)
    for pair in merges:
        bigram = " ".join(pair)
        replacement = "".join(pair)
        word_str = word_str.replace(bigram, replacement)
    return word_str.split()

new_word = "lowest"
tokens = apply_bpe(new_word, merges)
print(f"\nTokenizing NEVER-SEEN word '{new_word}':", tokens)

Expected Output:

Initial character-level representation:
  'l o w </w>' (freq=5)
  'l o w e r </w>' (freq=2)
  'n e w e s t </w>' (freq=6)
  'w i d e s t </w>' (freq=3)

Merge 1: ('e', 's') (count=9)

Merge 2: ('es', 't') (count=9)

Merge 3: ('est', '</w>') (count=9)

Merge 4: ('l', 'o') (count=7)

Merge 5: ('lo', 'w') (count=7)

Merge 6: ('n', 'e') (count=6)

Final learned merges (in order): [('e', 's'), ('es', 't'), ('est', '</w>'), ('l', 'o'), ('lo', 'w'), ('n', 'e')]

Tokenizing NEVER-SEEN word 'lowest': ['low', 'est</w>']

9. How It Works

BPE’s first three merges build up “est” step by step (e+s → es, then es+t → est, then est+</w> → est</w>) — a genuinely common pattern across “newest” and “widest” in this corpus, discovered purely from co-occurrence frequency, no linguistic rules hand-coded.
The word “lowest” — which never appeared anywhere in the training corpus — gets correctly tokenized into ['low', 'est</w>'], both of which were learned as units from the training data (“low” from merges 4-5, “est” from merges 1-3). This is the concrete, verified mechanism behind sub-word tokenization’s graceful handling of unseen words — precisely the fix for Module 9’s out-of-vocabulary problem.

10. Real-World Example

Real tokenizers (used by GPT-family models, BERT, and others) apply essentially this same BPE-style algorithm, but trained on vocabularies of tens of thousands of merges across enormous text corpora. A rare or technical word like “immunohistochemistry” would likely be split into several recognizable sub-word pieces (“immuno,” “histo,” “chemistry,” or similar), rather than failing outright or requiring that exact word to have appeared during training.

11. How Is This Used in Modern AI?

🤖 How Is This Used in Modern AI?

Every real LLM you’ll interact with uses sub-word tokenization, not whole-word tokenization — this is why token counts don’t match word counts, why API pricing is based on tokens specifically, and why unusual or technical terms sometimes get split into multiple tokens you might not expect.

Consideration	Why tokenization matters
Model input	LLMs process token IDs, not raw words — sub-word tokenization is the actual first processing step
Context windows	Measured in tokens, not words — a model’s context limit is a token count
API cost	Billed per token, not per word or character
Latency	More tokens generally means more processing time

🧠 The critical takeaway, restated directly: LLMs do not directly process words. They process token IDs — produced by exactly this kind of sub-word tokenization — which are then converted into vectors (embeddings, covered fully in the dedicated Transformers course).

Real systems you can recognize

OpenAI publishes tiktoken for working with OpenAI model encodings. Gemini documents token counting and explains that a token may be a character, a whole word, or part of a word; see Gemini token documentation.

Therefore, the same visible sentence can have different token IDs and token counts under different model vocabularies. Token IDs are stable only within the same tokenizer vocabulary and version—not across GPT, Gemini, BERT, or arbitrary Hugging Face tokenizers.

12. How Is This Used in Agentic AI?

Direct relevance to Agentic AI: High, practically. Every piece of context an agent assembles — system prompts, conversation history, retrieved documents, tool definitions — gets tokenized via this exact mechanism before hitting a model’s context window limit. Understanding sub-word tokenization directly explains why token-counting tools (not simple word counts) are necessary for accurately managing context budget and cost in real agent applications.

13. Common Mistakes / Misunderstandings

⚠️ Mistake: assuming one token always equals one word. As verified directly, “lowest” became two tokens (low, est</w>) — sub-word tokenization frequently splits words, especially longer or rarer ones.

⚠️ Mistake: believing sub-word tokenization is just character-level tokenization with extra steps. It specifically balances between pure character-level (tiny vocabulary, very long sequences) and pure word-level (huge vocabulary, out-of-vocabulary problem) — the merges discover a genuinely useful middle ground.

⚠️ Mistake: assuming all tokenizers produce the same tokens for the same text. Different models train their own tokenizers on their own data with their own merge counts — token counts for identical text genuinely vary between model providers.

14. Important Distinctions

Word Tokenization (Module 2)	Sub-word Tokenization (this module)
Out-of-vocabulary words fail entirely	Unseen words split into known pieces — verified directly
Simple, but doesn’t scale well to real vocabularies	The actual standard for modern NLP/LLM systems

Token	Word
The actual unit of model processing	One possible kind of token — not all tokens are full words

BPE	WordPiece
Merges based on raw frequency	Merges based on likelihood improvement (used by BERT)

15. When to Use

Sub-word tokenization is the standard, well-justified choice for any modern NLP or LLM system — it’s what every production tokenizer actually uses, for the exact out-of-vocabulary and vocabulary-size reasons demonstrated in this module.

16. When Not to Use

Whole-word or character-level tokenization remain reasonable for specific, narrow educational or research contexts, but aren’t the practical choice for any real, general-purpose modern system.

17. Production Considerations

Token counting tools matter directly for cost estimation — since the same text produces different token counts across different models’ tokenizers, accurate cost/context estimation requires using the specific target model’s actual tokenizer, not a generic word count.
Tokenization is fixed once a model is trained — you cannot meaningfully change a deployed model’s tokenizer without retraining, exactly like the vocabulary/positional encoding constraints discussed in the Transformers course.
Rare or technical vocabulary gets split into more tokens — this has real, practical cost implications for domain-specific applications (e.g., medical or legal text) using unusual terminology.

18. Interview Questions

Beginner

Q: Why do modern NLP systems use sub-word tokenization instead of whole-word tokenization?

Ans: Sub-word tokenization solves two problems whole-word tokenization has: it dramatically reduces vocabulary size requirements (since common sub-word pieces get reused across many different words), and it gracefully handles words never seen during training by breaking them into smaller, previously-learned pieces, rather than failing outright.

Intermediate

Q: How does BPE decide which characters or sub-word pieces to merge together?

Ans: BPE works iteratively: starting from individual characters, it repeatedly finds the most frequently occurring adjacent pair of symbols across the entire training corpus and merges them into a single new symbol. This process repeats for a set number of merges, progressively building up increasingly larger, statistically common sub-word units — entirely from co-occurrence frequency, without any hand-coded linguistic rules.

Advanced

Q: Explain, using this module’s verified result, exactly how BPE handles a word that never appeared in its training data.

Ans: BPE applies its learned sequence of merges (discovered during training) to the new word’s individual characters, in the same order they were learned. If the new word happens to contain character sequences that match previously-learned merged units, those units get applied. This was demonstrated directly: “lowest” was never in the training corpus, but because “low” and “est” (via several intermediate merges) had both been separately learned from other training words (“low,” “lower,” “newest,” “widest”), “lowest” was correctly tokenized into these two known pieces, rather than failing or requiring a special “unknown word” fallback.

Scenario

Q: A team is building a system for a specialized medical domain and notices their chosen LLM’s tokenizer splits many medical terms into many more tokens than everyday English words. Why does this happen, and what practical impact does it have?

Ans: The tokenizer’s merge vocabulary was learned from its training corpus’s statistics — if that training corpus was dominated by general-purpose text rather than medical literature, specialized medical terminology likely didn’t appear frequently enough to earn its own dedicated, larger merged tokens, so it gets broken down into more, smaller pieces. The practical impact is real: more tokens per medical document means higher API costs and faster consumption of the available context window for the same amount of actual content, compared to equivalent general-purpose text.

AI Engineering

Q: Why is it inaccurate to say “LLMs read words”?

Ans: As demonstrated directly throughout this module, LLMs process token IDs produced by sub-word tokenization — many individual tokens don’t correspond to complete words at all (like “est” from this module’s example), and a single word can be split across multiple tokens. The precise, accurate description is: raw text is tokenized into sub-word units, converted to token IDs, and only then converted into the numerical vectors (embeddings, covered fully in the Transformers course) the model actually processes — “words” are a convenient human framing, not the model’s actual unit of computation.

19. Next Step

Next: Module 15 — NLP Tasks — a practical taxonomy of understanding, retrieval, and generation tasks, connecting everything covered so far to concrete, real-world applications.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026