Tokens and Tokenization | TechByteByByte

Before you continue: three tools for this module

Token: a piece of text handled as one vocabulary item.
Token ID: the integer assigned to that item.
Tokenizer: the algorithm mapping text to and from tokens.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Tokens and Tokenization solve inside a real language-model system?

Keep that central question about Tokens and Tokenization in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

text → token pieces → token IDs → model vectors

1. What You Will Learn

Learning outcomes

Explain why models process tokens instead of raw words or characters alone.
Trace text through token pieces, token IDs, and model input vectors.
Understand why the same visible word can tokenize differently across contexts or tokenizers.
Connect token counts to context limits, latency, and API cost.

In one sentence

💡 Big picture

Before an LLM can read a sentence, a tokenizer breaks it into pieces and gives each piece a number called a token ID.

2. Why This Module Exists

The problem this module solves

A model cannot work directly with the letters we see; it needs numbers.
Token choices affect context space, speed, cost, and how unfamiliar words are handled.

3. Intuition

you already know a tokenizer breaks text into sub-word pieces using a learned vocabulary of merges (NLP course). Here’s the piece that matters specifically for building with LLMs: that learned vocabulary is different for every model. The same sentence sent to two different LLM providers can — and does — produce a different number of tokens, because each provider trained their own tokenizer on their own data.

Analogy: The Lego Building Blocks Vocabulary Think of sub-word tokenization vocabularies as custom packs of Lego pieces:

The Setup: Two kids, Alice (LLM A) and Bob (LLM B), are building models.

Alice’s Lego Pack (Vocab A): Contains long, pre-assembled blocks: “un”, “believ”, “able”. She builds the word “unbelievable” using exactly 3 pieces.

Bob’s Lego Pack (Vocab B): Contains only short, basic brick shapes: “u”, “n”, “b”, “e”, “l”, “ie”, “v”, “a”, “b”, “l”, “e”. He needs 12 pieces to build the same word.

When you send the word “unbelievable” to both models, Alice bills you for 3 tokens, while Bob bills you for 12 tokens. Neither child is “wrong” — their factory-supplied boxes were simply packed with different brick configurations.

📊 Visual Flowchart: Token Sequence Padding and Special Tokens Layout

Here is how text strings are converted into padded token arrays with boundary flags:

graph TD
    classDef boundary fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;
    classDef content fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
    classDef pad fill:#7f8c8d,stroke:#333,stroke-width:1px,color:#fff;

    Text["Raw Text: 'un real'"] --> Tokenize["1. Decompose to Subwords: ['un', 'real']"]
    Tokenize --> IDs["2. Map to Vocab Indexes: [0, 1]"]

    subgraph PaddedArray ["3. Form Batch Array (max_len = 6)"]
        BOS["Index 0: '<BOS>'<br>(ID: 12)"]:::boundary
        Token1["Index 1: 'un'<br>(ID: 0)"]:::content
        Token2["Index 2: 'real'<br>(ID: 1)"]:::content
        EOS["Index 3: '<EOS>'<br>(ID: 13)"]:::boundary
        PAD1["Index 4: '<PAD>'<br>(ID: 14)"]:::pad
        PAD2["Index 5: '<PAD>'<br>(ID: 14)"]:::pad
    end

    IDs --> BOS
    IDs --> Token1
    IDs --> Token2
    IDs --> EOS

4. Core Concept — The Full Pipeline

Text
 ↓
Tokenizer            (a fixed, pre-trained sub-word tokenizer —
                      BPE, WordPiece, or SentencePiece, NLP course)
 ↓
Tokens
 ↓
Token IDs

Term	Definition
Token	The actual unit of processing — often a sub-word piece, not a whole word (NLP course Module 14)
Vocabulary	The fixed set of tokens a specific model’s tokenizer recognizes
Special tokens	Reserved tokens with structural meaning, not literal text
`<BOS>`	Beginning-of-sequence marker
`<EOS>`	End-of-sequence marker — signals the model to stop generating
`<PAD>`	Padding token, used to make sequences in a batch the same length
`<UNK>`	Unknown token (rarely needed with sub-word tokenization, since unseen words decompose into known pieces)

5. Why LLMs Don’t Directly Consume Words

You already proved this directly in the NLP course: whole-word tokenization fails on any word never seen during training, and requires an impractically large vocabulary to cover a language’s full word space.

Sub-word tokenization (BPE, WordPiece, SentencePiece) solves both — a fixed, manageable vocabulary (typically 30,000-100,000+ tokens for real LLMs) that can represent virtually any input by decomposing unfamiliar words into familiar pieces.

6. How It Works — Step by Step

1. A tokenizer is TRAINED ONCE, on a large corpus, learning a
   fixed set of merge rules (BPE, NLP course Module 14) or an
   equivalent vocabulary-building procedure (WordPiece,
   SentencePiece)
2. This trained tokenizer is then FIXED and shipped with the
   model -- every future input is tokenized using these SAME
   learned merges/vocabulary, never retrained
3. Raw text is split into tokens using this fixed vocabulary
4. Each token is mapped to its TOKEN ID (an integer index)
5. SPECIAL TOKENS are added: BOS at the start, EOS at the end,
   PAD tokens if the sequence needs to reach a fixed batch length
6. The result -- a sequence of integers -- is what actually
   enters the model's embedding layer (Module 4)

7. Mathematical Intuition

Read the mathematics as a story

text → token pieces → token IDs → model vectors

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

No new math beyond BPE’s frequency-driven merge procedure (NLP course). The one thing worth being precise about here: two tokenizers trained on different corpora will learn different merge sequences, and therefore produce genuinely different token counts for identical input text — verified directly below.

8. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

If Tokenizer A was trained on a corpus rich in words like “unbelievable,” “unreal,” and “believe,” it likely learns useful merges for common sub-word patterns like “un-” and “-able.” If Tokenizer B was trained on a corpus emphasizing different vocabulary (“friend,” “friendly,” “happy”), its learned merges will differ — meaning the exact same input sentence can be split into a different number of pieces by each.

9. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Tokens and Tokenization.
# Follow the inputs, transformations, and output in order.
from collections import defaultdict

def get_pair_counts(word_freqs):
    pairs = defaultdict(int)
    for word, freq in word_freqs.items():
        symbols = word.split()
        for i in range(len(symbols) - 1):
            pairs[(symbols[i], symbols[i+1])] += freq
    return pairs

def merge_pair(pair, word_freqs):
    bigram, replacement = " ".join(pair), "".join(pair)
    return {w.replace(bigram, replacement): f for w, f in word_freqs.items()}

def train_bpe(corpus, num_merges):
    word_freqs = {" ".join(list(w)) + " </w>": f for w, f in corpus.items()}
    merges = []
    for _ in range(num_merges):
        pairs = get_pair_counts(word_freqs)
        if not pairs:
            break
        best = max(pairs, key=pairs.get)
        word_freqs = merge_pair(best, word_freqs)
        merges.append(best)
    return merges

def apply_bpe(word, merges):
    word_str = " ".join(list(word) + ["</w>"])
    for pair in merges:
        word_str = word_str.replace(" ".join(pair), "".join(pair))
    return word_str.split()

def tokenize_sentence(sentence, merges):
    tokens = []
    for word in sentence.split():
        tokens.extend(apply_bpe(word, merges))
    return tokens

# Simulate TWO DIFFERENT tokenizers -- e.g. two different LLM providers --
# trained on DIFFERENT corpora.
corpus_a = {"unbelievable": 5, "unreal": 3, "believe": 4, "real": 6, "unhappy": 2}
corpus_b = {"unbelievable": 1, "friend": 8, "friendly": 4, "happy": 6, "unhappy": 3}

merges_a = train_bpe(corpus_a, num_merges=10)
merges_b = train_bpe(corpus_b, num_merges=10)

sentence = "unbelievable unhappy"
tokens_a = tokenize_sentence(sentence, merges_a)
tokens_b = tokenize_sentence(sentence, merges_b)

print(f"Sentence: '{sentence}'")
print(f"\nTokenizer A: {tokens_a}")
print(f"  Token count: {len(tokens_a)}")
print(f"\nTokenizer B: {tokens_b}")
print(f"  Token count: {len(tokens_b)}")

# --- Special tokens ---
special_tokens = {"<BOS>": 12, "<EOS>": 13, "<PAD>": 14, "<UNK>": 15}

def encode_with_special_tokens(tokens, token_to_id, max_len=8):
    ids = [special_tokens["<BOS>"]] + [token_to_id.get(t, special_tokens["<UNK>"]) for t in tokens] + [special_tokens["<EOS>"]]
    while len(ids) < max_len:
        ids.append(special_tokens["<PAD>"])
    return ids[:max_len]

short_tokens = ["un", "real"]
token_to_id = {"un": 0, "real": 1, "believ": 2, "happy": 3}
encoded = encode_with_special_tokens(short_tokens, token_to_id, max_len=8)
print(f"\nTokens: {short_tokens}")
print(f"Encoded [BOS, ...tokens..., EOS, PAD, PAD, ...]: {encoded}")

# --- Token cost example ---
cost_per_1k_tokens = 0.002
document = "unbelievable unreal believe real unhappy unbelievable"
doc_tokens = tokenize_sentence(document, merges_a)
cost = (len(doc_tokens) / 1000) * cost_per_1k_tokens
print(f"\nDocument: '{document}'")
print(f"Token count: {len(doc_tokens)}")
print(f"Estimated cost at ${cost_per_1k_tokens}/1K tokens: ${cost:.6f}")

Expected Output:

Sentence: 'unbelievable unhappy'

Tokenizer A: ['un', 'believ', 'a', 'b', 'l', 'e</w>', 'un', 'h', 'a', 'p', 'p', 'y', '</w>']
  Token count: 13

Tokenizer B: ['u', 'n', 'b', 'e', 'l', 'ie', 'v', 'a', 'b', 'l', 'e', '</w>', 'u', 'n', 'happy</w>']
  Token count: 15

Tokens: ['un', 'real']
Encoded [BOS, ...tokens..., EOS, PAD, PAD, ...]: [12, 0, 1, 13, 14, 14, 14, 14]

Document: 'unbelievable unreal believe real unhappy unbelievable'
Token count: 26
Estimated cost at $0.002/1K tokens: $0.000052

10. How It Works

The exact same input sentence (“unbelievable unhappy”) produces 13 tokens under Tokenizer A but 15 tokens under Tokenizer B — direct, verified proof that token count is a property of the specific tokenizer, not the text itself. Tokenizer A learned a useful “un” merge from its training corpus (present in “unbelievable,” “unreal,” “unhappy”); Tokenizer B’s corpus didn’t reinforce that pattern as strongly, producing more granular splits.
The encoded sequence [12, 0, 1, 13, 14, 14, 14, 14] shows the complete special-token pattern: 12 (<BOS>) at the start, the two real token IDs (0, 1), 13 (<EOS>) marking the end, then 14 (<PAD>) repeated to fill the fixed max_len=8 — exactly how real batched LLM training/inference pads variable-length sequences to a uniform shape.
The cost example shows the direct, mechanical link between token count and dollar cost: 26 tokens at a hypothetical $0.002 per 1,000 tokens comes to $0.000052 — trivial here, but this exact calculation, multiplied by real production volumes, is precisely how LLM API costs are computed.

11. Why Tokenization Matters — Practically

Concern	Why tokenization is the direct cause
Token limits	A model’s context window (Module 3) is measured in tokens, not words or characters
Token cost	API pricing is per-token — verified directly above
Different token counts across models	Different tokenizers, different learned vocabularies — verified directly above
Latency	More tokens generally means more processing time, both for input and generation

12. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Every single LLM API call starts with this exact process — and because different providers train their own tokenizers, the same prompt can cost noticeably different amounts, or consume different fractions of the context window, depending on which model you send it to.

13. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: High, practically. An agent’s entire assembled context — system prompt, conversation history, retrieved documents, tool definitions and results — is tokenized via this exact mechanism before hitting the model’s context window (Module 3).

Accurate token counting (using the specific target model’s actual tokenizer, not an estimate) is a genuine, practical requirement for managing context budget and cost in production agent systems.

14. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming token counts are comparable across different LLM providers.

Why it is incorrect: As verified directly, identical text produces different token counts with different tokenizers — cost and context usage estimates must use the specific target model’s actual tokenizer.

⚠️ Mistake

Incorrect idea: forgetting special tokens consume context budget too.

Why it is incorrect: <BOS>, <EOS>, and any padding all count toward a sequence’s total token usage, not just the “real” content tokens.

⚠️ Mistake

Incorrect idea: assuming <UNK> tokens are common in modern LLMs.

Why it is incorrect: With sub-word tokenization, genuinely unknown words are rare — the tokenizer decomposes unfamiliar words into known sub-word pieces (NLP course Module 14) rather than falling back to <UNK> in most cases.

15. Important Distinctions

Token	Word
The actual unit of model processing	One possible kind of token — not all tokens are complete words

Tokenizer A	Tokenizer B
Trained on its own corpus, own learned merges	Different corpus, different merges — same text, different token count (verified directly)

`<EOS>`	`<PAD>`
Signals genuine end of the sequence/generation	Filler, used only to make batch sequences a uniform length

16. Production Considerations

Always use the target model’s actual tokenizer for cost/context estimation — a generic word-count approximation can be meaningfully wrong in either direction.
Padding tokens should generally be masked out of loss computation during training (a detail covered further in Module 8) and excluded from attention where appropriate, so they don’t distort the model’s learned behavior.
Domain-specific or rare vocabulary produces more tokens per word on average — a genuine, practical cost consideration for specialized applications (medical, legal, code-heavy content).

17. What You Should Remember

LLMs process token IDs, not words — produced by a fixed, pre-trained sub-word tokenizer (BPE/WordPiece/SentencePiece, NLP course).
Different models have different tokenizers — verified directly: identical text produced 13 vs. 15 tokens across two differently-trained tokenizers.
Special tokens (<BOS>, <EOS>, <PAD>, <UNK>) carry structural meaning and consume real token budget, verified directly in a full encoded sequence.
Token count directly determines API cost and context usage — verified with a real cost calculation.

18. Interview Questions

Beginner

Q: Why do LLMs process tokens instead of words directly?

Ans: Sub-word tokenization keeps the vocabulary a manageable, fixed size while still being able to represent virtually any input — including words never seen during training, which get decomposed into smaller, familiar sub-word pieces. Whole-word tokenization would require an impractically large vocabulary and fails entirely on genuinely new words.

Intermediate

Q: Why can the same sentence produce a different number of tokens when sent to two different LLM providers?

Ans: Each provider trains their own tokenizer on their own corpus, learning its own specific set of sub-word merge rules.

Verified directly in this module: the identical sentence “unbelievable unhappy” produced 13 tokens under one tokenizer and 15 under another, purely because each tokenizer’s training data led it to learn different useful sub-word patterns.

Advanced

Q: Why does padding need to be handled carefully during training, rather than just treated as extra tokens?

Ans: Padding tokens (<PAD>) carry no genuine content — they exist purely to make variable-length sequences uniform for batched processing. If included naively in loss computation or attention, the model could learn to predict or attend to meaningless padding patterns, degrading real performance.

Production training pipelines mask padding tokens out of the loss calculation and often out of attention computation entirely, so they only serve their structural purpose (uniform batch shape) without influencing what the model actually learns.

Scenario

**Q: A team estimating API costs for a new application uses a simple “words × 1.3” approximation for token counts, then finds their actual bill is notably different from the estimate.

What’s the likely cause, and what should they do instead?** A: Generic word-to-token ratio approximations are just that — approximations — and can be meaningfully wrong depending on the specific content (technical jargon, non-English text, or unusual formatting can all produce more tokens per word than typical English prose).

As demonstrated directly, token count is a property of the SPECIFIC tokenizer being used, not a fixed ratio to word count. The correct approach is using the target model’s actual tokenizer (many providers offer a tokenizer library or endpoint for exactly this purpose) to get precise, model-specific token counts for cost estimation.

AI Engineering

Q: Why does tokenization matter directly for context window management in a RAG or agent application?

Ans: Context windows (Module 3) are measured in tokens, not words or characters — every piece of assembled context (system prompt, conversation history, retrieved documents) consumes tokens according to the specific model’s tokenizer.

Accurately counting tokens before assembling a prompt is essential for staying within the context limit and for predicting cost, especially in RAG systems where retrieved document chunks can vary significantly in how many tokens they actually consume relative to their apparent length in words or characters.

19. Next Step

Next: Module 3 — Context Window — what a context window actually is, why it has limits, and its direct relationship with RAG.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026