TechByteByByte

Text and Language Representation

Understand the fundamental problem NLP exists to solve — computers operate on numbers, language consists of symbols and meaning — and trace the complete path from raw text to tokens to numbers.

#NLP#AI#Tokenization#Vocabulary#Text Representation

Begin with the central question

Before a model can understand text, what numbers should represent a word or emoji?

Essential words

A token is one unit produced by a tokenizer. A vocabulary is its collection of known token pieces. A token ID is an integer address, not a meaning score.

What You Will Understand

The complete, concrete path from raw text to numbers: characters, words, tokens, vocabulary, and token IDs. By the end, “text → tokens → numbers” will be a pipeline you’ve actually built by hand, not just a phrase.

Raw text -> tokens -> token IDs -> numerical vectors

Why Text Must Become Numbers

Module 1 established that computers operate on numbers, while language is symbolic. This module answers the immediate, practical question that raises: exactly how do you turn a sentence into numbers a model can process? Every NLP and LLM system, no matter how sophisticated, ultimately rests on some version of this same basic conversion.

From Symbols to Token IDs

think of building a vocabulary as creating a numbered coat-check ticket system. Every unique word that ever shows up gets assigned its own ticket number, once. From then on, instead of handling the actual word (a symbol), you just hand around its ticket number (a number) — much easier for a computer to store, compare, and process.

Analogy: The Ticket Broker / Coat-Check System Imagine running a busy nightclub check-room where guests leave their heavy winter coats:

  • The Problem (Bulky Text): Storing and moving physical coats (words like “elephant”, “hippopotamus”) inside the club office is slow, takes up a lot of space, and is hard to organize.
  • The Solution (The Ticket IDs): When a guest hands over their coat, you hang it on a specific numbered hanger in the closet (The Vocabulary) and hand the guest a small plastic tag containing a number like 42 (The Token ID).
  • Processing (Lookup): Instead of moving coats around during calculations, the computer simply records that customer 7 has tag 42 and customer 8 has tag 107. The computer can sort, save, and retrieve these lightweight numbers instantly.
  • When it is time to leave (Inference/Generation), the model hands back the numbers [42, 107], and the check-room clerk returns the original, physical coats (“cat sleeps”) back to the user.

📊 Visual Flowchart: Text to Token ID Numerical Pipeline

Here is how raw human sentences are converted into index vectors suitable for numerical processing:

graph TD
    Raw["Raw Text Input:<br>'cat eats fish'"] --> Tokenize["1. Tokenizer (Splits by whitespace)<br>['cat', 'eats', 'fish']"]
    Tokenize --> VocabCompare{"2. Vocabulary Lookup<br>(Check unique dictionary IDs)"}

VocabCompare -->|'cat'| ID0["ID: 0"]
    VocabCompare -->|'eats'| ID2["ID: 2"]
    VocabCompare -->|'fish'| ID3["ID: 3"]

ID0 --> Concat["3. Build Output Array"]
    ID2 --> Concat
    ID3 --> Concat

Concat --> FinalVector["Final Numerical Sequence:<br>[0, 2, 3]"]

4. Core Concept

TermDefinition
CharacterA single letter, digit, or symbol
WordA sequence of characters forming a linguistic unit
TokenThe actual unit of processing — often a word, but not always (Module 14 covers sub-word tokens)
SentenceA sequence of tokens forming a complete thought
DocumentA full piece of text (could be one sentence or many)
CorpusA collection of documents used for training or analysis
VocabularyThe complete set of unique tokens recognized by a system
Token IDA unique integer assigned to each vocabulary token
Text  →  Tokens  →  Token IDs  →  Numbers a model can process

⚠️ This module deliberately uses simple whitespace-based tokenization for clarity. Module 14 covers modern sub-word tokenization (BPE, WordPiece, SentencePiece) — the actual approach real LLMs use, which is meaningfully different and more sophisticated. Don’t assume “token” and “word” are interchangeable going forward.

5. How It Works — Step by Step

1. Collect a CORPUS -- your full collection of text documents
2. TOKENIZE each document -- split it into individual tokens
3. Build a VOCABULARY -- the set of every unique token across
   the entire corpus
4. Assign each vocabulary token a unique TOKEN ID (just an
   integer index)
5. Convert every document into a sequence of TOKEN IDs, by
   looking up each of its tokens in the vocabulary
6. The result is pure numbers -- ready for further numerical
   processing (Module 4 onward)

6. Mathematical Intuition

No formulas needed here — the “math” is simply assigning integers 0 through vocabulary_size - 1 to each unique token, in some consistent order. The one subtlety worth flagging early: these integers are arbitrary labels, not meaningful quantities — token ID 4 isn’t “more” than token ID 1 in any meaningful sense, even though they’re both just numbers. (This exact issue — and why it matters — reappears directly in Module 8, when embeddings are introduced specifically to fix it.)

7. Simple Example

For the tiny corpus ["cat eats fish", "dog eats fish", "cat sleeps"], the vocabulary (every unique word) is {cat, dog, eats, fish, sleeps} — 5 words. Assigning IDs alphabetically: cat=0, dog=1, eats=2, fish=3, sleeps=4. The sentence “cat eats fish” becomes [0, 2, 3] — three numbers, fully derived from the vocabulary lookup, with zero remaining symbolic content.

8. Build It in Python

What the code will demonstrate

We will build the conversion pipeline by hand before relying on a tokenizer library. Watch one sentence travel through four states: raw text → lowercase word tokens → vocabulary lookup → token IDs.

The IDs are only addresses in a vocabulary. A larger ID does not mean a more important word, and the ID itself does not yet contain meaning.

corpus = [
    "cat eats fish",
    "dog eats fish",
    "cat sleeps",
]

# Step 1: Text -> tokens (simple whitespace splitting for this module)
tokenized = [doc.split() for doc in corpus]
print("Tokenized corpus:")
for doc in tokenized:
    print(" ", doc)

# Step 2: Build a VOCABULARY -- the set of all unique tokens
vocabulary = sorted(set(word for doc in tokenized for word in doc))
print("\nVocabulary:", vocabulary)
print("Vocabulary size:", len(vocabulary))

# Step 3: Assign each vocabulary word a unique TOKEN ID
word_to_id = {word: idx for idx, word in enumerate(vocabulary)}
print("\nWord -> Token ID mapping:", word_to_id)

# Step 4: Convert each document into a sequence of TOKEN IDs
token_id_sequences = [[word_to_id[word] for word in doc] for doc in tokenized]
print("\nDocuments as token ID sequences:")
for original, ids in zip(corpus, token_id_sequences):
    print(f"  '{original}' -> {ids}")

import numpy as np
seq = np.array(token_id_sequences[0])
print("\nAs a NumPy array:", seq, "dtype:", seq.dtype)

Expected Output:

Tokenized corpus:
  ['cat', 'eats', 'fish']
  ['dog', 'eats', 'fish']
  ['cat', 'sleeps']

Vocabulary: ['cat', 'dog', 'eats', 'fish', 'sleeps']
Vocabulary size: 5

Word -> Token ID mapping: {'cat': 0, 'dog': 1, 'eats': 2, 'fish': 3, 'sleeps': 4}

Documents as token ID sequences:
  'cat eats fish' -> [0, 2, 3]
  'dog eats fish' -> [1, 2, 3]
  'cat sleeps' -> [0, 4]

As a NumPy array: [0 2 3] dtype: int64

9. Real-World Example

Notice token_id_sequences[0] ([0, 2, 3]) and token_id_sequences[1] ([1, 2, 3]) share two of their three numbers — reflecting that “cat eats fish” and “dog eats fish” share the words “eats” and “fish.” This shared structure is genuinely useful signal — but notice the representation is still purely about which words appear, with no encoding yet of what those words mean or how similar “cat” and “dog” are as concepts. That gap is exactly what Module 8’s embeddings exist to close.

10. How Is This Used in Modern AI?

🤖 How Is This Used in Modern AI?

Every LLM you’ll ever use performs exactly this text → tokens → token ID conversion as its very first processing step — before any neural network computation happens at all. The specific tokenization method is far more sophisticated in real systems (Module 14), but the fundamental shape — text becomes a sequence of integers via a fixed vocabulary — is identical to what you just built by hand.

Real systems you can recognize

The Gemini API documentation states that its inputs and outputs are tokenized and provides a count_tokens operation for measuring request size. It also gives the rough teaching estimate that 100 tokens correspond to about 60–80 English words, while warning through the API itself that actual counting depends on the content. See Gemini token counting.

OpenAI publishes tiktoken, the tokenizer library used with OpenAI model encodings. These are real examples of text becoming tokenizer-specific token IDs before model computation.

11. How Is This Used in Agentic AI?

Direct relevance to Agentic AI: High, foundational. Every piece of text an agent handles — user messages, tool results, retrieved documents — passes through this exact conversion before an LLM can reason about it. Understanding this mechanism concretely is also practically useful: token count (not word or character count) is what determines context window usage and API cost in real agent systems (Module 14 covers this directly).

12. Common Mistakes / Misunderstandings

⚠️ Mistake: assuming “token” always means “word.” As flagged in Section 4, this module simplifies for clarity — real tokenizers (Module 14) often split words into smaller sub-word pieces.

⚠️ Mistake: treating token IDs as having numerical meaning. ID 4 being larger than ID 1 says nothing about the words they represent being more “important” or “related” — they’re arbitrary labels, a point this module deliberately flags early because it directly motivates Module 8’s embeddings.

⚠️ Mistake: assuming vocabulary is built the same way for every system. Vocabulary size, construction method, and what counts as a “token” vary significantly between classical NLP systems and modern LLM tokenizers.

13. Important Distinctions

TokenWord
The actual unit of processingA specific kind of token — not all tokens are full words (Module 14)
TokenToken ID
A piece of text (e.g., “cat”)The integer index representing that token in the vocabulary
VocabularyCorpus
The set of unique tokens a system recognizesThe actual collection of text documents used for training/analysis

14. When to Use

Simple whitespace/word-level tokenization (as used in this module) is a reasonable starting point for prototyping classical NLP techniques (Bag of Words, TF-IDF — Modules 4-5) and understanding the fundamental mechanism, before moving to more sophisticated approaches.

15. When Not to Use

Don’t use simple whitespace tokenization for real modern NLP/LLM systems — it handles punctuation, unknown words, and multiple languages poorly compared to sub-word tokenization (Module 14), which is the actual standard in production systems.

16. Production Considerations

  • Vocabulary size is a real design trade-off — larger vocabularies can represent more distinct words directly but require more memory (each word needs, eventually, its own embedding vector — Module 8) and can make rare words harder to learn good representations for.
  • Out-of-vocabulary handling — what happens when a system encounters a word that wasn’t in its training vocabulary? Simple word-level vocabularies typically fail here entirely; sub-word tokenization (Module 14) is specifically designed to handle this gracefully.

17. What You Should Remember

  • The fundamental pipeline: text → tokens → vocabulary → token IDs → numbers — built by hand here, exactly what every NLP/LLM system does as its first processing step.
  • Token IDs are arbitrary labels, not meaningful quantities — a deliberate limitation this module flags, directly motivating Module 8’s embeddings.
  • “Token” ≠ “word” in modern systems — this module simplified for clarity; Module 14 covers the real, more sophisticated approach.

18. Interview Questions

Beginner

Q: What is a vocabulary, in the context of NLP?

Ans: The complete set of unique tokens a system recognizes — built by collecting every distinct token that appears across a corpus (or, for production systems, a fixed, predetermined set of tokens the tokenizer was trained to recognize).

Intermediate

Q: Why can’t token IDs be used directly as meaningful numerical features for a model, without further processing?

Ans: Token IDs are arbitrary integer labels assigned based on vocabulary construction order — there’s no meaningful numerical relationship between them. A model naively treating token ID 4 as “larger than” or “more significant than” ID 1 would be learning from noise, not genuine signal, since the actual assignment of IDs to words is essentially arbitrary. This is exactly why further representation (one-hot encoding, and eventually embeddings, Module 8) is needed before token IDs become useful model input.

Advanced

Q: Why does building a vocabulary from a training corpus create a fundamental limitation for handling new text?

Ans: Any word that doesn’t appear in the training corpus has no corresponding entry in the vocabulary, and therefore no token ID — a word-level vocabulary built this way has no principled way to handle genuinely novel words (out-of-vocabulary words) at all. This is a real, practical limitation of simple word-level tokenization, and is one of the core motivations behind sub-word tokenization (Module 14), which can represent virtually any word — even ones never seen during training — by breaking it into smaller, previously-seen sub-word pieces.

Scenario

Q: You build a vocabulary from a customer support corpus and later deploy your system, where it encounters a completely new product name that never appeared in training. What would happen with simple word-level tokenization, and how does this motivate later modules?

Ans: The new product name wouldn’t exist in the vocabulary, so there would be no token ID to assign it — simple word-level tokenization typically handles this either by failing outright or mapping it to a generic “unknown” token, losing essentially all information about what that specific word was. This exact limitation is why modern tokenization (Module 14) uses sub-word units instead — an unfamiliar product name could still be broken down into smaller, previously-seen character/ sub-word sequences, preserving at least partial information rather than discarding it entirely.

AI Engineering

Q: When an LLM API charges you based on “tokens,” what precisely does that refer to, based on this module?

Ans: It refers to the number of individual units produced by the model’s specific tokenizer when it processes your input (and output) text — not the number of words or characters. As this module demonstrated, text is converted into a sequence of these discrete units before any model computation happens, and API pricing/limits are based on counting these units directly, which is why the same text can have different “token counts” depending on which model/tokenizer is used (Module 14 covers this precisely).

19. Next Step

Next: Module 3 — Text Preprocessing — lowercasing, stemming, lemmatization, and the important, often-overlooked distinction between how classical NLP prepares text versus how modern LLM pipelines deliberately preserve more of the original.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed