Begin with the central question
What exactly enters the first Transformer block after a user types a sentence?
Essential words
A token ID identifies one tokenizer vocabulary entry. An embedding converts that ID into a vector. Positional information tells the model where the token occurs.
What You Will Understand
The complete, concrete path from raw text to the actual numbers that enter a Transformer: tokenization, token IDs, the embedding matrix, and positional information — with a real, verified example showing exactly why a repeated word needs positional information to be distinguished from its earlier occurrence.
text -> tokens -> token IDs -> embeddings + positions -> block 1The problem this module solves
Everything in this course — attention, Transformer blocks, next-token prediction — operates on numbers, specifically vectors. Text is not numbers. This module exists to make completely concrete the translation step that happens before any Transformer computation begins: how “the cat sat on the mat” becomes a matrix of numbers a neural network can actually process.
Build the intuition
a Transformer never sees words — it only ever sees vectors. Tokenization and embedding are the translation layer, turning human language into a numerical form, and positional encoding is what stamps each vector with “and this is where I sit in the sentence,” since — as you’ll verify directly — the same word appearing twice would otherwise produce the exact same vector both times.
4. Real-World Analogy
Think of a librarian’s cataloging system: each book (token) gets a unique catalog number (token ID), and that number maps to a specific shelf location described by coordinates (the embedding vector).
But two identical copies of the same book on different shelves need something more than just “this is book #482” — they need their specific shelf location (positional information) to be told apart as physically distinct copies.
Analogy: The Library Catalog ID & The Shelf Coordinate tags Imagine managing database records for two identical copies of a popular book in a large warehouse:
- The Catalog ID (Token ID): Both physical books have the exact same ISBN barcode catalog number
482(Token ID =0for both occurrences of “the”).- The Target Coordinates (Embedding Vector): When searched, the database looks up catalog ID
482and reports its dimensional shape: “Paperback, 300 pages, green cover” (the embedding vector[0.248, -0.069, 0.324, 0.762]).- The Position tag (Positional Encoding): If you only use catalog records, you can’t tell which book sits in the Lobby Display Case (Position 0) and which book sits in the Basement Vault (Position 4).
- To break the tie, you stamp each book vector with a location-tracking coordinates sticker (
pevector). Now copy A is[0.248, 0.931, 0.324, 1.762]and copy B is[-0.509, -0.723, 0.364, 1.761]. They are physically distinct files.
📊 Visual Flowchart: From Raw Sentence Text to Transformer Vectors
Here is the step-by-step pipeline converting letters into position-aware inputs:
graph TD
Text["Raw Sentence:<br>'the cat sat on the mat'"] --> Tokenize["1. Tokenizer: Split into list<br>['the', 'cat', 'sat', 'on', 'the', 'mat']"]
Tokenize --> MapIDs["2. Vocabulary Mapping<br>[0, 1, 2, 3, 0, 4]"]
subgraph Lookups ["Embedding & Positional Layers"]
MapIDs --> EmbedLook["3. Embedding Matrix Row-Lookup<br>(Gets base semantic vectors)"]
PosEnc["4. Positional Encoding Calculator<br>(Generates sin/cos location offsets)"]
end
EmbedLook --> MatrixAdd["5. Vector Addition:<br>Token Embedding + Position offset"]
PosEnc --> MatrixAdd
MatrixAdd --> FinalInput["Final Input Matrix (Sequence Length x d_model)<br>(Fed to first self-attention layer)"]5. Core Concept
Text
↓
Tokens (splitting text into discrete units)
↓
Token IDs (mapping each token to an integer, via a
fixed vocabulary)
↓
Token Embeddings (looking up each ID's row in a learned
embedding matrix — DL Module 12)
↓
+ Positional Information (added so the model can distinguish
token ORDER, not just token IDENTITY)
↓
Transformer Input| Term | Definition |
|---|---|
| Tokenization | Splitting raw text into discrete units (words, sub-words, or characters) |
| Vocabulary | The fixed set of all possible tokens the model recognizes, each with a unique ID |
| Token ID | An integer index representing one specific token in the vocabulary |
| Embedding matrix | A learned lookup table: one row (a dense vector) per vocabulary entry |
| Embedding lookup | Selecting the row from the embedding matrix corresponding to a token’s ID |
Embedding dimension (d_model) | How many numbers make up each token’s embedding vector |
6. How It Works — Step by Step
1. TOKENIZE the raw text into a sequence of tokens
2. Map each token to its TOKEN ID using the vocabulary
3. For each token ID, LOOK UP the corresponding row in the
embedding matrix -- this produces one dense vector per token
4. ADD positional information to each token's embedding vector
(Module 8 covers exactly how this is computed)
5. The resulting sequence of vectors is what actually enters
the Transformer's first block🧠 You already saw embeddings and embedding lookups conceptually in Deep Learning Module 12. Here, the focus is specifically on their role as the Transformer’s literal entry point, and why they alone aren’t sufficient — order information has to be added on top.
7. Mathematical Intuition
Read the mathematics as a story
A token ID selects one row from an embedding table. Position information is then added so two copies of the same token can start differently when they occur in different places.
token ID 7 -> embedding row 7
same token at position 0: embedding + position 0
same token at position 4: embedding + position 4The embedding lookup is mathematically just indexing: if E is the
embedding matrix (shape vocab_size × d_model) and token_ids is a
list of integers, then E[token_ids] selects exactly those rows —
token_embeddings[i] = E[token_ids[i]]. Every occurrence of the same
token ID selects the exact same row — a direct, testable consequence
that matters, as shown below.
8. Small Worked Example
Walk through the example
- Tokenize a sentence containing a repeated word. 2. Look up both identical token IDs. 3. Confirm their token embeddings match. 4. Add different position vectors and compare again.
Consider the sentence “the cat sat on the mat” — notice “the” appears twice (positions 0 and 4). Since both occurrences have the identical token ID, their embedding lookup produces the identical vector — before any positional information is added, the Transformer has no way to tell these two occurrences apart at all.
9. Python / NumPy Example
What the code will demonstrate
This small NumPy example makes Transformer Input: From Text to Vectors visible with inspectable numbers and shapes. Read it in three passes: identify each input, follow the transformation line by line, and connect the printed output to the diagram above. The arrays are intentionally tiny teaching values; unless the text explicitly says otherwise, they are not weights or measurements from GPT, Gemini, or another trained model.
# The arrays are intentionally small so each transformation can be inspected.
# Printed values illustrate the mechanism; they are not trained-model measurements.
import numpy as np
# --- Step 1: Tokenization (simplified word-level, for clarity) ---
text = "the cat sat on the mat"
tokens = text.split()
print("Tokens:", tokens)
# --- Step 2: Vocabulary and Token IDs ---
vocab = {"the": 0, "cat": 1, "sat": 2, "on": 3, "mat": 4, "dog": 5, "ran": 6}
token_ids = [vocab[t] for t in tokens]
print("Token IDs:", token_ids)
# --- Step 3: Embedding matrix (lookup table) ---
np.random.seed(42)
vocab_size = len(vocab)
d_model = 4 # tiny, for a hand-traceable example
embedding_matrix = np.round(np.random.randn(vocab_size, d_model) * 0.5, 3)
print(f"\nEmbedding matrix shape: {embedding_matrix.shape}")
print("Embedding matrix:\n", embedding_matrix)
# --- Step 4: Embedding lookup ---
token_embeddings = embedding_matrix[token_ids]
print("\nToken embeddings:\n", token_embeddings)
print("\n'the' at position 0:", token_embeddings[0])
print("'the' at position 4:", token_embeddings[4])
print("Identical (same token ID -> same embedding)?", np.array_equal(token_embeddings[0], token_embeddings[4]))
# --- Step 5: Positional information ---
def positional_encoding(seq_len, d_model):
pos = np.arange(seq_len)[:, np.newaxis]
i = np.arange(d_model)[np.newaxis, :]
angle_rates = 1 / np.power(10000, (2 * (i // 2)) / np.float32(d_model))
angles = pos * angle_rates
pe = np.zeros((seq_len, d_model))
pe[:, 0::2] = np.sin(angles[:, 0::2])
pe[:, 1::2] = np.cos(angles[:, 1::2])
return pe
pe = positional_encoding(len(tokens), d_model)
transformer_input = token_embeddings + pe
print("\nFinal Transformer input (embedding + positional encoding):\n", np.round(transformer_input, 3))
print("\n'the' at position 0 (with position):", np.round(transformer_input[0], 3))
print("'the' at position 4 (with position):", np.round(transformer_input[4], 3))
print("Still identical?", np.array_equal(transformer_input[0], transformer_input[4]))Expected Output:
Tokens: ['the', 'cat', 'sat', 'on', 'the', 'mat']
Token IDs: [0, 1, 2, 3, 0, 4]
Embedding matrix shape: (7, 4)
Embedding matrix:
[[ 0.248 -0.069 0.324 0.762]
[-0.117 -0.117 0.79 0.384]
[-0.235 0.271 -0.232 -0.233]
[ 0.121 -0.957 -0.862 -0.281]
[-0.506 0.157 -0.454 -0.706]
[ 0.733 -0.113 0.034 -0.712]
[-0.272 0.055 -0.575 0.188]]
Token embeddings:
[[ 0.248 -0.069 0.324 0.762]
[-0.117 -0.117 0.79 0.384]
[-0.235 0.271 -0.232 -0.233]
[ 0.121 -0.957 -0.862 -0.281]
[ 0.248 -0.069 0.324 0.762]
[-0.506 0.157 -0.454 -0.706]]
'the' at position 0: [ 0.248 -0.069 0.324 0.762]
'the' at position 4: [ 0.248 -0.069 0.324 0.762]
Identical (same token ID -> same embedding)? True
Final Transformer input (embedding + positional encoding):
[[ 0.248 0.931 0.324 1.762]
[ 0.724 0.423 0.8 1.384]
[ 0.674 -0.145 -0.212 0.767]
[ 0.262 -1.947 -0.832 0.719]
[-0.509 -0.723 0.364 1.761]
[-1.465 0.441 -0.404 0.293]]
'the' at position 0 (with position): [0.248 0.931 0.324 1.762]
'the' at position 4 (with position): [-0.509 -0.723 0.364 1.761]
Still identical? FalseThe two occurrences of “the” produce identical embedding vectors on
their own (True) — the model literally cannot distinguish token
identity from token position using embeddings alone. Once positional
encoding is added, the two occurrences become distinct (False) —
concrete, verified proof of why positional information is a structural
necessity, not an optional enhancement, addressed fully in Module 8.
10. How Is This Used in Modern AI?
Where this concept lives
Follow the concept at three levels: inside the model, where the computation happens; inside the AI product, where that computation supports a visible feature; and inside production, where engineers measure speed, memory, quality, and failure cases. The details below connect those levels.
🤖 How Is This Used in Modern AI?
Every LLM’s very first computational step is exactly this pipeline. The embedding matrix is one of the model’s largest sets of learned parameters — for a vocabulary of, say, 50,000+ tokens and an embedding dimension in the thousands, the embedding matrix alone can contain hundreds of millions of parameters, learned during pretraining just like every other weight in the network (DL Module 7-9).
11. How Is This Used in LLMs?
Trace one model call
User text → tokens → Transformer computation → output-token probabilities
this topic affects one part of that computationAn LLM does not apply this idea as a separate magic step. It uses it as part of the repeated numerical pipeline that transforms token vectors and produces the next-token probabilities.
Real LLMs use sub-word tokenization (e.g., byte-pair encoding), not the simplified whole-word tokenization used here for clarity — a word like “unbelievable” might be split into several sub-word tokens. This lets the model handle rare or unseen words gracefully (by composing them from known sub-word pieces) while keeping the vocabulary size manageable.
The core mechanism — token ID → embedding lookup → add positional information — is identical regardless of tokenization granularity.
Real systems you can recognize
Gemini documents that all model input and output is tokenized, including non-text modalities; see Gemini token counting. Hugging Face tokenizers return IDs and attention masks that feed compatible Transformer models.
12. How Is This Used in Agentic AI?
Trace one agent step
Goal + history + tool results
↓
LLM processes the context
↓
Suggested answer or tool call
↓
Agent runtime validates and executes itThis distinction matters: the Transformer helps produce the proposal, while the surrounding agent software controls tools, permissions, retries, memory, and execution.
Direct relevance to Agentic AI: Moderate, foundational. Every piece of context an agent assembles — system prompt, conversation history, retrieved documents, tool results — ultimately goes through exactly this tokenization-and-embedding pipeline before the LLM can reason over any of it.
Understanding this is also directly relevant to context window budgeting (Module 17): every token in an agent’s assembled context consumes a real “slot” in this pipeline.
When this knowledge is useful
Use Transformer Input: From Text to Vectors when you need to explain, implement, debug, evaluate, or optimize the corresponding part of a Transformer pipeline. It is also useful when a model API behaves unexpectedly and you need to trace the behavior back to tokens, tensor shapes, attention visibility, training, or inference mechanics.
When it is not enough
Understanding this mechanism does not by itself prove that a complete model or application is accurate, safe, fast, or cost-effective. Production decisions still require representative evaluation data, latency and memory measurements, model-specific documentation, and tests of the surrounding retrieval or agent code.
13. Common Beginner Mistakes
⚠️ Mistake
Incorrect idea: assuming a “token” is always a whole word.
Why it is incorrect: Modern LLMs use sub-word tokenization — a single word can become multiple tokens, and token count doesn’t map cleanly onto word count.
⚠️ Mistake
Incorrect idea: believing embeddings alone capture word order.
Why it is incorrect: As demonstrated directly above, they don’t — two identical tokens produce identical embeddings regardless of position; positional information must be added separately.
⚠️ Mistake
Incorrect idea: confusing the embedding matrix with a dictionary definition lookup.
Why it is incorrect: The embedding matrix’s rows are learned dense vectors capturing distributional/semantic patterns (DL Module 12), not hand-written definitions or explicit semantic categories.
14. Important Distinctions
| Token | Token ID |
|---|---|
| A discrete unit of text (a word or sub-word piece) | The integer index representing that token in the vocabulary |
| Token ID | Embedding |
|---|---|
| A single integer — no inherent notion of similarity to other IDs | A dense vector — learned so that similar tokens have similar vectors (DL Module 12) |
| Token Embedding | Positional Encoding |
|---|---|
| Encodes WHAT the token is | Encodes WHERE the token sits in the sequence |
15. Production / Engineering Considerations
- Vocabulary size is a real design trade-off: larger vocabularies mean fewer tokens per sentence (shorter sequences, less compute) but a larger embedding matrix (more parameters, more memory).
- Tokenization is deterministic — the same text always produces the same tokens, but different models can use different tokenizers, meaning token counts (and therefore cost/context usage, Module 17) vary between models for the same input text.
16. Interview Questions
Beginner
Q: What is the difference between a token and a token ID?
Ans: A token is a discrete piece of text (a word or sub-word unit) produced by tokenization. A token ID is the integer index representing that specific token within a fixed vocabulary — the numerical form used to look up the token’s embedding.
Intermediate
Q: Why can’t a Transformer distinguish two occurrences of the same word using embeddings alone?
Ans: The embedding lookup is a deterministic function of token ID only — the same token ID always produces the exact same embedding vector, regardless of where it appears in the sequence. As demonstrated directly, two occurrences of “the” at different positions produce byte-for-byte identical embeddings; only after positional information is added do they become distinguishable.
Advanced
Q: Why is the embedding matrix considered a set of learned parameters, and what does that imply about how it changes during training?
Ans: Each row of the embedding matrix is a set of weights, exactly like any other layer’s weights in the network (DL Module 2) — they start randomly initialized and are adjusted via backpropagation and gradient descent (DL Module 7-8) during training, based on how well they help the model predict next tokens.
This means the specific vector representing any given token is not fixed or hand-designed — it emerges from training data, which is why embeddings from differently-trained models are not comparable (a distinction covered in DL Module 12).
Scenario
Q: You’re comparing two LLM providers’ pricing, both quoted per “token.” Why might the same input text cost a different number of tokens between the two?
Ans: Different models typically use different tokenizers, each with its own vocabulary and sub-word splitting rules. The same text can therefore be split into a different number of tokens depending on which model’s tokenizer processes it — this is a real, practical cost consideration when choosing between providers, not just an implementation detail.
Architecture
Q: Why is positional information ADDED to the embedding rather than processed as a completely separate input stream?
Ans: Adding it directly into the same vector space means every downstream computation (attention, the feed-forward network) automatically has access to both identity and position information simultaneously, without needing any special handling — the Transformer block’s mechanics (Module 9) don’t need to know or care that a token’s vector is actually a combination of two different kinds of information; they just operate on it as one vector.
AI Engineering
Q: When building a RAG or agent system, why does understanding tokenization matter beyond just “the model reads text”?
Ans: Context window limits (Module 17) are measured in tokens, not words or characters — accurately estimating how much of your assembled prompt (retrieved documents, conversation history, instructions) will fit requires understanding tokenization, not just character or word counts.
This directly affects real engineering decisions: how many documents to retrieve, how much conversation history to include, and API cost estimation, since most LLM providers bill per token.
17. What You Should Remember
- The complete pipeline: text → tokens → token IDs → embedding lookup → + positional information → Transformer input.
- The embedding matrix is a learned lookup table — same token ID always produces the same vector.
- Embeddings alone cannot represent token order — verified directly: two occurrences of the same word are indistinguishable until positional information is added.
18. How This Helps Me Build AI Systems
Every prompt you send to an LLM — including everything a RAG or agent system assembles into context — goes through exactly this pipeline before any “reasoning” happens. Understanding it concretely is the foundation for everything from token-budget planning (Module 17) to understanding why positional encoding (Module 8) is a genuine architectural requirement, not an optional refinement.
Next: Module 3 — Self-Attention From First Principles — the most important early module in this course: building attention from intuition, with real numbers, before ever seeing the formula.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed