Pretraining – LLMs | TechByteByByte

Before you continue: three tools for this module

Parameter: a learned number inside the model.
Loss: the training score that optimization tries to reduce.
Example: one input-target or preference record used for learning.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Pretraining solve inside a real language-model system?

Keep that central question about Pretraining in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

massive text corpus → next-token training → broadly capable base model

1. What You Will Learn

Learning outcomes

Explain why broad pretraining happens before task-specific adaptation.
Show how raw text supplies enormous numbers of next-token examples.
Describe what parameters learn and what pretraining does not guarantee.
Connect data quality, compute, checkpoints, and evaluation to the final base model.

In one sentence

💡 Big picture

Pretraining is the long learning stage where a model studies huge amounts of text by practicing next-token prediction again and again.

2. Why This Module Exists

The problem this module solves

A new model begins without useful language behavior.
Pretraining builds broad patterns, but it does not automatically make the model truthful, safe, or good at following instructions.

3. Intuition

you already know the training loop completely: forward pass → loss → backpropagation → gradient descent → repeat. You’ll formalize self-supervision precisely in Module 9. Pretraining is exactly this loop, run on raw internet-scale text, with “predict the next token” as the one and only training objective.

4. Core Concept — Dataset Construction

Raw text collection    (web pages, books, code, etc. -- an
                       enormous, diverse corpus)
   ↓
Cleaning                  (remove HTML artifacts, boilerplate,
                          malformed text)
   ↓
Deduplication                (remove exact or near-duplicate
                             documents -- prevents the model from
                             over-learning repeated content)
   ↓
Filtering                       (remove low-quality, too-short,
                                or otherwise unsuitable documents)
   ↓
Data mixture                       (deliberately balance different
                                   data SOURCES/TYPES — code, web
                                   text, books — in the final
                                   training set)
   ↓
Tokenization                          (Module 2)
   ↓
Training-ready dataset

5. How It Works — Step by Step (The Training Loop)

1. Take a batch of TOKENIZED training sequences
2. Create SHIFTED-TOKEN training examples (NLP/Transformers
   courses): input = sequence[:-1], labels = sequence[1:]
3. FORWARD PASS: run the input through the full model (Module 4,
   10), producing logits AT EVERY POSITION (not just the last,
   Module 5's inference-time behavior — training uses every
   position simultaneously, exactly as covered in the
   Transformers course)
4. LOSS: compute cross-entropy loss (Module 6) between the
   predicted distribution at every position and the TRUE next
   token at that position
5. BACKPROPAGATION (your Neural Networks course): compute
   gradients for EVERY parameter in the model, from this loss
6. GRADIENT DESCENT / OPTIMIZER (your Optimization course,
   typically AdamW): update EVERY parameter using these gradients
7. REPEAT across billions to trillions of tokens, over many
   training steps

🧠 Nothing here is new mechanically. Steps 3-6 are exactly your Neural Networks and Optimization courses’ training loop — the only thing distinguishing LLM pretraining is the sheer scale (Module 13) and the specific training objective (next-token prediction, Module 9).

Analogy: The Library Reading Marathon & The Covered-Finger Game Think of pretraining as a student reading through every book in a giant public library:

The Setup: Instead of a teacher writing separate quizzes for each page, the student plays a simple self-supervised game:

The Finger Rule (Shifted Labels): As they read a sentence, they cover the next word with their finger, guess what it is, and then lift their finger to check.

The Feedback (Gradient Descent):

Sentence: “The cat sat on the mat”

Read: “The cat sat on the” -> Guess: “floor” -> Lift finger: “the” (Error!).

The student instantly corrects their mental pattern weights (backprop and optimization) to increase the likelihood of predicting “the” in similar future contexts.

By playing this finger-lifting game over trillions of words, the student learns grammar, programming syntax, and factual knowledge without a single manual test-writer.

📊 Visual Flowchart: Shifted Labels and Parallel Training Loss Loop

Here is how input lists and shifted label targets are aligned during a parallel training step:

graph TD
    classDef inputs fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;
    classDef targets fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;
    classDef loss fill:#f1c40f,stroke:#333,stroke-width:1px,color:#fff;

    subgraph Alignment ["Sequence Shift Alignment"]
        InSeq["Input IDs: ['<bos>', 'the', 'cat', 'sat', 'on', 'the', 'mat']"]:::inputs
        LabelSeq["Label IDs: ['the', 'cat', 'sat', 'on', 'the', 'mat', '<eos>']"]:::targets
    end

    InSeq --> Model["1. Forward Pass: Transformer Blocks Stack"]
    Model --> Logits["2. Output Logits (vocab-sized vector at every position)"]

    Logits -.-> Compare["3. Parallel Cross Entropy comparison"]
    LabelSeq -.-> Compare

    Compare --> LossVec["4. Per-position Loss Values"]:::loss
    LossVec --> AvgLoss["5. Average Batch Loss (Single Scalar)"]:::loss

    AvgLoss --> Backpropagation["6. Backpropagate Gradients"]
    Backpropagation --> Optimization["7. AdamW Weight Update Step"]

6. Mathematical Intuition

Read the mathematics as a story

massive text corpus → next-token training → broadly capable base model

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

The total training loss for one batch is the average cross-entropy loss (Module 6) across every position, in every sequence, in the batch. This single scalar number is what backpropagation computes gradients from — exactly the same mechanism as any neural network training you’ve already covered, just with a next-token-prediction loss and an enormous amount of data.

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

The raw sentence “the cat sat on the mat” becomes a training example via the shifted-token trick: the input is everything except the last token, and the labels are the same sequence shifted by one position — the model’s task at every position is simply “predict the actual next token that appeared in this real text.” No human ever manually labeled this example; the label is the text itself.

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Pretraining.
# Follow the inputs, transformations, and output in order.
import numpy as np

def softmax(x, axis=-1):
    exp_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)

# --- Dataset construction ---
raw_documents_with_dupe = [
    "the cat sat on the mat",
    "the dog ran in the park",
    "the cat sat on the mat",  # exact duplicate
]
deduped = list(dict.fromkeys(raw_documents_with_dupe))
print(f"Before dedup: {len(raw_documents_with_dupe)} documents")
print(f"After dedup:  {len(deduped)} documents")

too_short = ["ok", "the cat sat on the mat"]
filtered = [doc for doc in too_short if len(doc.split()) >= 3]
print(f"\nBefore filtering: {too_short}")
print(f"After filtering (min 3 words): {filtered}")

# --- Shifted-token training example ---
vocab = ["<bos>", "the", "cat", "sat", "on", "mat", "<eos>"]
sentence = ["<bos>", "the", "cat", "sat", "on", "the", "mat", "<eos>"]
token_ids = [vocab.index(t) if t in vocab else vocab.index("the") for t in sentence]
input_ids = token_ids[:-1]
label_ids = token_ids[1:]
print(f"\nInput IDs:  {input_ids}")
print(f"Label IDs:  {label_ids}")

# --- One training step: forward -> loss ---
d_model = 4
vocab_size = len(vocab) + 1
np.random.seed(1)
embedding_table = np.random.randn(vocab_size, d_model) * 0.3
W_lm_head = np.random.randn(vocab_size, d_model) * 0.3

x = embedding_table[input_ids]
logits = x @ W_lm_head.T  # simplified: full Transformer stack already verified in Module 4/10

losses = []
for pos in range(len(input_ids)):
    probs = softmax(logits[pos])
    true_id = label_ids[pos] if label_ids[pos] < vocab_size else 0
    loss = -np.log(probs[true_id] + 1e-10)
    losses.append(loss)

total_loss = np.mean(losses)
print(f"\nPer-position losses: {[round(l,4) for l in losses]}")
print(f"Average loss for this training step: {total_loss:.4f}")

Expected Output:

Before dedup: 3 documents
After dedup:  2 documents

Before filtering: ['ok', 'the cat sat on the mat']
After filtering (min 3 words): ['the cat sat on the mat']

Input IDs:  [0, 1, 2, 3, 4, 1, 5]
Label IDs:  [1, 2, 3, 4, 1, 5, 6]

Per-position losses: [2.2844, 2.0916, 2.3294, 2.0044, 2.0778, 2.1169, 2.0943]
Average loss for this training step: 2.1427

9. How It Works

Deduplication removed the exact-duplicate document (3 → 2 documents) — preventing the model from disproportionately learning patterns from repeated content.
Filtering removed the too-short “ok” document, keeping only text meeting a minimum quality bar.
Label IDs are exactly Input IDs shifted by one position — label_ids[i] == token_ids[i+1] for every position, verified directly — the shifted-token trick that turns raw text into free, automatically labeled training data.
The average loss (2.1427) is one single number, computed from every position simultaneously in this one training example — this is precisely the scalar that would flow into backpropagation (your Neural Networks course) to compute gradients, and into an optimizer like AdamW (your Optimization course) to update every parameter.

10. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Every LLM’s foundational capability comes from pretraining, run at a scale of hundreds of billions to trillions of tokens, over weeks to months of compute (Module 13). This is the single most expensive phase of an LLM’s lifecycle, and the one most directly connected to the mechanics you already know from Neural Networks and Optimization.

11. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: Low, directly — you won’t pretrain a foundation model to build an agent. The value here is understanding exactly what capability an agent’s underlying LLM has, and doesn’t have, straight out of pretraining — a purely next-token-predicting base model (Module 17 covers precisely why this alone doesn’t produce a useful, instruction-following assistant).

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming pretraining involves a fundamentally different training mechanism than what you already learned.

Why it is incorrect: It doesn’t — forward pass, loss, backpropagation, gradient descent are exactly the same mechanics; only the objective (next-token prediction) and the scale differ.

⚠️ Mistake

Incorrect idea: assuming data quality doesn’t matter if you have enough quantity.

Why it is incorrect: As shown directly, deduplication and filtering are real, deliberate steps — training data quality genuinely affects what patterns a model learns, not just quantity.

⚠️ Mistake

Incorrect idea: believing pretraining alone produces a helpful, instruction-following assistant.

Why it is incorrect: It produces a model that predicts plausible next tokens for arbitrary text — Module 17 covers precisely why additional training (instruction tuning) is needed to make this genuinely useful as an assistant.

13. Important Distinctions

Pretraining	Fine-Tuning (Module 16)
Massive, general-purpose text corpus	Smaller, task/domain-specific dataset
Starts from random initialization	Starts from pretrained weights
Establishes broad language capability	Adapts/specializes existing capability

Data Cleaning	Data Deduplication
Removes malformed/low-quality text	Removes exact or near-duplicate content specifically

14. When to Use

Pretraining a foundation model from scratch is only undertaken by a small number of organizations with the necessary data and compute scale (Module 13). For essentially every practical AI engineering purpose, you’ll work with already-pretrained models, via fine-tuning (Module 16) or prompting alone.

15. When Not to Use

Don’t pretrain from scratch for domain adaptation or task-specific needs — fine-tuning an existing pretrained model (Module 16) is dramatically cheaper and faster, and is the practical, standard approach for nearly every real-world application.

16. Production Considerations

Data quality directly shapes model behavior — biases, errors, or low-quality patterns in training data propagate into the trained model’s behavior, a genuine, practical concern (Module 22 covers bias as a limitation directly).
Deduplication and filtering are non-trivial infrastructure challenges at the scale of real pretraining corpora — genuinely complex data engineering problems, not simple scripts.

17. What You Should Remember

Pretraining uses exactly the training loop you already know (forward pass, loss, backpropagation, gradient descent) — nothing mechanically new, just next-token prediction as the objective, at massive scale.
Dataset construction is a real, deliberate pipeline: cleaning, deduplication, filtering, and data mixture — verified directly with real before/after examples.
The shifted-token trick (verified directly: labels are inputs shifted by one position) is what turns raw, unlabeled text into free training data.

18. Interview Questions

Beginner

Q: What is pretraining, and what training objective does it use?

Ans: Pretraining is the initial, large-scale training phase where an LLM learns from a massive, general text corpus using next-token prediction as its objective — learning to predict, at every position in a sequence, what token comes next, using nothing but the raw text itself as its training signal.

Intermediate

Q: Why is deduplication a deliberate, necessary step in pretraining data construction, rather than something quantity alone would fix?

Ans: If duplicate or near-duplicate documents appear many times in the training corpus, the model effectively sees those specific patterns disproportionately often relative to the rest of the training distribution, potentially over-learning them or wasting training compute re-learning the same content repeatedly.

Deduplication (demonstrated directly in this module) ensures the training data’s diversity is genuinely reflected in how often different patterns are seen, rather than being skewed by repetition.

Advanced

Q: Explain precisely how the concepts from your Neural Networks and Optimization courses apply, unchanged, to LLM pretraining.

Ans: The core training loop is identical: a forward pass produces predictions (Module 5’s logits, computed at every position during training rather than just the last), a loss function (cross-entropy, Module 6) measures how wrong those predictions were relative to the true next tokens, backpropagation computes the gradient of this loss with respect to every parameter in the model, and an optimizer (typically AdamW, from your Optimization course) uses these gradients to update every parameter.

Nothing about this loop is unique to LLMs — what distinguishes pretraining is the training objective (next-token prediction across raw text) and running this identical loop across an enormous number of training steps and tokens.

Scenario

**Q: A team pretraining a domain-specific language model from scratch notices the model producing biased or low-quality outputs reflecting issues clearly present in parts of their training corpus.

What does this suggest about their pretraining pipeline?** A: This strongly suggests insufficient data cleaning and filtering (Section 4) — the model learns statistical patterns directly from whatever text it was trained on, with no separate mechanism for distinguishing “good” patterns from “bad” ones during pretraining itself.

This is a direct, expected consequence of training data quality issues, not a separate model flaw — the fix is improving the data pipeline’s cleaning and filtering steps, since the model faithfully learned exactly what was in its training corpus.

AI Engineering

Q: Why do most AI engineers never pretrain a model from scratch, and what do they do instead?

Ans: Pretraining requires enormous compute and data resources (Module 13) that are only practical for a small number of organizations.

Most AI engineers instead work with already-pretrained models, either using them directly via prompting, or fine-tuning them (Module 16) on a much smaller, task-specific dataset — leveraging the broad language capability already established during pretraining, and adapting it to specific needs at a small fraction of the cost and time pretraining from scratch would require.

19. Next Step

Next: Module 9 — Self-Supervised Learning — precisely why this training process needs no manual labeling, formalizing what Module 8 demonstrated directly.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026