Self-Supervised Learning

Every article in this phase has assumed pretraining has enough labeled data to work with, without ever explaining where that label actually comes from at this enormous scale. The answer, closing out this entire phase, is self-supervised learning.

The simple definition

Self-supervised learning is a training approach where the correct answer. The label — gets generated automatically from the data itself, rather than requiring a human to manually annotate it. Recall from the Machine Learning article’s distinction between supervised learning (labeled data) and unsupervised learning (no labels at all).

Self-supervised learning is a clever middle path: it’s technically supervised, in that real, genuine labels exist and get used exactly as covered in the Label and Ground Truth articles, but those labels are extracted automatically from the raw data itself, with zero human annotation required.

Why this solves pretraining’s single biggest practical problem

Recall from the Annotation and Labeling articles’ discussion of how expensive and slow human labeling genuinely is — real, substantial industry, real cost, real time. Recall from the Training Data article that GPT-3 was pretrained on roughly 300 billion tokens.

Manually labeling a dataset at that scale would be entirely, practically impossible — no annotation workforce, however large, could produce genuine human labels for hundreds of billions of individual training examples in any reasonable amount of time or budget.

Self-supervised learning sidesteps this problem entirely by not requiring human labels at all.

flowchart LR
    A[Raw, unlabeled text] --> B[Self-Supervised Learning: generate labels automatically from the text itself]
    B --> C[Massive labeled training signal, at zero annotation cost]

How this actually works, mechanically, for language models

This connects directly and precisely to the Next-Token Prediction article, which has been describing self-supervised learning all along without naming it. Recall the mechanism: a model is trained to predict the next token, given everything before it.

The “label” for this task. The correct next token — isn’t something a human had to annotate at all. it’s simply whatever word actually comes next in the real, existing text the model is training on.

Take any ordinary sentence from a book or webpage, and the text itself already contains its own ground truth. Given “The capital of France is,” the very next word in that real sentence, whatever it happens to be, is the label, extracted automatically, for free, from data that already existed.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a student practicing reading comprehension by covering up the last word of every sentence in a book with their thumb, guessing what it is, then lifting their thumb to check. The book itself, already written, provides both the practice question and the correct answer, with no separate answer key needed at all.

Where this breaks down: A student consciously chooses this practice method and understands why it helps.

Self-supervised learning isn’t a deliberate study strategy chosen by the model — it’s simply how the training data gets structured before the ordinary gradient descent and backpropagation process, covered throughout the Training Mechanics phase, is applied. the “covering up and checking” happens automatically and mechanically, for every single token in the entire training corpus, without any conscious strategy involved.

Why this specific trick made today’s scale of pretraining possible at all

It’s worth stating this plainly, since it’s genuinely one of the most consequential ideas underlying the entire modern era of large language models covered throughout this glossary.

Because self-supervised learning generates its own labels automatically from raw text, pretraining can scale to use essentially any text that exists. The hundreds of billions of tokens from Common Crawl and similar sources covered in the Training Data article — without that scale being bottlenecked by human labeling capacity at all.

This is precisely why pretraining datasets can be orders of magnitude larger than any deliberately human-labeled dataset covered throughout the Data Handling phase. The labeling constraint simply doesn’t apply in the same way.

A concrete example, layered

For a simple beginner example: training on the sentence “The dog chased the ball across the yard” using self-supervised next-token prediction automatically produces several free training examples — given “The,” predict “dog”. given “The dog,” predict “chased”. given “The dog chased,” predict “the” — each one a genuine input-label pair, extracted entirely from one ordinary sentence that required no human labeling at all.

For a production example: virtually every major large language model covered throughout this glossary — GPT, Gemini, Claude, Llama — is pretrained using exactly this self-supervised, next-token-prediction approach on massive amounts of raw text, precisely because it’s the only practical way to generate a genuine training signal at the trillion-token scale these models require.

Why self-supervised learning isn’t limited to just text

It’s worth knowing this technique generalizes well beyond language, since the same core idea shows up across other domains covered elsewhere in this glossary.

Some image-based self-supervised approaches train a model to predict a missing or masked-out portion of an image, using the rest of the image as automatically-generated context. The same “cover part of it up and predict what’s missing” logic applied to pixels instead of tokens.

The underlying principle stays identical: extract a genuine training signal automatically from the structure of the raw data itself, without needing a human to manually annotate anything.

Real self-supervised learning applications across different fields

Self-supervised learning’s core trick — extracting labels automatically from raw data — generalizes well beyond text, and it’s worth naming real examples across different kinds of data:

• Language models broadly — GPT, Gemini, Claude, and Llama, covered throughout this glossary, are all pretrained using self-supervised next-token prediction on raw text, the flagship, most widely known application of this technique.
• Computer vision — published self-supervised approaches like SimCLR and MoCo train image models by generating their own training signal directly from unlabeled images — for instance, learning to recognize that two different, randomly-cropped views of the same photo should produce similar internal representations — without any human ever labeling what’s actually depicted in any of those images.
• Speech recognition — wav2vec 2.0, a real, published self-supervised approach, learns useful audio representations directly from raw, unlabeled speech recordings, later enabling far more accurate speech recognition even in languages with very little manually transcribed training data available.
• Biomedical imaging — published research has applied self-supervised, contrastive learning techniques to large collections of unlabeled medical scans, letting a model learn generally useful visual features from raw imaging data before any radiologist-labeled diagnostic data is introduced, reducing how much expensive expert annotation a final, specialized model needs.
• Protein science — models like ProtGPT2, mentioned earlier in this phase, apply the same self-supervised, next-token-style prediction approach to protein sequences instead of language, learning meaningful patterns directly from the sequences themselves with no separate labeling process required at all.

Where do the labels come from?

Self-supervised learning creates the learning target from the data itself:

Original sentence: The moon reflects light from the Sun.

Input shown:       The moon reflects light from the
Target created:    Sun

The word Sun acts like a label for this training example, but a human did not manually annotate it. The training pipeline shifted the same sequence to create input-target pairs automatically.

Another style: hide and recover

Original:  Bees collect nectar from flowers.
Corrupted: Bees collect [MASK] from flowers.
Target:    nectar

Decoder-style GPT models commonly learn by predicting the next token. Encoder models such as BERT became famous for recovering masked tokens. Both are self-supervised because the answers are produced from the original data.

How this appears in GPT and Gemini

GPT’s next-token pretraining is a direct self-supervised task: each token in existing text supplies a target for the tokens before it. Gemini extends generative pretraining across multimodal representations, allowing relationships among text, images, audio, and video to contribute training signals.

Human-created data can still enter later stages for instruction following, preferences, safety, and evaluation. “Self-supervised” describes how the pretraining targets are obtained. it does not mean no humans were involved anywhere in collecting, filtering, designing, or governing the system.

Compare the major self-supervised objectives

Objective	What the model sees	Automatically created target	Famous example
Causal language modeling	Earlier tokens	The next token	GPT-style language pretraining
Masked language modeling	Text with hidden tokens	The original hidden tokens	BERT
Contrastive image learning	Two altered views of an image	Treat views of the same image as related	SimCLR and MoCo
Image-text contrastive learning	Images and captions	Match related image-text pairs	CLIP-style training
Speech representation learning	Raw or masked audio segments	Recover or distinguish audio representations	wav2vec 2.0

Different objectives create different strengths. Causal prediction naturally supports generation. Masked prediction supports bidirectional representations. Contrastive learning helps place related items close together in an embedding space.

A batch creates many language targets

For the token sequence Cats drink milk, causal language modeling can score several positions during training:

Input prefix    Target
Cats            drink
Cats drink      milk

A causal attention mask stops each position from seeing its future target. Although generation happens one token at a time, training can calculate loss for many positions in parallel.

What is known about GPT and Gemini

GPT-style base models are publicly described as using autoregressive next-token prediction. Gemini is also a generative Transformer family, but its multimodal training includes text, images, audio, and video, and the complete mixture of private objectives is not publicly specified.

It is safest to say that self-supervised generative learning is central to these foundation models—not that every modality in every proprietary model is trained with one identical text-only objective.

When to use self-supervised learning

Use it when raw data is abundant but human labels are scarce and a useful automatic prediction task can be constructed. It is usually followed by supervised or preference-based adaptation when the final product needs precise categories, instruction following, safety behavior, or human-defined quality.

Common misconception

A frequent beginner assumption: that self-supervised learning means a model is training without any labels or correct answers at all — essentially the same thing as unsupervised learning, first introduced back in the Machine Learning article.

As this article has explained, this is the key distinction worth holding onto: self-supervised learning genuinely does use real, correct labels, exactly like supervised learning, and those labels get used in the exact same loss-function-and-gradient-descent process covered throughout the Training Mechanics phase. The only difference is that those labels were extracted automatically from the data itself, rather than manually attached by a human annotator.

Closing out this phase

This article completes the Training Paradigms phase. Pretraining builds a model’s broad foundation through massive-scale training.

That scale is practical because self-supervised learning can generate labels automatically. Otherwise, the Training Data article’s trillion-token datasets would need an impossibly large human-labeling effort.

Transfer Learning explains why the expensive foundation is worth building once and reusing broadly. Fine-Tuning, Instruction Tuning, PEFT, LoRA, and QLoRA provide progressively specialized or efficient ways to adapt it.

Together, this phase closes the loop on how a model actually goes from raw internet text to a genuinely useful, customized AI system.

In one sentence

Self-supervised learning generates training labels automatically from the structure of raw data itself — no human annotation required — and it’s precisely this trick, applied to next-token prediction across hundreds of billions of tokens of ordinary text, that made pretraining at the scale covered throughout this entire glossary practically possible in the first place.

Related Terms

• Pretraining
• Next-Token Prediction
• Machine Learning (ML)
• Training Data
• Label