Epoch

The Batch and Batch Size articles both mentioned this word in passing, without stopping to define it properly: an epoch is what you get once every batch in the training set has been processed exactly once.

The simple definition

An epoch is one complete pass through the entire training dataset. If a training set is split into many batches, as covered in the previous two articles, one epoch means the model has worked through every single one of those batches — every training example has contributed to a weight update exactly once. Training then typically starts over from the beginning of the dataset and does it again, for many epochs in a row.

One complete pass through training data

Suppose there are 1,000 examples and batch size is 100:

10 batches = 1 epoch
20 batches = 2 epochs
50 batches = 5 epochs

During each epoch, every training example is normally used once. The order is often shuffled before the next epoch.

Epoch 1: all training examples → 10 updates
Epoch 2: all training examples → 10 more updates
Epoch 3: all training examples → 10 more updates

After three epochs, the model has seen each example approximately three times and performed 30 batch updates.

Watching training and validation

Epoch	Training loss	Validation loss	Interpretation
1	0.80	0.85	Learning begins
5	0.35	0.40	Both improve
10	0.18	0.42	Validation worsens; possible overfitting

More epochs are not automatically better. Early stopping may keep the checkpoint from the epoch with the best validation result.

Real GPT example: 16 fine-tuning epochs

The InstructGPT paper reports training its supervised fine-tuning models for 16 epochs.

1 epoch  → one pass through the supervised fine-tuning examples
16 epochs → sixteen passes through those examples

This does not mean GPT-3 pretraining or every InstructGPT stage used 16 epochs. The published value belongs specifically to the supervised fine-tuning stage described in that paper.

This distinction matters because a modern GPT-style system can have several stages:

large-scale pretraining
        ↓
supervised instruction fine-tuning
        ↓
preference/reward modeling
        ↓
reinforcement-learning or other post-training

Each stage can use different data, losses, batch sizes, learning rates, and epoch definitions.

Why a dataset may be visited more than once

This directly answers a question the Training article raised but didn’t fully resolve: why would a model need to see the same data more than once?

A single pass through the data, batch by batch, nudges the weights a little in a better direction with each update — but one pass rarely gets weights anywhere close to their best values, especially early in training when they still start out essentially random, as described in the Parameters article.

Each additional epoch gives the model more opportunities to keep refining those same weights, using the same data again, gradually converging toward genuinely good values — the same way rereading a textbook chapter a second or third time reveals details and connections a single read-through missed.

flowchart LR
    A[Epoch 1: full pass through all batches] --> B[Epoch 2: full pass again]
    B --> C[Epoch 3: full pass again]
    C --> D[...continues until stopping condition met]
    D --> E[Trained Model]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of practicing a piece of music by playing through it start to finish, repeatedly. The first full play-through reveals the rough shape and the obvious mistakes. Each subsequent full play-through — each “epoch” of practice — smooths out finer and finer details, until the piece sounds genuinely polished.

Where this breaks down: A musician consciously notices which specific measures need work and can deliberately focus extra attention there. An epoch, in ML training, doesn’t discriminate this way — every batch in the dataset gets processed in essentially the same mechanical fashion each time through, with no built-in sense of “linger longer here, this part needs more work” (though the batches themselves are often reshuffled between epochs specifically to avoid the model learning some artifact of a fixed batch order, a small but genuine practical detail).

How many epochs is actually right, and how that gets decided

There’s no fixed universal number, and this is precisely the overfitting/underfitting trade-off first introduced in the Training article, now revisited with its proper vocabulary in place. Too few epochs, and the model hasn’t had enough repetition to learn the underlying pattern well — a case of underfitting, as covered in the Parameters article’s discussion of capacity.

Too many epochs, and the model starts memorizing specific quirks of the training data rather than the general pattern — overfitting, first raised in the Dataset article.

In practice, engineers watch the loss on the Validation Data set after every epoch, and stop training once validation loss stops improving or starts getting worse even as training loss keeps falling — a technique called early stopping, briefly mentioned back in the Validation Data article, now fully explained: the point where those two curves diverge is usually the signal that additional epochs would only make the model worse at generalizing, even though it looks better and better on the data it’s already seen.

A concrete example, layered

For a simple beginner example: training the one-weight house model for 5 epochs on a small dataset might leave the weight noticeably far from ideal, while training for 50 epochs on the same data gets it much closer — and training for 500 epochs might actually start overfitting, memorizing quirks of that particular small set of houses.

For a production example: OpenAI’s published GPT-3 training recipe processed roughly 300 billion tokens total, drawn from a much larger underlying dataset (including hundreds of billions of tokens from Common Crawl alone) — meaning large portions of that underlying data were seen once or not at all, rather than the many-epoch repetition typical of smaller datasets, illustrating exactly the less-than-one-epoch regime described above at real, documented scale.

Check your understanding

Is an epoch one parameter update? No. It contains one update per batch.

Does ten epochs guarantee the model has learned well? No. The appropriate count depends on validation behavior and the task.

Common misconception

A common assumption, especially among beginners working with smaller, more classical ML setups: that more epochs are always better, since more repetition sounds like it should always mean more learning.

As the overfitting discussion above makes clear, this isn’t true past a certain point — extra epochs beyond what the data and model can meaningfully support don’t just fail to help, they actively make the model worse at its real job, which is performing well on new, unseen data rather than on the training set it’s memorizing.

The right number of epochs is a genuine tuning decision, discovered by watching validation performance, not a “more is always better” dial.

Epochs, steps, and tokens

If a dataset has 1,000 examples and batch size is 50, one epoch normally contains 20 training steps. Five epochs normally contain 100 steps.

Small fixed datasets are often discussed in epochs. Very large language-model runs are commonly planned in training tokens and optimizer steps because datasets may be streamed, mixed, or so large that an epoch is not the most useful unit. One pass can be too little, enough, or too much depending on the task and data.

Where this fits in what comes next

You now have the complete outer structure of a training run: a batch is the unit of work between updates, an epoch is one full pass through every batch, and training repeats epochs until validation performance stops improving. The next article, Backpropagation, goes back inside a single update and finally explains the specific algorithm that makes calculating a gradient across a model’s many layers computationally possible at all — the missing mechanical piece this entire phase has been building toward.

In one sentence

An epoch is one complete pass through the training dataset, and deciding how many epochs to run — watched carefully against validation performance rather than fixed in advance — is one of the most direct, practical levers an engineer has over whether a model ends up underfit, overfit, or genuinely well-trained.

Related Terms

• Batch
• Batch Size
• Training
• Validation Data
• Backpropagation