Checkpoint

Every article in this phase has described training as a long, repeated process — epochs of batches, gradients calculated and applied, over and over, sometimes for weeks on end, as covered in the Training article. A process that long, running on expensive hardware, needs a practical safety net. That safety net is called a checkpoint.

The simple definition

A checkpoint is a saved snapshot of a model’s current weights and biases at a specific point during training. Recall from the Parameters article that a model, at its core, is just a specific configuration of numbers, saved to a file. A checkpoint is exactly that — a file capturing exactly what those numbers were at a particular moment, partway through (or at the end of) a training run, so that state can be reloaded and picked back up later without having to start over from scratch.

Saving training state at a moment in time

Suppose training is planned for 20 epochs and a checkpoint is saved every 5 epochs:

checkpoint-05
checkpoint-10
checkpoint-15
checkpoint-20

If the machine fails during epoch 17, training can resume near epoch 15 instead of restarting from epoch 1.

What may be stored

Checkpoint
├── model parameter values
├── optimizer state
├── current epoch and training step
├── learning-rate scheduler state
├── random-number state
└── important configuration metadata

A weights-only file can run inference, but it may not contain everything required to resume training exactly.

Selecting the best checkpoint

The final epoch is not necessarily the best model.

Checkpoint	Validation loss
Epoch 5	0.42
Epoch 10	0.31
Epoch 15	0.36

The epoch-10 checkpoint may be selected because its validation loss is lowest, even though training continued afterward.

Production checkpoint safety

• Version checkpoints immutably.
• Record the data, code, and configuration used.
• Verify file integrity before loading.
• Restrict access to valuable or sensitive model artifacts.
• Test a checkpoint before deployment.
• Keep an approved earlier version for rollback.

Why this is a genuine necessity, not a nice-to-have

Training runs for large models take an enormous amount of time and cost real money every hour, as covered throughout the Training article’s discussion of GPU clusters and multi-week schedules. Over that much time, on that much specialized hardware, running continuously, something eventually goes wrong — a hardware failure, a software crash, a power outage, or simply a scheduled maintenance window.

Without checkpoints, any such interruption would mean losing all progress made since the very beginning of training, potentially wasting days or weeks of expensive compute time in an instant. Checkpoints exist specifically to prevent this: save the current state regularly, and if something fails, resume from the most recent save rather than starting completely over.

flowchart LR
    A[Training begins] --> B[Progress: epoch 1]
    B --> C[Checkpoint saved]
    C --> D[Progress: epoch 2]
    D --> E[Checkpoint saved]
    E --> F{Something fails?}
    F -->|Yes| G[Resume from latest checkpoint]
    F -->|No| H[Continue to completion]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of saving progress in a long video game. Rather than risking having to replay an entire multi-hour campaign from the very beginning if the game crashes or the console loses power, you save your progress at regular intervals, so a crash only costs you the time since your last save, not all your progress.

Where this breaks down: A video game save captures a rich, specific narrative state — position, inventory, story progress. A model checkpoint captures something much more narrowly defined but equally complete for its purpose: the exact numerical values of every weight and bias at that moment, plus typically some additional bookkeeping information (like which epoch training had reached, and the optimizer’s own running statistics, referenced in the Optimization article’s discussion of Adam’s momentum tracking) needed to resume training exactly as if it had never stopped.

What a checkpoint actually contains

A well-formed checkpoint typically saves more than just the raw weights, precisely because resuming training cleanly requires more context than the weights alone. This commonly includes: the model’s current parameter values, the current epoch or step number, and the optimizer’s internal state — for an Adam-based optimizer, as covered in the Optimization article, this means the running averages of recent gradients that Adam depends on for its momentum and adaptive step sizing.

Restoring only the weights, without this additional state, would technically work, but would effectively restart the optimizer’s “memory” from scratch, producing a subtly different — and often worse — training trajectory than a true, complete resume.

Why checkpoints matter beyond just crash recovery

Checkpointing serves several genuinely distinct practical purposes beyond simple disaster recovery:

• Evaluation along the way. Recall from the Epoch article that engineers watch validation performance to decide when to stop training. Doing this meaningfully requires having an actual saved model to evaluate at each of those points — the checkpoint from a specific epoch is exactly what gets loaded and tested against the validation set.
• Choosing the best version, not just the last one. Because of the overfitting risk covered in the Epoch article, the checkpoint from partway through training sometimes performs better on validation data than the very final checkpoint. Saving checkpoints throughout the run lets an engineer go back and select whichever specific saved version actually performed best, rather than being stuck with only the final state.
• Enabling fine-tuning by others entirely. Every mention throughout this glossary of “downloading a pretrained model” — from the Algorithm article onward — is really describing downloading someone else’s final training checkpoint, ready to be used as-is or further fine-tuned, exactly as covered in the Training article’s discussion of adapting an existing model rather than training one from scratch.

A concrete example, layered

For a simple beginner example: training the one-weight house model for 100 epochs might involve saving a checkpoint every 10 epochs, so if the process crashes at epoch 47, training resumes from the epoch-40 checkpoint rather than starting over entirely from epoch 0.

For a production example: training runs for models at the scale of GPT-3 — reportedly run across a supercomputer-scale cluster of roughly 10,000 GPUs over weeks, as covered in the Gradient article — save checkpoints regularly throughout the process specifically because, across that much hardware running that long, some individual component failure during the run is a near statistical certainty rather than a remote possibility; frontier labs’ training infrastructure is built around the assumption that checkpointing and automatic resume will be needed, not treated as an optional safeguard.

The real cost checkpointing itself adds

Checkpointing isn’t free, and it’s worth naming the trade-off honestly rather than presenting it as a pure win. Saving a checkpoint means writing a model’s entire weight file — potentially hundreds of gigabytes for a large model, echoing the file-size math from the Parameters article — to storage, which takes real time and pauses useful training progress while it happens.

Checkpoint too often, and a meaningful fraction of total training time gets spent on saving rather than learning; checkpoint too rarely, and a failure risks losing more progress than necessary. Real training setups tune this frequency deliberately, balancing safety against overhead, rather than either extreme.

Check your understanding

Is a checkpoint always just model weights? No. Resumable training often needs optimizer and training state too.

Must the newest checkpoint be deployed? No. Validation and safety evaluation determine the approved version.

Common misconception

A frequent assumption: that a checkpoint is basically the same thing as the “final model,” just saved a bit early. In practice, as the sections above explained, a checkpoint captures more than final model weights — it’s a complete, resumable snapshot of an in-progress process, including optimizer state that a plain, standalone trained model file typically doesn’t need to include at all.

The final released version of a model (what gets published or deployed) is usually just the weights extracted from the best or final checkpoint; the checkpoint itself, with all its extra training-resumption bookkeeping, is a working engineering artifact rather than the finished, distributable product.

Full, weights-only, and sharded checkpoints

A full training checkpoint may include weights, optimizer state, scheduler state, random-number state, and the current step. A weights-only checkpoint can be enough for inference but usually cannot resume training at the exact same point.

Large checkpoints may be split into several shard files. Teams record the model architecture, tokenizer, software version, file list, and checksums so the checkpoint can be loaded correctly and verified as unaltered. Untrusted checkpoint formats or loading code can be a security risk.

Where this fits in what comes next

You now understand how a long training run protects its own progress and enables flexible evaluation and reuse along the way. The next article, Seed, closes out this phase by covering a small but important practical detail: how randomness itself — present throughout training, from initial weight values to batch shuffling — gets controlled and made reproducible.

In one sentence

A checkpoint is a saved snapshot of a model’s weights and training state at a specific point in time, and it’s the practical mechanism that makes long, expensive, failure-prone training runs survivable, evaluable along the way, and reusable by others afterward.

Related Terms

• Training
• Parameters
• Optimization
• Epoch
• Seed