Optimization

The Gradient Descent article closed with a preview: modern training runs typically go a step further than the plain, fixed-step version described in that article, using refined variants like Adam. This article is that promised full explanation — the broader idea of optimization, and the specific, smarter algorithms that have quietly become the real, practical standard.

The simple definition

Optimization, in this context, refers to the broader family of strategies for actually using gradients to update a model’s weights — of which plain gradient descent is just the simplest, original member. An optimizer is the specific algorithm chosen to do this job in a real training run. Gradient descent gave you the basic idea: step opposite the gradient, scaled by the learning rate. Optimization is the ongoing engineering effort to do that same basic job better — faster, more stably, and more reliably across the enormous, complicated loss landscapes of real neural networks.

Turning gradients into parameter updates

Optimization is the larger process of finding parameter values that reduce the chosen loss.

prediction → loss → gradients → optimizer → updated parameters

For basic gradient descent:

new weight = old weight - learning rate × gradient

If weight = 5, gradient = 4, and learning rate = 0.25:

new weight = 5 - 0.25 × 4 = 4

Why optimizers keep extra state

Momentum remembers a moving direction from earlier gradients. Adaptive optimizers adjust effective step sizes for different parameters.

Current gradient
      +
Optimizer state from earlier steps
      +
Learning-rate settings
      ↓
Parameter update

That optimizer state is why a resumable checkpoint often stores more than model weights.

Optimization is not the product objective

Reducing training loss does not automatically optimize user satisfaction, fairness, factuality, latency, or cost. Teams must choose losses, evaluation metrics, and constraints that connect mathematical training to the actual system goal.

Real GPT example: Adam and mixed precision

The InstructGPT paper reports using Adam with:

β₁ = 0.9
β₂ = 0.95

These values control moving averages inside Adam:

β₁ → how strongly earlier gradient direction influences momentum
β₂ → how strongly earlier squared gradients influence adaptive scaling

The paper also reports FP16 weights and activations with FP32 master copies of weights.

FP16 computation and storage where appropriate
                 +
FP32 master weight copies for more stable updates
                 ↓
mixed-precision training

This illustrates why real optimization state is larger than the model file used only for inference. Training may keep parameter values, gradients, Adam’s moving averages, and master copies at the same time.

Why plain gradient descent needed improving

Recall from the Gradient Descent and Learning Rate articles: a single, fixed learning rate applied uniformly to every parameter is a genuinely blunt instrument. Some weights might need larger updates than others at any given moment; the ideal step size can vary meaningfully across the millions or billions of parameters in a real network, and even change over the course of training for any single one of them.

Plain gradient descent has no way to account for any of this — it treats every parameter identically, using the same learning rate, every single step. Optimizers exist specifically to fix this blind spot, adjusting how the gradient gets applied in smarter, parameter-specific ways.

flowchart LR
    A[Gradient calculated via Backpropagation] --> B{Which optimizer?}
    B -->|Plain gradient descent| C[Fixed step, same rule for every parameter]
    B -->|Adam / modern optimizer| D[Adaptive step, tuned per parameter, using recent gradient history]

The core idea behind modern optimizers: momentum and adaptivity

Two related ideas, layered together, explain almost every meaningful improvement over plain gradient descent:

• Momentum. Rather than reacting only to the current gradient, an optimizer with momentum keeps a running memory of recent gradients, and lets that accumulated direction carry some influence into the next update — similar to a ball rolling downhill, which doesn’t stop and restart its direction fresh at every instant, but carries momentum from where it’s already been rolling. This helps smooth out noisy, batch-to-batch gradient fluctuations (recall the noise trade-off from the Batch article) and can help push through small bumps in the loss landscape that might otherwise stall progress.
• Adaptive learning rates. Rather than using one global learning rate for every parameter, an adaptive optimizer tracks each parameter’s own recent gradient history and adjusts its effective step size individually — giving smaller, more cautious steps to parameters whose gradients have been large and volatile recently, and larger, more confident steps to parameters whose gradients have been small and stable.

Adam and AdamW: widely used modern optimizers

Adam (short for Adaptive Moment Estimation), introduced by Kingma and Ba in 2015, combines both ideas above into one widely adopted, genuinely standard algorithm — and a refinement of it called AdamW has become, in practice, the de facto default optimizer for training large language models across the field.

Adam works by tracking two running statistics for every single parameter: a moving average of recent gradients (providing momentum) and a moving average of recent squared gradients (providing the adaptive, per-parameter step-size adjustment). Two settings control how much recent history these running averages weigh, conventionally called beta1 and beta2, with commonly used default values of 0.9 and 0.999 respectively — themselves additional hyperparameters, on top of the learning rate, that an engineer can tune.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of the difference between a hiker who resets their sense of direction from scratch at every single step (plain gradient descent) versus a skier who carries genuine downhill momentum from recent turns, and who has also learned, from the last several turns, which parts of the slope tend to be icy and need more caution, and which parts are smooth and can be skied more confidently (Adam).

Where this breaks down: A skier’s sense of “icy versus smooth” comes from genuine physical sensation and experience. Adam’s “caution versus confidence” per parameter comes purely from tracking two running numerical averages — recent gradient size and recent squared gradient size — recalculated mechanically at every step, with no sensation or experience involved, just arithmetic applied consistently to every one of a model’s parameters individually.

A concrete example, layered

For a simple beginner example: training the one-weight house model with plain gradient descent might take a fixed-size step every time, regardless of how the loss has been behaving recently; the same training run using Adam would automatically take smaller, more careful steps if recent updates have been noisy or overshooting, and larger, more confident steps once the weight has settled into a smoother, more stable region of the loss landscape.

For a production example: OpenAI’s GPT-3 paper confirms the model was trained using the Adam optimizer, and AdamW — Adam combined with a refinement called decoupled weight decay, which tends to improve how well a model generalizes — has since become close to a universal default across the field for training large language models, used by essentially every major lab building GPT-, Gemini-, Claude-, and Llama-class models, precisely because of the stability and efficiency gains momentum and adaptivity provide at this enormous scale.

Where optimizers still have real limitations

Even the most refined modern optimizers don’t eliminate every challenge covered earlier in this phase. Adam and its relatives still inherit the fundamental limitations of gradient-based learning discussed in the Gradient article — including the possibility of settling into a local minimum — and they introduce their own new hyperparameters (beta1, beta2, and others) that themselves need reasonable default values or tuning.

Adaptive optimizers can also, in some documented cases, generalize slightly worse than plain, carefully-tuned gradient descent on certain tasks, which is part of why optimizer choice remains an active, ongoing area of research rather than a fully solved, one-size-fits-all decision — new refinements to Adam are still being actively published in AI research even now, a direct sign that this space is still evolving, not settled.

Check your understanding

Are optimizer and loss function the same? No. Loss measures error; the optimizer changes parameters.

Does Adam remove the need to choose a learning rate? No. Adam still has important settings and trade-offs.

Common misconception

A frequent early mix-up: assuming “optimization” refers to some separate, additional step layered on top of training, rather than being the actual mechanism inside training itself. It isn’t separate — the optimizer is the specific rule governing every single weight update throughout the entire training process described across this whole phase; choosing Adam over plain gradient descent doesn’t add an extra stage to training, it changes how each of the update steps you’ve already learned about actually gets computed.

Optimization is larger than the optimizer

The optimizer is the update algorithm, such as AdamW. Optimization is the broader effort to reduce the objective, including the optimizer, learning-rate schedule, initialization, batches, gradient clipping, regularization, and stopping decisions.

AdamW is common in Transformer training because it combines adaptive updates with decoupled weight decay. It is not the only valid optimizer, and the best choice depends on architecture, scale, memory, and the training objective.

Where this fits in what comes next

You now understand the complete update mechanism this phase has assembled: a loss function measures error, backpropagation calculates gradients efficiently across every layer, and an optimizer — plain gradient descent or a refined version like Adam — uses those gradients to actually update every weight, batch by batch, epoch by epoch. The next article, Checkpoint, covers a practical necessity that follows directly from everything covered so far: saving a model’s weights partway through this long, expensive process, so a training run doesn’t have to restart from nothing if something goes wrong.

In one sentence

Optimization is the broader discipline of applying gradients to update weights effectively, and Adam — combining momentum and per-parameter adaptive step sizes — is the specific, refined optimizer that has become the practical, near-universal standard for training the large language models described throughout this glossary.

Related Terms

• Gradient Descent
• Gradient
• Learning Rate
• Backpropagation
• Checkpoint