Training

You’ve now met every ingredient: Data organized into a Dataset, broken into Features and Labels, fed into an Algorithm to produce a Model. This article is about the process that actually connects all of them — the part where the algorithm and the data come together, repeatedly, until a trained model comes out the other end. That process is called training.

The simple definition

Training is the process of repeatedly showing a model examples, checking how wrong its guesses are, and adjusting it to be less wrong — over and over, until it gets good at the task. It’s the “studying” phase referenced loosely throughout earlier articles in this glossary, and it’s finally time to look at it directly, step by step.

One training update

Suppose a tiny model predicts exam score using hours studied:

predicted score = hours studied × weight

For one student:

Hours studied:  4
Current weight: 10
Prediction:     4 × 10 = 40
Correct label:  60

The prediction is too low. Training measures the error and adjusts the weight so that later predictions can move closer to the labels.

The complete training loop

flowchart LR
    A[Choose a batch of examples] --> B[Model makes predictions]
    B --> C[Loss measures error]
    C --> D[Backpropagation calculates gradients]
    D --> E[Optimizer updates parameters]
    E --> A

Take the loop one stage at a time:

1. Batch selection: Load a manageable group of examples.
2. Forward pass: Run input through the model to produce predictions.
3. Loss calculation: Compare predictions with the expected answers.
4. Backpropagation: Calculate how parameters contributed to the loss.
5. Optimization step: Change parameters slightly.
6. Repeat: Continue through more batches and epochs.

Training versus inference

Training
examples + labels → repeated parameter updates → trained model

Inference
new input + trained model → prediction

Training changes learned parameters. Inference normally uses those parameters without changing them.

A small code example

from sklearn.linear_model import LinearRegression

training_features = [[1], [2], [3], [4], [5]]
training_labels = [15, 30, 45, 60, 75]

model = LinearRegression()
model.fit(training_features, training_labels)

print(model.coef_)       # Learned weight
print(model.intercept_)  # Learned bias

fit(...) performs training. It uses the examples and labels to calculate parameter values that fit their relationship. The resulting model can later process a new value such as [[6]].

How training can go wrong

• Overfitting: The model works well on training examples but poorly on new ones.
• Underfitting: The model is too limited or insufficiently trained to capture the useful pattern.
• Data leakage: The training input improperly reveals the answer.
• Bad labels: Incorrect targets teach incorrect behavior.
• Unrepresentative data: The model learns a world unlike production.
• Unstable optimization: Poor settings cause training to diverge or fail to improve.

Production training practices

• Keep reproducible data, code, configuration, and model versions.
• Track experiments instead of relying on memory.
• Evaluate checkpoints on validation data.
• Stop or adjust training when validation quality stops improving.
• Test the final candidate on untouched test data.
• Record compute, duration, cost, and important limitations.
• Protect training data and model artifacts from unauthorized access.

Why this deserves its own close look

Earlier articles described the training loop in outline: guess, compare, adjust, repeat. That’s accurate, but it hides a lot of practical detail that actually matters once you start dealing with real systems — how many times you repeat, how big each adjustment should be, and how you know when to stop. Those details are the difference between a model that trains successfully and one that wastes enormous amounts of time and money without ever getting good.

What actually happens, step by step

flowchart LR
    A[Model with random parameters] --> B[Show it a batch of training examples]
    B --> C[Model makes predictions]
    C --> D[Compare predictions to labels: measure error]
    D --> E[Adjust parameters to reduce error]
    E --> F{Seen enough of the dataset?}
    F -->|No| B
    F -->|Yes, one full pass done| G[Repeat the whole dataset again: next 'epoch']
    G --> B
    F -->|Training complete| H[Trained Model]

A few pieces of vocabulary make this concrete:

• Epoch — one complete pass through the entire training dataset. Training almost never happens in a single epoch; models typically need to see the full dataset many times — sometimes dozens or hundreds of times — before their parameters settle into good values.
• Batch — rather than adjusting parameters after every single example (which would be slow and noisy) or after the entire dataset at once (which would be memory-intensive and slow to react), training usually processes a small group of examples at a time, called a batch, and updates parameters after each one.
• Loss — the numerical measure of how wrong the model’s predictions were, compared to the actual labels. Training’s whole objective, mechanically, is to make this number smaller over time.
• Learning rate — how big a step the model takes when adjusting its parameters after each batch. This is one of the most consequential settings in the entire process, discussed further below.

The underlying mathematical procedure that actually performs the adjustment step, called gradient descent, is the same one referenced in the Algorithm article as the family behind most modern Deep Learning systems, including the Transformer-based models powering GPT, Gemini, and Claude.

ANALOGY vs. TECHNICAL REALITY

Analogy: Picture a golfer trying to sink a putt on a green they can’t fully see, relying only on someone telling them “too far left” or “too far right” after each attempt. Each swing is a small adjustment based on the last mistake. Given enough attempts and honest feedback, they eventually get remarkably accurate — without ever seeing the hole directly.

Where this breaks down: The golfer consciously reasons about direction and force. A model doesn’t reason about anything — the “adjustment” is a precise mathematical calculation (gradient descent) that computes exactly which direction and how far to nudge each of potentially billions of parameters, all simultaneously, every single batch. It’s less like intuition improving with practice and more like an extremely disciplined, automated search procedure.

What happens if you change the key settings

This is a place where beginners often assume “more is always better,” and it’s worth correcting directly:

• Too high a learning rate and the model’s parameters swing wildly with each adjustment, often failing to settle into good values at all — like overcorrecting so hard on the golf green that you send the ball flying past the hole in the opposite direction every time.
• Too low a learning rate and training crawls along, technically working but taking an impractically long time — or getting stuck making tiny, ineffective adjustments.
• Too many epochs and a model can start to memorize the specific training examples rather than learning the general pattern — the overfitting problem introduced in the Dataset article, where a model excels on data it’s already seen and performs poorly on the held-out test set.
• Too few epochs and the model simply hasn’t seen enough repetition to learn the pattern well, a problem called underfitting.

Part of an ML engineer’s real, day-to-day job is tuning exactly these kinds of settings — often called hyperparameters — through careful experimentation, watching how the model performs on the validation set (from the Dataset article) as these settings change.

Why training is expensive, and what companies actually do about it

Training a model like GPT-4 or Gemini from scratch involves running this loop across enormous datasets, for enormous numbers of parameters, on specialized hardware (GPUs or TPUs) running for weeks — a process that can cost tens of millions of dollars and is realistically only undertaken by a handful of major AI labs.

To make that concrete: Meta has confirmed that its Llama 2 models were trained using thousands of Nvidia A100 GPUs running for weeks. GPT-4’s exact training details were never officially confirmed by OpenAI, but widely circulated industry estimates put it at roughly 25,000 GPUs training for around three months, on trillions of words’ worth of text, at an estimated cost in the tens of millions of dollars.

Google has similarly described training Gemini across large clusters of its custom TPU chips rather than GPUs, built specifically to make this kind of large-scale training faster and more efficient.

Treat any specific number you read about frontier model training as a rough, often-unofficial estimate rather than a confirmed fact — these companies rarely publish exact figures — but the scale itself is real and consistent across every credible report: thousands of specialized chips, running continuously for weeks to months, on datasets built from a large fraction of publicly available text on the internet, plus licensed and curated sources.

This is exactly why the Algorithm article emphasized that most companies don’t train large models from scratch. Instead, a very common and far cheaper approach is fine-tuning: taking an already-trained model — one whose parameters have already been shaped by someone else’s massive training run — and running a much smaller, faster round of additional training on your own, specific dataset, nudging its existing parameters to specialize for your particular task.

A company building a customer-support chatbot, for instance, will typically fine-tune an existing large language model on their own support transcripts rather than training an entirely new model from nothing.

Fine-tuning a model this way might take a single GPU a few hours, compared to the thousands of GPUs and weeks of continuous training a frontier lab needs to build the base model in the first place — a difference of several orders of magnitude in both cost and time.

A concrete example, start to finish

Recall the hospital readmission model from earlier articles. Training it means: initializing the chosen algorithm (say, a decision-tree-based one) with essentially blank parameters, feeding it batches of real patient records (features) along with the known readmission outcomes (labels), measuring how far off its early predictions are, and adjusting repeatedly across many passes through the training set — until its predictions on the held-out validation set stop meaningfully improving, at which point training is considered complete and the resulting trained model is ready for the next phase: actually being used.

Key terms

• Epoch: One pass through the training dataset.
• Batch: A group of examples processed together.
• Loss: A numerical measure of training error.
• Gradient: Information describing how a parameter change affects loss.
• Optimizer: The method that updates parameters.
• Learning rate: The size of each update step.

Check your understanding

Does training mean storing every example and looking it up later? No. Training adjusts parameters to encode useful statistical relationships.

Does low training loss prove the model is production-ready? No. It must work on unseen, representative data and satisfy product, safety, latency, and cost requirements.

Common misconception

A common assumption is that training happens once and is simply “finished,” full stop. In practice, as mentioned in the Model article, real-world models often get retrained periodically as new data comes in and the world changes — training isn’t always a single, one-time event, but sometimes an ongoing cycle a company repeats to keep a model’s predictions accurate over time.

Four ways people say a model is being trained

Training activity	What changes
Training from scratch	Parameters begin untrained and learn from a large dataset.
Continued pretraining	An existing model learns further from additional raw-domain data.
Fine-tuning	An existing model is adapted with a smaller task or domain dataset.
Instruction tuning	Instructions and desired responses teach the model to follow requests better.

The same broad loop—predict, measure loss, calculate gradients, update parameters—can appear in all four. The starting checkpoint, data, objective, cost, and number of updated parameters differ.

Where this fits in what comes next

Training is the process that produces a working model. What happens once that model is actually put to use — asked to make a judgment call on new, real-world input it’s never seen — is the subject of the next two articles: Prediction, which covers what a model’s output actually represents, and Inference, which covers the broader process of running a trained model to get that output in a live system.

In one sentence

Training is the repeated, mechanical cycle of guessing, measuring error, and adjusting parameters — usually across many passes through a dataset — that turns a freshly initialized algorithm into a genuinely useful model, and getting its settings right is as much art as it is science.

Related Terms

• Algorithm
• Model
• Dataset
• Machine Learning (ML)
• Prediction