Hyperparameters

The Bias article ended with a preview worth picking up immediately: deciding how many layers a network has, or how many units sit in each layer, isn’t something training figures out on its own — it’s a choice someone makes beforehand. That category of upfront, human-made choice has a name: hyperparameters.

The simple definition

A hyperparameter is a setting an engineer chooses before training begins, rather than a value the model learns automatically during training. This is the cleanest possible contrast to draw against everything covered so far in this phase: Weights and Bias are parameters — numbers training discovers on its own, adjusted automatically to reduce error. Hyperparameters are the opposite: numbers and choices a person sets in advance, which training does not touch or adjust at all.

Parameters versus hyperparameters

Parameters
→ learned by training
→ examples: weights and biases

Hyperparameters
→ chosen around the training process
→ examples: learning rate, batch size, number of epochs

Suppose training uses:

learning rate = 0.01
batch size    = 32
epochs        = 10

These settings influence how training proceeds. Training may then produce millions of learned weights and biases.

Why choices matter

Hyperparameter	Too small	Too large
Learning rate	Learning may be very slow	Training may jump past good values
Batch size	Noisy updates and lower hardware use	High memory use and fewer updates
Epoch count	Undertraining	Overfitting and wasted compute

Teams normally compare choices using validation data, not test data.

A tuning flow

flowchart LR
    A[Choose hyperparameters] --> B[Train on training data]
    B --> C[Measure validation result]
    C --> D[Compare configurations]
    D --> A
    D --> E[Select final configuration]

Real GPT example: published InstructGPT settings

OpenAI’s InstructGPT paper publishes these settings for supervised fine-tuning of its 175B model:

Training concept	Published value
Model parameters	175 billion
Initial learning rate	5.03 × 10⁻⁶
Batch size	8
Epochs	16
Optimizer	Adam
Adam β₁	0.9
Adam β₂	0.95
Weight and activation precision	FP16
Master weight-copy precision	FP32

The learning rate can also be written as:

5.03 × 10⁻⁶ = 0.00000503

That tiny number does not mean the model barely learns. At this scale, even small parameter changes across billions of values can meaningfully change behavior.

These numbers describe one published supervised fine-tuning stage—not GPT-3 pretraining, all RLHF stages, GPT-4, or Gemini training.

Why the “hyper” prefix, and why this distinction matters so much

The prefix hints at the relationship directly: hyperparameters sit “above” or “outside” the parameters — they govern how training happens, rather than being something training produces. Get this distinction solid, because confusing the two is one of the most common early mix-ups in the field: if someone asks “did the model learn that setting?”, the answer is genuinely different depending on whether they mean a weight (yes, learned) or a hyperparameter (no, chosen by a human beforehand).

flowchart LR
    A[Engineer chooses hyperparameters: learning rate, epochs, layers...] --> B[Training runs using those fixed settings]
    B --> C[Weights and biases get automatically adjusted]
    C --> D[Trained Model]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of baking a cake. The oven temperature and baking time are decisions the baker sets in advance, based on experience and the recipe — 350°F for 35 minutes. Once the cake is in the oven, those settings don’t change themselves; the baker fixed them ahead of time. What does change, inside the oven, is the cake itself — rising, browning, setting — through a process governed by, but distinct from, those fixed oven settings.

Where this breaks down: A baker can watch the cake and open the oven door to adjust something mid-bake. In ML, hyperparameters are typically fixed for an entire training run once it starts — an engineer doesn’t reach in and change the learning rate mid-training the way a baker might peek at a cake. Any adjustment happens between training runs: try one set of hyperparameters, see how it performed, choose a different set, and run training again from scratch (or from a saved checkpoint, covered later in this phase).

Common hyperparameters you’ll actually encounter

You’ve already met several of these in passing, in earlier articles, without their proper name attached:

• Learning rate — how big a step the model’s weights take with each adjustment during training, first mentioned in the Training article. Getting this one wrong, as that article explained, can make training unstable or impossibly slow.
• Number of epochs — how many complete passes through the training data to make, also introduced in the Training article.
• Batch size — how many training examples get processed together before each parameter update, briefly mentioned in the Training article and covered fully later in this phase.
• Architecture choices — how many layers a neural network has, and how many units sit in each layer, exactly as previewed at the end of the Bias article. These directly determine, among other things, how many weights and biases the model ends up with.

Each of these is fixed before training starts, stays constant throughout that specific training run, and directly shapes how well — or how badly — the automatic learning of weights and biases actually goes.

Who decides hyperparameters, and how

Unlike weights and biases, which training finds automatically, hyperparameters are chosen by a person — typically an ML engineer or researcher — using a mix of established defaults, experience, and deliberate experimentation.

This is exactly the kind of tuning work described in the Validation Data article: an engineer trains a model with one set of hyperparameters, checks its performance on the validation set, tries a different combination, and repeats — searching for the combination that produces the best validation performance.

This process even has its own name, hyperparameter tuning, and ranges from manual trial-and-error to more systematic, automated search techniques that try many combinations methodically and compare the results.

A concrete example, layered

For a simple beginner example: training a small model to predict house prices, an engineer might choose a learning rate of 0.01 and decide to run 100 epochs — two hyperparameter choices made before training starts, based on common defaults for this kind of problem.

For a production example: OpenAI’s published GPT-3 paper specifies its architecture hyperparameters exactly — 96 transformer layers, a hidden dimension of 12,288, 96 attention heads, and a context window of 2,048 tokens — every one of these a hyperparameter chosen before training began, fixed for the entire run, and together directly responsible for GPT-3 landing at its well-known 175 billion parameter count.

Why getting hyperparameters right matters so much

Two models with identical architecture, trained on identical data, using the exact same learning algorithm, can end up performing very differently purely because of different hyperparameter choices. A learning rate that’s too high can prevent a perfectly good architecture from ever training successfully; a network with too few layers may lack the capacity to learn a genuinely complex pattern, no matter how well everything else is tuned.

This is why experienced ML engineers treat hyperparameter tuning as a serious, often time-consuming part of a project, not an afterthought — the “right” values are usually specific to the particular task and dataset at hand, rather than universal defaults that work everywhere.

Check your understanding

Does ordinary training learn the learning rate as a model weight? Usually no. It is a training setting.

Should the test set choose hyperparameters? No. That would make the final evaluation less honest.

Common misconception

A frequent early mix-up: assuming hyperparameters are somehow less important than the “real” learned parameters, since they’re not what the model directly learns. In practice, hyperparameters set the entire conditions under which learning happens — a brilliant learning algorithm with badly chosen hyperparameters can fail to train at all, or train something far worse than its potential. Weights and biases get all the credit for a model’s final intelligence, but hyperparameters are what determined whether those weights and biases had any real chance of becoming useful in the first place.

Three groups of settings

Setting group	Examples
Architecture choices	Number of layers, hidden width, attention heads
Training hyperparameters	Learning rate, batch size, weight decay, number of steps
Inference settings	Temperature, top-p, maximum output tokens

Inference settings are often casually called hyperparameters, but they do not train the model and do not become learned weights. They control how an already-trained model produces a response.

Where this fits in what comes next

You now understand the crucial split between what training learns (weights, biases) and what a person decides beforehand (hyperparameters). The next several articles in this phase zoom into the actual training mechanics these hyperparameters govern: Loss Function defines what “error” even means for a model to reduce, and Gradient and Gradient Descent cover the precise mathematical procedure — referenced loosely throughout this glossary so far — that actually uses that error to adjust weights and biases.

In one sentence

Hyperparameters are the settings a person chooses before training begins — governing how learning happens, rather than being learned themselves — and getting them right is often just as decisive to a model’s success as the learning process they control.

Related Terms

• Weights
• Bias (model parameter)
• Training
• Validation Data
• Model