Hyperparameters and Model Selection

Begin with the central question

Who chooses the settings that the model does not learn for itself?

This question explains why Hyperparameters and Model Selection deserves its own lesson. Each definition, calculation, and code example below will answer a specific part of it.

candidate settings → train models → validation comparison → selected configuration → final test

Before you continue: three tools for this module

Parameter: a value learned from training data.
Hyperparameter: a setting chosen outside the learning process.
Cross-validation: repeated train/validation splits for a steadier comparison.

You do not need to memorize these yet. Return to this small map whenever a term reappears.

What You Will Understand

Hyperparameter Optimization: Discover the differences between parameters (weights updated via gradients) and hyperparameters (algorithm settings chosen by engineers).
Grid & Random Search: Master systematic tuning techniques using cross-validation to search hyperparameter spaces without leaking validation data.
Cost-Benefit Trade-offs: Gain the practical judgment to determine when hyperparameter tuning is worth the computational cost and when it represents diminishing returns.

Training learns parameters; engineers choose hyperparameters:

candidate hyperparameters
        ↓ train on training folds
validation score compares candidates
        ↓ select one configuration
retrain as planned → evaluate once on untouched test set

Trying many choices can itself overfit the validation process. The test set must remain outside model and hyperparameter selection if it is to provide an honest final estimate.

Why Model Settings Need a Fair Selection Process

Every model in this course has hyperparameters — settings chosen before training that meaningfully affect how training happens and how well the final model performs. Picking good hyperparameter values by hand, through guesswork, is unreliable and inefficient. Systematic hyperparameter tuning exists to replace guesswork with a structured, repeatable search process for finding hyperparameter values that genuinely perform well.

Choosing an Oven Setting Without Tasting the Final Cake

Think of hyperparameters as the settings on an oven before you bake something — temperature, rack position, bake time. The recipe (your model type) is fixed, but these settings dramatically affect the outcome, and the “right” settings often depend on the specific dish (dataset) you’re making.

Hyperparameter tuning is the systematic process of trying different oven settings and seeing which one actually produces the best result — rather than just guessing once and hoping.

4. Core Concept

A recap and consolidation of hyperparameters across this course

Algorithm	Key hyperparameters
Linear/Logistic Regression	Regularization strength (Module 16)
Decision Trees	`max_depth`, minimum samples per split/leaf
Random Forest	`n_estimators` (number of trees), `max_depth`
Gradient Boosting	`n_estimators`, `learning_rate`, `max_depth`
KNN	`K` (number of neighbors)
SVM	Kernel type, regularization strength, kernel-specific parameters
K-Means	`K` (number of clusters)
Neural networks / LLM fine-tuning	Learning rate, batch size, number of epochs

🧠 The consistent theme: every algorithm you’ve learned has at least one setting that isn’t learned from data — it’s a choice the engineer makes, and that choice materially affects the bias-variance trade-off (Module 6) and overall model quality.

Grid Search vs. Random Search

Grid Search:      try EVERY combination of a predefined
                   set of hyperparameter values

Random Search:     try a RANDOM sample of combinations from
                    the hyperparameter space, for a fixed
                    "budget" of attempts

Grid search, e.g., for a random forest:
  n_estimators: [50, 100, 200]
  max_depth:    [3, 5, 10]
  → tries all 3 × 3 = 9 combinations, exhaustively

Random search, same space:
  → randomly samples, say, 15 combinations from the
    same possible range, without trying every single one

🧠 Why random search is often preferred over grid search at scale: grid search’s cost grows multiplicatively with each additional hyperparameter (3 hyperparameters with 5 values each = 125 combinations) — quickly becomes computationally infeasible.

Random search, given the same computational budget, often finds comparably good (sometimes better) hyperparameter values, because it explores the space more broadly rather than exhaustively covering a predefined grid that may waste effort on unpromising regions.

5. How It Works — Step by Step

1. Define the hyperparameter search space (which values/ranges
   to consider for each hyperparameter)
2. Choose a search strategy (grid search, random search, or
   more advanced methods like Bayesian optimization)
3. For EACH candidate hyperparameter combination:
   a. Train a model using those hyperparameters
   b. Evaluate it using cross-validation (Module 4) — NOT the test set
4. Select the hyperparameter combination with the best
   cross-validation performance
5. Train a FINAL model using those chosen hyperparameters on
   the full training set
6. Evaluate this final model ONCE on the held-out test set
   (recall Module 4's discipline — this is the honest, final check)

🧠 This directly reuses Module 4’s validation/cross-validation machinery — hyperparameter tuning is precisely the kind of “iterative decision-making” that validation sets/cross-validation exist to support, keeping the test set genuinely untouched until the very end.

6. Mathematical Intuition

Read the mathematics as a story

candidate settings → train models → validation comparison → selected configuration → final test

First identify the input, the operation, and the output. Then read the symbols as a shorter way to describe that same journey; do not begin by memorizing the formula.

No new formulas here — the “math” of hyperparameter search is really just counting:

Grid search combinations = (values for hp1) × (values for hp2) × ... × (values for hpN)

Example: 4 hyperparameters, each with 5 candidate values
= 5 × 5 × 5 × 5 = 625 total combinations to train and evaluate

If training one model takes 10 minutes, a full grid search here would take over 100 hours — a concrete illustration of why this cost grows unmanageable quickly, and why random search (or smarter methods) become necessary at larger scale.

7. Small Worked Example

Walk through the example

Identify what each input number represents.
Follow one operation at a time and keep the units or class meanings attached.
Translate the result back into an ordinary sentence about the original problem.

The goal is not merely to obtain the answer; it is to expose the model’s decision process.

Tuning KNN’s K for a small classification dataset, using 5-fold cross-validation:

K	Average CV accuracy
1	0.78
3	0.85
5	0.88
7	0.86
15	0.79

The pattern here is classic bias-variance behavior (Module 6): very small K (K=1) overfits to local noise (high variance, lower accuracy); very large K (K=15) oversmooths and underfits (high bias, lower accuracy); K=5 hits the sweet spot for this particular dataset. This kind of systematic sweep — trying several values and picking the best via cross-validation — is hyperparameter tuning in its simplest form.

8. Python Example

What the code will demonstrate

The following Hyperparameters and Model Selection code turns the worked example into an experiment you can repeat. First predict the result; then prepare the small dataset, apply the technique, inspect the important intermediate values, and compare the actual output with your prediction.

Python and library symbols used below

NumPy (np) stores and calculates with numeric arrays.
pandas (pd) represents table-shaped data when it is used.
scikit-learn provides tested implementations with a consistent .fit(...) and .predict(...) workflow.

# Build a small, inspectable example of Hyperparameters and Model Selection.
# Follow the data, learned values, predictions, and evaluation in order.
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np

X, y = make_classification(n_samples=500, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# --- Grid Search: exhaustively try every combination ---
param_grid = {
    "n_estimators": [50, 100, 200],
    "max_depth": [3, 5, 10],
}
grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=5,                    # 5-fold cross-validation for each combination
    scoring="accuracy",
)
grid_search.fit(X_train, y_train)

print("Grid search - best params:", grid_search.best_params_)
print("Grid search - best CV score:", grid_search.best_score_)

# --- Random Search: sample a fixed number of random combinations ---
param_distributions = {
    "n_estimators": [50, 100, 150, 200, 250],
    "max_depth": [3, 5, 7, 10, 15, None],
}
random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_distributions,
    n_iter=10,               # only try 10 random combinations, not all
    cv=5,
    scoring="accuracy",
    random_state=42,
)
random_search.fit(X_train, y_train)

print("\nRandom search - best params:", random_search.best_params_)
print("Random search - best CV score:", random_search.best_score_)

# --- Final, ONE-TIME evaluation on the untouched test set ---
best_model = grid_search.best_estimator_
test_accuracy = best_model.score(X_test, y_test)
print("\nFinal test accuracy (one-time, honest estimate):", test_accuracy)

Expected Output (approximate — exact numbers vary by environment):

Grid search - best params: {'max_depth': 10, 'n_estimators': 100}
Grid search - best CV score: 0.9175

Random search - best params: {'n_estimators': 150, 'max_depth': 10}
Random search - best CV score: 0.915

Final test accuracy (one-time, honest estimate): 0.92

How It Works

GridSearchCV and RandomizedSearchCV both automatically combine hyperparameter search with cross-validation (Module 4) internally — every candidate combination is evaluated using proper cross-validation, not a single lucky/unlucky split.
Notice random search found a comparably good result (0.915 vs. 0.9175) while only trying 10 combinations instead of grid search’s full 9 — in this small example the difference is minor, but at larger hyperparameter spaces, this efficiency gap becomes far more significant.
best_estimator_ retrieves the actual trained model using the winning hyperparameters — evaluated on the test set exactly once, exactly matching Module 4’s discipline.

9. Real-World Example

A team building a gradient boosting model for credit risk scoring needs to tune n_estimators, max_depth, and learning_rate — three interacting hyperparameters where the “right” combination genuinely depends on the others (a higher learning rate might need fewer estimators to avoid overfitting, for instance).

Given limited compute budget and time constraints, they use random search with a reasonable iteration budget, combined with 5-fold cross-validation for each candidate, rather than attempting an exhaustive grid search across all three hyperparameters simultaneously — a decision that reflects a real, practical trade-off between thoroughness and available time/compute.

10. How This Is Used in AI

From mechanism to product

Model selection applies to classical ML and LLM applications: teams compare model versions, prompts, retrieval settings, and decoding options using representative validation tasks.

How this connects to LLMs

request → data or context preparation → model computation → evaluated output

An LLM may use this idea during training, or an AI application may use a separate ML component around the LLM. Those are different locations in the system, and the explanation below identifies which one applies.

🤖 How Is This Used in AI?

Direct relevance to Agentic AI: Moderate, primarily through fine-tuning configuration.

ML Concept	AI Equivalent
Hyperparameter tuning	Choosing learning rate, batch size, and number of epochs when fine-tuning an LLM
Grid/random search	Less commonly done exhaustively for LLM fine-tuning (each run is expensive) — often replaced by a small number of carefully-chosen configurations based on established best practices or prior experience
Model selection	Choosing between candidate LLM providers/model sizes for a given task, based on evaluation results, cost, and latency trade-offs
Cross-validation-guided tuning	Used at smaller scale for auxiliary classifiers, rerankers, or other supporting ML components in an AI pipeline

🧠 Why blindly tuning everything is a bad engineering strategy — directly connected to AI system design: exhaustively tuning every possible hyperparameter for an expensive-to-train model (especially LLM fine-tuning, where each run can cost real money and take real time) is often simply not economically sensible.

A more practical AI-engineering approach: start with well-established default hyperparameters (often provided by the model/framework documentation, based on what’s worked well for similar tasks), and only invest in more extensive tuning when there’s clear evidence the defaults are meaningfully underperforming for your specific use case.

11. How This Is Used in Agentic AI

Trace one agent step

goal + state → model proposes → runtime validates → tool or response → evaluation

The model produces a prediction or proposal. The agent runtime is ordinary software that manages tools, permissions, state, retries, and execution; it may use this ML concept directly, indirectly through an LLM, or not at all.

🤖 When fine-tuning a smaller model for a specific agent capability (e.g., a specialized tool-selection classifier, or a lightweight intent router), the tuning principles in this module apply directly and are genuinely practical to run at that smaller scale — cross-validation and random/grid search over a handful of hyperparameters is entirely feasible for these smaller, cheaper-to-train components, even when it wouldn’t be practical for a full LLM fine-tuning run.

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: Tuning hyperparameters using the test set instead of cross-validation or a separate validation set

Why it is incorrect: This is exactly Module 4’s leakage warning, reapplied here: doing so compromises the test set’s honesty as a final, unbiased performance estimate.

⚠️ Mistake

Incorrect idea: Tuning far more hyperparameters than the dataset size can reliably support

Why it is incorrect: With a small dataset, extensive hyperparameter search increases the risk of finding a combination that happens to perform well on your specific validation folds by chance, rather than genuinely generalizing — a subtle form of overfitting to the validation process itself.

⚠️ Mistake

Incorrect idea: Assuming more tuning always helps

Why it is incorrect: Beyond a certain point, additional hyperparameter search yields diminishing (or even negative, due to validation-set overfitting) returns, while continuing to consume real time and compute — this is precisely the “blindly tuning everything is a bad engineering strategy” warning this module opens with.

13. Important Distinctions

Grid Search	Random Search
Exhaustively tries every combination	Samples a fixed number of random combinations
Guaranteed to find the best combination within the defined grid	Not guaranteed, but often nearly as good, for much less compute
Cost grows multiplicatively with more hyperparameters	Cost stays fixed regardless of how many hyperparameters/values are considered

Parameters	Hyperparameters
Learned automatically during training	Chosen manually, before training, by the engineer
Different every training run (usually)	Set explicitly, often via a tuning process like this module covers

14. When Should You Use This?

Whenever a model’s performance is meaningfully sensitive to its hyperparameters (most non-trivial models), and you have the computational budget to systematically search for better values.
Grid search: when the hyperparameter space is small (few hyperparameters, few candidate values each), and you want a guaranteed, exhaustive search.
Random search: when the hyperparameter space is large, or compute budget is limited, and a “good enough, efficiently found” combination is preferable to an exhaustive but expensive search.
Always combine hyperparameter tuning with cross-validation (Module 4), never with the test set directly.

15. When Should You NOT Use This?

Don’t invest in extensive hyperparameter tuning before first ensuring your data quality, features, and overall modeling approach are sound (Modules 3, 5, 6) — tuning a fundamentally flawed pipeline’s hyperparameters is a poor use of effort compared to fixing the underlying issues first.
Don’t run extensive hyperparameter search on an extremely small dataset — as Section 12 notes, this risks overfitting to the validation process itself, producing hyperparameter choices that don’t genuinely generalize.
For very expensive training runs (like LLM fine-tuning), exhaustive search is often simply impractical — lean on established best-practice defaults and targeted, limited experimentation instead.

16. Production Considerations

Compute budget awareness — always estimate the total cost (time, money) of a planned hyperparameter search before starting it; an unconstrained search can silently consume far more resources than intended.
Diminishing returns tracking — monitor whether additional tuning iterations are meaningfully improving cross-validation performance; stop once gains become negligible relative to continued cost.
Reproducibility — record the exact hyperparameters (and random seeds, where relevant) used for your final chosen model, so results can be reproduced and audited later.

17. AI Engineer Takeaway

🎯 AI Engineer Takeaway: Hyperparameter tuning is a systematic search process, not guesswork — but it’s also genuinely expensive, and the engineering judgment of how much tuning effort is actually worth it (given compute cost, dataset size, and how sensitive the model actually is to its hyperparameters) matters as much as the search technique itself.

This judgment call becomes directly practical the moment you fine-tune an LLM: you’ll be choosing a learning rate, batch size, and number of epochs yourself, almost always guided by established defaults and limited, targeted experimentation rather than an exhaustive grid search — because at LLM scale, exhaustive tuning is rarely economically sensible.

18. Interview Questions

Basic Questions

Q: What is the difference between a parameter and a hyperparameter?

A: (Revisited from Module 1, with more concrete grounding now.) A parameter is learned automatically by the model during training — you never set it directly. A hyperparameter is a setting chosen by the engineer before training begins, controlling how the training process itself behaves (e.g., learning rate, tree depth, number of neighbors) — and it’s exactly what hyperparameter tuning searches over.

Q: What is grid search?

A: Grid search is a hyperparameter tuning method that exhaustively tries every combination of a predefined set of hyperparameter values, training and evaluating a model for each combination (typically using cross-validation), and selecting whichever combination performed best.

Intermediate Questions

Q: Why might random search be preferred over grid search when tuning several hyperparameters simultaneously?

A: Grid search’s computational cost grows multiplicatively with each additional hyperparameter or candidate value — quickly becoming impractical as the search space grows. Random search samples a fixed number of combinations from the same space, and in practice often finds comparably good (sometimes better) hyperparameter values for a given compute budget, because it explores the space more broadly rather than exhaustively covering every combination, many of which may turn out to be unpromising.

Q: Why is it important to use cross-validation (rather than the test set) when performing hyperparameter tuning?

A: Using the test set to guide hyperparameter choices means you’re indirectly “training” on the test set through your own repeated decisions — this compromises its ability to provide an honest, unbiased final performance estimate (exactly Module 4’s leakage concern). Cross- validation (or a separate validation set) exists precisely to absorb this iterative decision-making, keeping the test set genuinely untouched until one final, honest evaluation.

Scenario-Based Questions

Q: A team wants to fine-tune a large language model for their specific use case and asks whether they should run a full grid search over learning rate, batch size, and number of epochs, similar to what they’d do for a smaller scikit-learn model. How would you advise them?

A: Thought process: This question is directly testing whether the hyperparameter-tuning mindset from smaller classical ML models transfers naively to LLM fine-tuning, or whether the cost realities of LLM training change the practical calculus.

Investigation: Each fine-tuning run of an LLM is dramatically more expensive (in both time and direct cost) than training a small classical ML model — running dozens or hundreds of combinations, as a full grid search would require, is very likely economically impractical for most teams. Additionally, LLM fine-tuning best practices are relatively well-established in current documentation and community experience, providing reasonable starting defaults that don’t require rediscovering from scratch via exhaustive search.

Correct answer: Advise against a full grid search. Instead, recommend starting with well-established default hyperparameters for their specific fine-tuning method and model size, running a small number of targeted experiments (perhaps 2-4 configurations varying just the parameter most likely to matter for their use case, such as learning rate), and evaluating each against a genuinely held-out validation set — a pragmatic, budget-conscious middle ground between “no tuning at all” and “exhaustive grid search.”

Production consideration: This decision should explicitly factor in the actual cost of each fine-tuning run (compute cost, and the API/platform’s pricing if using a managed fine-tuning service) against the expected performance gain from additional tuning — for many practical business use cases, the gain from moving past well-chosen defaults is often smaller than the cost of extensive additional search, especially compared to the fundamentals-first advice in Section 15: getting the fine-tuning data quality right (Module 3) usually matters far more than hyperparameter perfection.

Next: Module 16 — Regularization — why regularization exists, L1 vs. L2, dropout, and early stopping, and how these directly connect to preventing overfitting in neural networks and fine-tuning.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026