Train, Validation and Test Sets

Begin with the central question

How do we know a model learned a pattern instead of memorizing its practice questions?

This question explains why Train, Validation and Test Sets deserves its own lesson. Each definition, calculation, and code example below will answer a specific part of it.

available data → training set | validation set | untouched test set

Before you continue: three tools for this module

Training set: examples used to fit model parameters.
Validation set: separate examples used to make development choices.
Test set: untouched examples used for the final estimate.

You do not need to memorize these yet. Return to this small map whenever a term reappears.

What You Will Understand

Train-Validation-Test Splitting: Learn why data must be split into three distinct subsets, and the precise, isolated role each set plays in modeling.
Cross-Validation: Master how cross-validation yields a more robust, stable estimation of performance than a single, arbitrary data split.
Generalization Realities: Discover why models can appear flawless during development yet degrade severely when exposed to live production traffic.

Each split has one protected job:

training set   → learn model parameters
validation set → compare choices and tune hyperparameters
test set       → one final, untouched estimate
new production data → the real future exam

If information from validation or test data influences training, the estimate is no longer independent. This is data leakage, even when it happens through a preprocessing step rather than direct model training.

Why One Dataset Must Play Three Roles

If you train a model and then evaluate it on the exact same data it was trained on, you’re not measuring whether it learned a general pattern — you’re measuring whether it can recall what it already saw. A model can achieve near-perfect performance on training data simply by memorizing it, without having learned anything that generalizes to new, unseen situations.

Splitting data into separate sets exists specifically to catch this difference.

The Student and the Unseen Exam

Imagine a student who memorizes the exact answers to last year’s exam questions, word for word, without understanding the underlying material. They’ll ace a test that reuses those exact questions — and fail completely the moment the questions are phrased even slightly differently.

Testing a model only on its training data is exactly this flawed exam: it tells you almost nothing about whether real learning happened.

4. Core Concept

Set	Purpose
Training set	Data the model directly learns from — parameters are adjusted using this data
Validation set	Data used to tune hyperparameters and make development decisions, without touching the test set
Test set	Data used exactly once, at the very end, to get an honest, final estimate of real-world performance

                    ┌───────────────┐
Full Dataset  ───→  │  Training Set │  ───→  model learns from this
                    ├───────────────┤
                    │ Validation Set│  ───→  tune hyperparameters, compare models
                    ├───────────────┤
                    │   Test Set    │  ───→  final, one-time honest evaluation
                    └───────────────┘

A typical split might be 70% training / 15% validation / 15% test — though exact proportions vary by dataset size and problem.

🧠 Intuition for why three sets, not two: If you only had train and test, and you kept tweaking your model based on test-set performance, the test set would slowly stop being a fair, unseen measure — you’d be indirectly “training” on it through your own repeated decisions. The validation set exists specifically to protect the test set’s honesty, by absorbing all that iterative tuning instead.

5. How It Works — Step by Step

1. Split the full dataset into train / validation / test
2. Train the model using ONLY the training set
3. Evaluate on the validation set
4. Adjust hyperparameters / try different models based on validation results
5. Repeat steps 2-4 as many times as needed
6. Once satisfied, evaluate ONCE on the test set
7. The test set result is your honest, final estimate of real-world performance
   — do NOT go back and tune further based on it

Random splitting vs. stratified splitting

Random splitting — just randomly assign samples to each set. Works fine when classes/categories are reasonably balanced.

Stratified splitting — ensures each set has the same proportion of each class/category as the full dataset. Essential when classes are imbalanced.

# Imagine a fraud dataset: 99% legitimate, 1% fraud
# A purely random split MIGHT accidentally put almost all fraud
# examples into the training set, leaving the test set with none
# to evaluate on — stratified splitting prevents this

Cross-validation

Instead of a single train/validation split, K-fold cross-validation splits the training data into K equal parts (“folds”), trains K times — each time using a different fold as validation and the rest as training — and averages the results.

Fold 1: [VALID][train][train][train][train]
Fold 2: [train][VALID][train][train][train]
Fold 3: [train][train][VALID][train][train]
Fold 4: [train][train][train][VALID][train]
Fold 5: [train][train][train][train][VALID]

Final score = average of all 5 validation results

🧠 Intuition: A single validation split gives you one estimate of performance, which could be lucky or unlucky depending on which samples happened to land in it. Cross-validation averages across many different splits, giving a far more reliable, less luck-dependent estimate — at the cost of training the model K times instead of once.

6. Mathematical Intuition

Read the mathematics as a story

available data → training set | validation set | untouched test set

First identify the input, the operation, and the output. Then read the symbols as a shorter way to describe that same journey; do not begin by memorizing the formula.

No heavy math needed — just the arithmetic of K-fold averaging:

scores = [0.82, 0.79, 0.85, 0.81, 0.80]   # validation accuracy from each of 5 folds
average_score = sum(scores) / len(scores)  # = 0.814

The average is more trustworthy than any single fold’s score, and the spread between fold scores (how much they vary) tells you something extra: a tight spread means stable, reliable performance; a wide spread means the model’s performance is sensitive to exactly which data it sees — worth investigating further.

7. Small Worked Example

Walk through the example

Identify what each input number represents.
Follow one operation at a time and keep the units or class meanings attached.
Translate the result back into an ordinary sentence about the original problem.

The goal is not merely to obtain the answer; it is to expose the model’s decision process.

Imagine 10 labeled examples. With a 60/20/20 split:

Training (6 examples):    used to fit the model
Validation (2 examples):  used to check "should I use K=3 or K=5 for KNN?"
Test (2 examples):        used ONCE, at the very end, for the final honest score

If the model scores 95% on training but only 60% on validation, that gap itself is the signal — something is wrong (overfitting, covered fully in Module 6) well before you’d ever even touch the test set.

8. Python Example

What the code will demonstrate

The following Train, Validation and Test Sets code turns the worked example into an experiment you can repeat. First predict the result; then prepare the small dataset, apply the technique, inspect the important intermediate values, and compare the actual output with your prediction.

Python and library symbols used below

NumPy (np) stores and calculates with numeric arrays.
pandas (pd) represents table-shaped data when it is used.
scikit-learn provides tested implementations with a consistent .fit(...) and .predict(...) workflow.

# Build a small, inspectable example of Train, Validation and Test Sets.
# Follow the data, learned values, predictions, and evaluation in order.
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
import numpy as np

# Simulated features and labels (imbalanced: mostly 0s, few 1s)
np.random.seed(42)
X = np.random.rand(100, 3)
y = np.array([0]*90 + [1]*10)   # imbalanced classes

# --- Stratified train/test split ---
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
print("Training class balance:", np.bincount(y_train))
print("Test class balance:", np.bincount(y_test))

# --- K-fold cross-validation on the training set ---
model = LogisticRegression()
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=skf, scoring="accuracy")

print("\nFold scores:", scores)
print("Average CV accuracy:", scores.mean())

# --- Final, ONE-TIME evaluation on the untouched test set ---
model.fit(X_train, y_train)
test_accuracy = model.score(X_test, y_test)
print("\nFinal test accuracy (one-time, honest estimate):", test_accuracy)

Expected Output (approximate — exact numbers vary by environment):

Training class balance: [72  8]
Test class balance: [18  2]
Fold scores: [0.9375 0.875  0.9375 0.9375 0.875 ]
Average CV accuracy: 0.9125

Final test accuracy (one-time, honest estimate): 0.9

How It Works

stratify=y in train_test_split attempts to preserve the full dataset’s class proportions in both subsets. Here it keeps the ratio close to 90/10. With very small classes, integer rounding—or too few examples to place in every split—can limit what is possible.
StratifiedKFold does the same stratification, fold by fold, during cross-validation.
Notice model.fit(X_train, y_train) and model.score(X_test, y_test) are called exactly once, at the very end — mirroring the rule from Section 5: the test set is touched only once, for a final honest number.

9. Real-World Example

A team builds a model to predict loan default risk using five years of historical loan data.

Training set: the majority of historical loans, used to fit the model.
Validation set (or cross-validation): used to compare several candidate models (logistic regression vs. a gradient-boosted tree) and tune their hyperparameters.
Test set: a final, untouched slice — often deliberately chosen as the most recent loans, to simulate “how would this model perform on loans it genuinely hasn’t seen yet,” which is a closer proxy to real production conditions than a purely random split would be.

10. How This Is Used in AI

From mechanism to product

Large AI teams preserve held-out evaluations for the same reason: repeatedly checking the test set turns it into another validation set and makes reported quality too optimistic.

How this connects to LLMs

request → data or context preparation → model computation → evaluated output

An LLM may use this idea during training, or an AI application may use a separate ML component around the LLM. Those are different locations in the system, and the explanation below identifies which one applies.

🤖 How Is This Used in AI?

ML Concept	AI Equivalent
Training set	The corpus an LLM is pretrained or fine-tuned on
Validation set	Held-out data used during LLM training to decide when to stop training, or to compare checkpoints
Test set	A benchmark dataset (e.g., MMLU, HumanEval) used to report a model’s final capabilities
Cross-validation	Less common at LLM-scale (training is too expensive to repeat 5x), but still used for smaller classifiers/rerankers in AI pipelines
Data leakage between splits	Benchmark contamination — when benchmark questions (or very similar ones) accidentally appear in an LLM’s training data, inflating its reported benchmark score

🧠 RAG evaluation, specifically: a RAG system’s “test set” is a set of realistic questions with known-correct answers or known-relevant documents, held separate from anything used to tune the retrieval or prompting strategy.

If you tune your chunking strategy by repeatedly checking performance against the same evaluation questions you’ll later report results on, you’ve reintroduced exactly the train/test contamination problem this module warns about — just in a RAG context instead of classic ML.

🤖 Agent evaluation: the same discipline applies — a set of held-out tasks the agent has never been tuned against gives an honest measure of how well it generalizes, versus tasks used repeatedly during development to iterate on the agent’s prompts/tools, which will look artificially good.

11. How This Is Used in Agentic AI

Trace one agent step

goal + state → model proposes → runtime validates → tool or response → evaluation

The model produces a prediction or proposal. The agent runtime is ordinary software that manages tools, permissions, state, retries, and execution; it may use this ML concept directly, indirectly through an LLM, or not at all.

🤖 When building or improving an agent, it’s tempting to iterate directly against the same handful of test tasks you use for every debugging session — but doing so means your “it’s working now!” impression is built on data the agent’s prompts/tools have effectively been tuned against, exactly like overfitting to a validation set.

A disciplined agent development process keeps a genuinely held-out set of tasks — untouched during iteration — for a final, honest read on whether changes actually generalize, not just whether they fix the specific cases you happened to be staring at.

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: Tuning hyperparameters using the test set

Why it is incorrect: The moment you make decisions based on test-set performance and then report that same performance as your final result, it’s no longer an honest, unseen evaluation — you’ve leaked information from the test set into your model-selection process.

⚠️ Mistake

Incorrect idea: Preprocessing before splitting

Why it is incorrect: Computing normalization statistics (like mean/standard deviation, Module 5) using the entire dataset, and only then splitting into train/test — this leaks information about the test set’s distribution into the training process. Always split first, then compute preprocessing statistics using only the training set.

⚠️ Mistake

Incorrect idea: Random (non-stratified) splitting on imbalanced data

Why it is incorrect: With rare classes, a random split can easily leave the test set with very few (or zero) examples of the minority class, making evaluation results unreliable or meaningless for that class.

13. Important Distinctions

Validation Set	Test Set
Used repeatedly, throughout development	Used once, at the very end
Guides model/hyperparameter decisions	Provides a final, honest performance estimate
Can indirectly “leak into” the model through repeated tuning	Should remain genuinely unseen until final evaluation

Random Splitting	Stratified Splitting
Assigns samples to sets purely randomly	Preserves class proportions across all sets
Fine for balanced data	Important for imbalanced data

14. When Should You Use This?

Always split data into at least train and test sets for any supervised model you intend to evaluate honestly.
Use a validation set or cross-validation whenever you need to make any decision (choice of model, hyperparameters, feature set) based on performance — never make those decisions using the test set.
Use cross-validation specifically when your dataset is small enough that a single validation split would give a noisy, unreliable estimate, or when you want a more robust performance estimate before committing to a final model.
Use stratified splitting whenever classes are imbalanced, or more generally as a safe default even when they’re not.

15. When Should You NOT Use This?

Cross-validation is often impractical at LLM training scale — training a foundation model even once is extremely expensive; repeating it 5-10x for cross-validation is rarely realistic. Large-scale training instead typically relies on a single, carefully-chosen held-out validation set.
For time-series data (e.g., stock prices, sensor readings), naive random splitting (or standard K-fold) can leak future information into training — a specialized time-based split (train on earlier data, test on strictly later data) is needed instead.

16. Production Considerations

Test set staleness — a test set built years ago may no longer reflect current real-world data patterns; periodically refresh it.
Reporting discipline — teams should agree on a process where the test set is genuinely touched only once (or on a strict, infrequent cadence) to preserve its honesty as an evaluation signal over the model’s lifetime.
Realistic splitting strategy — for production systems, consider splitting by time (train on past, test on more recent data) rather than purely randomly, since this better simulates the real deployment scenario: predicting on data that genuinely didn’t exist during training.

17. AI Engineer Takeaway

🎯 AI Engineer Takeaway: The gap between training performance and real-world performance is one of the most consequential and recurring problems in ML and AI — and disciplined data splitting is the primary defense against being fooled by it.

The exact same discipline — “don’t let your evaluation data quietly influence what you’re evaluating” — reappears throughout modern AI as benchmark contamination, RAG evaluation-question reuse, and agent tuning against the same handful of test tasks. Recognizing the pattern once means recognizing it everywhere.

18. Interview Questions

Basic Questions

Q: Why do we split data into training, validation, and test sets instead of just training and testing?

A: If you only had train and test sets, iteratively tuning your model based on test-set results would slowly compromise the test set’s honesty — you’d effectively be training on it indirectly through your own repeated decisions. The validation set absorbs all that iterative tuning, keeping the test set genuinely unseen until one final, honest evaluation.

Q: What is K-fold cross-validation, and why use it instead of a single train/validation split?

A: K-fold cross-validation splits the training data into K parts, trains K separate times (each time validating on a different part while training on the rest), and averages the results. It gives a more reliable performance estimate than a single split, because a single split’s result can be skewed by which specific samples happened to land in the validation set — averaging across K different splits reduces that luck factor.

Intermediate Questions

Q: Why is stratified splitting important for imbalanced datasets?

A: With imbalanced classes (e.g., 1% fraud, 99% legitimate), a purely random split risks leaving very few — or even zero — minority-class examples in one of the sets, especially the test set, which makes evaluating performance on that class unreliable or outright impossible. Stratified splitting preserves the original class proportions across every split, so each set remains representative of the real class balance.

Q: What’s wrong with computing normalization statistics (like mean and standard deviation for scaling) on the full dataset before splitting it into train and test?

A: This leaks information about the test set into the training process — the model’s preprocessing has now been influenced, even indirectly, by data it should never have seen. The correct order is: split first, then compute any preprocessing statistics using only the training set, then apply those same statistics to transform the validation and test sets.

Scenario-Based Questions

Q: A model achieves 96% accuracy on the test set, but performs noticeably worse once deployed to real users. The team is confident the split was done correctly and there’s no data leakage. What else might explain the gap?

A: Thought process: If leakage and split correctness are genuinely ruled out, the next most likely explanation is a mismatch between the test set’s data distribution and production’s real-world data distribution.

Investigation: Check when the test data was collected relative to when the model is being deployed — real-world patterns can shift over time (data drift, covered fully in Module 21). Also check whether the test set was collected from the same population/conditions as production traffic — for example, a model tested on data from one region or user segment may perform differently on a broader, more diverse real-world population.

Correct answer: The most likely explanations are data drift (the world has changed since the test data was collected) or a distribution mismatch (the test set wasn’t fully representative of real production traffic) — not a flaw in the train/validation/test methodology itself, which the team has already ruled out.

Production consideration: This is exactly why production ML systems need ongoing monitoring, not just a one-time pre-deployment test set evaluation — a test set is a snapshot in time, and production performance needs to be tracked continuously as real-world conditions evolve.

Next: Module 05 — Feature Engineering and Preprocessing — scaling, encoding, and the shift from hand-crafted features toward learned representations (embeddings).

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026