Feature Engineering and Preprocessing

Begin with the central question

Why can the same algorithm succeed or fail depending on how its inputs are prepared?

This question explains why Feature Engineering and Preprocessing deserves its own lesson. Each definition, calculation, and code example below will answer a specific part of it.

raw records → clean/encode/scale/create features → model-ready table

Before you continue: three tools for this module

Preprocessing: making raw data consistent and usable.
Encoding: turning categories or text into numbers.
Scaling: putting numeric features on comparable ranges.

You do not need to memorize these yet. Return to this small map whenever a term reappears.

What You Will Understand

Feature Preprocessing: Understand how to prepare numerical and categorical data using scaling, standardization, and one-hot encoding without introducing leakage.
Precomputation & Storing: Master the engineering discipline of saving fitted scaler and encoder parameters alongside the trained model for production serving.
Representation Learning: Trace the paradigm shift from manually engineered features to representation learning (embeddings) where models learn features automatically.

Raw values must become model-ready features:

raw age, income, country
          ↓ fit preprocessing on training data only
scaled age + scaled income + encoded country
          ↓
model-ready numeric row

The fitted preprocessing rules are part of the model system. Production input must use the same column order, category mapping, and scaling values used during training.

Why Raw Data Often Cannot Enter a Model Directly

Most ML models don’t work directly on raw data. A model like linear regression or KNN (Module 10) is sensitive to the scale of numbers — a feature ranging 0-1,000,000 (income) will completely dominate a feature ranging 0-1 (a ratio) in distance-based calculations, even if the smaller feature is actually more predictive.

And most models can’t handle text categories ("India", "USA") directly at all — they need numbers. Preprocessing exists to translate messy, inconsistently-scaled, mixed-type raw data into a form a model can learn from fairly and effectively.

Putting Features on a Fair Measuring Scale

Imagine comparing two job candidates using “years of experience” (0-40) and “interview score out of 5” (0-5) without adjusting for scale — a naive comparison would let tiny differences in years of experience completely overwhelm huge differences in interview score, just because the numbers are bigger.

Scaling puts features on a level playing field so a model judges them by actual importance, not by which one happens to use bigger numbers.

4. Core Concept

Term	Definition
Scaling	Adjusting numerical features to a common range or distribution
Normalization	In this module, min-max scaling into a fixed range, typically [0, 1]; in vector work, the same word can instead mean scaling each vector to unit length
Standardization	Rescaling values to have mean 0 and standard deviation 1
One-hot encoding	Representing a category as a set of binary (0/1) columns, one per possible category
Label encoding	Representing a category as a single integer (e.g., “red”=0, “blue”=1, “green”=2)
Feature selection	Choosing which existing features to keep, discarding unhelpful ones
Feature extraction	Deriving new features from existing raw data (e.g., extracting “day of week” from a timestamp)

Normalization vs. standardization, formulas explained plainly

Here, normalization means min-max scaling. In embedding and vector-search contexts, “normalize a vector” commonly means divide it by its length so its magnitude becomes 1. Always check which meaning a library or document uses.

Normalization:    x_scaled = (x - min) / (max - min)        → squeezes values into [0, 1]
Standardization:  x_scaled = (x - mean) / std_deviation      → centers around 0, unit spread

x = the original value
min, max = the smallest/largest value of that feature in the training data
mean, std_deviation = the average and spread of that feature in the training data

🧠 When to use which: normalization is intuitive and bounded, useful when you want training values within a known range. Standardization is often used for models whose optimization or geometry benefits from centered features.

Both methods are affected by outliers: min-max scaling can compress most values when an extreme minimum or maximum exists, while the mean and standard deviation used by standardization can also be pulled by extreme values. Robust scaling is another option when outliers are important.

One-hot vs. label encoding, concretely

Category column: ["red", "blue", "green"]

Label encoding:      red=0, blue=1, green=2
                      → implies an ORDER (2 > 1 > 0) that doesn't
                        actually exist between colors!

One-hot encoding:     is_red   is_blue   is_green
                          1        0         0
                          0        1         0
                          0        0         1
                      → no false ordering implied

Label encoding is only appropriate for ordinal categories that have a genuine natural order (e.g., "low", "medium", "high") — using it for unordered categories (colors, countries) can mislead models that interpret the numbers as having a real magnitude relationship.

5. How It Works — Step by Step

1. Identify feature types           → numerical, categorical, text, datetime, etc.
2. Handle missing values             → drop or impute (Module 3)
3. Handle outliers                   → investigate, cap, or transform (Module 3)
4. Scale numerical features          → normalization or standardization
5. Encode categorical features       → one-hot or label encoding
6. Engineer new features             → derive more useful signals from raw data
7. Select relevant features          → drop unhelpful/redundant ones
8. Fit all preprocessing steps       → using ONLY the training set (Module 4!)
9. Apply the SAME fitted transforms  → to validation and test sets

🧠 Step 8/9 is critical and directly connects to Module 4’s leakage warning: you compute scaling statistics (like mean/std) from the training data only, then apply that exact same transformation to validation/test data — never recompute statistics separately on them.

6. Mathematical Intuition

Read the mathematics as a story

raw records → clean/encode/scale/create features → model-ready table

First identify the input, the operation, and the output. Then read the symbols as a shorter way to describe that same journey; do not begin by memorizing the formula.

Worked standardization example:

Feature values: [10, 20, 30, 40, 50]
mean = 30
std_deviation ≈ 14.14

standardized(10) = (10 - 30) / 14.14 ≈ -1.41
standardized(30) = (30 - 30) / 14.14 = 0.00
standardized(50) = (50 - 30) / 14.14 ≈ 1.41

After standardization, the feature is centered at 0, with values roughly in the range of about -2 to +2 for typical (non-extreme) data — regardless of what the original units or scale were. This is exactly why it puts very differently-scaled features on comparable footing.

7. Small Worked Example

Walk through the example

Identify what each input number represents.
Follow one operation at a time and keep the units or class meanings attached.
Translate the result back into an ordinary sentence about the original problem.

The goal is not merely to obtain the answer; it is to expose the model’s decision process.

Two features before scaling:

Income ($)	Years of experience
45,000	3
120,000	15

A distance-based model (like KNN, Module 10) computing “how different are these two people?” using raw values would find the income difference (75,000) completely dwarfs the experience difference (12) — even though 12 years of experience is actually a huge difference in context.

After standardizing both features, they’re expressed on a comparable scale, and the model can weigh them by genuine predictive importance instead of by raw numeric magnitude.

8. Python Example

What the code will demonstrate

The following Feature Engineering and Preprocessing code turns the worked example into an experiment you can repeat. First predict the result; then prepare the small dataset, apply the technique, inspect the important intermediate values, and compare the actual output with your prediction.

Python and library symbols used below

NumPy (np) stores and calculates with numeric arrays.
pandas (pd) represents table-shaped data when it is used.
scikit-learn provides tested implementations with a consistent .fit(...) and .predict(...) workflow.

# Build a small, inspectable example of Feature Engineering and Preprocessing.
# Follow the data, learned values, predictions, and evaluation in order.
import pandas as pd
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer

data = pd.DataFrame({
    "income": [45000, 120000, 60000, 200000],
    "years_experience": [3, 15, 5, 20],
    "country": ["IN", "US", "IN", "UK"],
})

# Define which columns need which preprocessing
preprocessor = ColumnTransformer(transformers=[
    ("scale", StandardScaler(), ["income", "years_experience"]),
    ("encode", OneHotEncoder(), ["country"]),
])

# Fit on this data and transform it (in real use: fit on TRAINING data only)
transformed = preprocessor.fit_transform(data)
print(transformed)

feature_names = preprocessor.get_feature_names_out()
print(feature_names)

Expected Output (approximate):

[[-1.02  -0.88   0.     1.     0.  ]
 [ 1.32   0.82   0.     0.     1.  ]
 [-0.66  -0.63   0.     1.     0.  ]
 [ 0.37   1.51   1.     0.     0.  ]]
['scale__income' 'scale__years_experience' 'encode__country_IN'
 'encode__country_UK' 'encode__country_US']

How It Works

StandardScaler() applies exactly the standardization formula from Section 6 to income and years_experience.
OneHotEncoder() turns the single country column into three binary columns — no false ordering implied between "IN", "UK", "US".
ColumnTransformer lets you apply different preprocessing to different columns in a single, organized step — exactly matching how real datasets mix numerical and categorical features.

In real workflows, you’d call .fit(X_train) once, then .transform(X_train), .transform(X_validation), and .transform(X_test) separately — never re-fitting on validation/test data (recall Module 4’s leakage warning).

9. Real-World Example

A real estate pricing model uses raw listing data:

Scaling: square footage (100s-10,000s) and price history (100,000s) need standardization so neither dominates the model purely due to scale.
Encoding: property_type (“apartment”, “house”, “condo”) needs one-hot encoding — there’s no inherent order between property types.
Feature engineering: deriving price_per_square_foot from raw price and square_footage often turns out to be a far more directly useful feature than either raw number alone — a classic example of engineered features outperforming raw ones.

10. How This Is Used in AI

From mechanism to product

Classical ML pipelines rely heavily on preprocessing. LLM applications also preprocess text, documents, metadata, and retrieved context, although the transformations differ from tabular ML.

How this connects to LLMs

request → data or context preparation → model computation → evaluated output

An LLM may use this idea during training, or an AI application may use a separate ML component around the LLM. Those are different locations in the system, and the explanation below identifies which one applies.

🤖 The evolution from hand-crafted features to learned representations

Classic ML required humans to manually decide which features mattered and how to encode them — as seen throughout this module. Deep learning (and especially modern AI) shifted much of this work into the model itself:

Classic ML:            human designs features  →  model learns from them
Representation learning: model learns its own features directly from raw data

An embedding (covered fully in Module 18) is a learned feature representation — instead of a human hand-crafting “word count,” “contains urgent keyword,” etc., a neural network learns, from vast amounts of data, its own dense numerical representation of meaning. This didn’t eliminate the need for good features — it moved the responsibility for creating them from a human data scientist to the learning process itself.

Classic ML approach	Modern AI approach
Human manually encodes “is this word a spam-related word”	Model learns its own representation of “spam-ness” from data
One-hot encode “product category”	An embedding captures product similarity along many learned dimensions at once
Hand-engineer “sentiment score” from keyword lists	A language model learns sentiment as an emergent property of its trained representations

🤖 Even so, preprocessing is far from obsolete in AI systems: structured/tabular data pipelines feeding into rerankers, classifiers, or recommendation systems still rely heavily on exactly the scaling and encoding techniques in this module — embeddings replace text feature-engineering, not tabular preprocessing generally.

11. How This Is Used in Agentic AI

Trace one agent step

goal + state → model proposes → runtime validates → tool or response → evaluation

The model produces a prediction or proposal. The agent runtime is ordinary software that manages tools, permissions, state, retries, and execution; it may use this ML concept directly, indirectly through an LLM, or not at all.

🤖 Agent systems that incorporate structured signals (e.g., “how long has this tool call been running,” “how many retries so far,” “user’s account tier”) alongside LLM reasoning often still preprocess those structured features conventionally — scaling numerical signals, one-hot encoding categorical ones — before feeding them into any small supporting ML model (like a routing classifier or an anomaly detector monitoring agent behavior).

The LLM’s own text understanding uses learned representations (embeddings), but any classic ML component bolted onto an agent pipeline still needs classic preprocessing.

12. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: One-hot encoding a category with hundreds or thousands of unique values

Why it is incorrect: (e.g., “user ID” or “product SKU”). This explodes the number of columns, often to a degree that’s computationally wasteful and can actually hurt model performance. Very high-cardinality categorical features usually need different handling (e.g., target encoding, embeddings, or grouping rare categories).

⚠️ Mistake

Incorrect idea: Label encoding a non-ordinal category

Why it is incorrect: Encoding "red"=0, "blue"=1, "green"=2 implies to many models that "green" is somehow “more” than "red" — a false relationship for unordered categories.

⚠️ Mistake

Incorrect idea: Fitting a scaler on the full dataset before splitting

Why it is incorrect: Exactly the leakage issue flagged in Module 4 — always fit preprocessing steps on the training data only.

13. Important Distinctions

Normalization	Standardization
Rescales to a fixed range, usually [0, 1]	Rescales to mean 0, standard deviation 1
Sensitive to outliers (min/max can be distorted)	Less sensitive to outliers
Good when you need bounded values	Good default for most models assuming roughly normal data

One-Hot Encoding	Label Encoding
Creates one binary column per category	Creates a single integer column
No implied order between categories	Implies an order (0 < 1 < 2…)
Best for unordered (nominal) categories	Best for genuinely ordered (ordinal) categories

14. When Should You Use This?

Scale numerical features whenever using distance-based models (KNN, SVM) or gradient-based models (linear/logistic regression, neural networks) — these are all sensitive to feature scale.
One-hot encode unordered categorical features with a reasonably small number of categories.
Label encode genuinely ordinal categories, or as an input to tree-based models (Module 9), which are largely insensitive to encoding choice and monotonic scale differences.
Engineer new features whenever domain knowledge suggests a derived value (a ratio, a difference, a rate) would be more directly predictive than the raw inputs alone.

15. When Should You NOT Use This?

Tree-based models (decision trees, random forests, gradient boosting — Module 9) generally don’t need feature scaling at all — they split on threshold values, not distances, so scale doesn’t affect their behavior the way it does KNN or linear models.
Don’t manually engineer elaborate hand-crafted text features (keyword counts, TF-IDF scores) when a modern embedding model would likely capture richer semantic signal with far less manual effort — this is precisely the shift described in Section 10.
Don’t one-hot encode extremely high-cardinality categorical features without considering alternatives first (grouping rare categories, target encoding, or embeddings).

16. Production Considerations

Save your fitted preprocessing objects (the scaler, the encoder) alongside the trained model — at inference time, new data must go through the exact same fitted transformation, not a freshly re-fit one.
New categories at inference time — a one-hot encoder trained on ["IN", "US", "UK"] will fail or behave unexpectedly if a brand-new country code appears in production; plan for an explicit “unknown category” handling strategy.
Feature drift — the statistics used for scaling (mean, std, min, max) can become stale if the underlying data distribution shifts over time (Module 21) — periodically re-evaluate whether preprocessing needs to be refit on more recent data.

17. AI Engineer Takeaway

🎯 AI Engineer Takeaway: Preprocessing exists to give models a fair, consistent, well-scaled view of raw data — and getting it wrong (leakage, inappropriate encoding, scale mismatches) can silently degrade a model just as much as bad data itself.

The larger arc worth internalizing: much of the manual feature-engineering effort this module teaches has been partially automated away by representation learning and embeddings in modern deep learning and LLMs — but the underlying judgment (what information actually matters, and in what form) is still exactly the skill an AI engineer needs, just applied at a different layer of the system.

18. Interview Questions

Basic Questions

Q: What’s the difference between normalization and standardization?

A: Normalization rescales values into a fixed range, typically [0, 1], using the feature’s min and max. Standardization rescales values to have a mean of 0 and a standard deviation of 1. Standardization is generally more robust to outliers, since a single extreme value distorts normalization’s min/max more severely than it distorts a mean/standard-deviation calculation.

Q: Why can’t you just feed raw categorical text data (like “USA”, “India”) directly into most ML models?

A: Most ML models perform mathematical operations (distances, weighted sums, gradients) that require numerical input — they have no native way to interpret a text category. Encoding (one-hot or label encoding) translates categories into a numerical form the model can actually compute with.

Intermediate Questions

Q: Why is one-hot encoding generally preferred over label encoding for unordered categories?

A: Label encoding assigns arbitrary integers to categories (e.g., red=0, blue=1, green=2), which many models will interpret as implying a meaningful order or magnitude relationship between categories that doesn’t actually exist. One-hot encoding avoids this by representing each category as an independent binary column, with no implied ordering.

Q: Why do tree-based models generally not require feature scaling, while KNN and linear regression do?

A: Tree-based models make decisions by splitting on threshold values for each feature independently (e.g., “is income > 50,000?”) — this decision is unaffected by the overall scale of the feature. KNN relies on computing distances between data points across all features simultaneously, where features on a larger scale will dominate the distance calculation unless scaled. Linear regression’s gradient-based training also converges more reliably and predictably when features are on comparable scales.

Scenario-Based Questions

Q: You’re building a model to predict customer churn, and one feature is customer_id (a unique string per customer). A teammate suggests one-hot encoding it. What’s wrong with this idea, and what would you do instead?

A: Thought process: customer_id is unique per row — this is an extreme case of the high-cardinality problem flagged in Section 12.

Investigation: One-hot encoding a unique identifier would create one column per customer — as many columns as there are rows in the dataset. This provides essentially zero generalizable signal (the model would just be memorizing which specific customer IDs happened to churn in training, with no way to generalize to new customers it’s never seen), while making the dataset enormous and computationally wasteful.

Correct answer: Drop customer_id as a feature entirely — it carries no genuinely predictive signal about why a customer would churn. Instead, focus feature engineering on customer behavior (tenure, support ticket count, usage frequency, plan type) — attributes that actually generalize to describe churn risk, rather than an arbitrary identifier.

Production consideration: This exact mistake (accidentally including an identifier-like feature) is a subtle relative of data leakage (Module 3) — it’s worth explicitly auditing every feature for “does this carry genuine, generalizable signal, or is it effectively just labeling the row?” before finalizing a feature set.

Next: Module 06 — Bias, Variance and Generalization — underfitting, overfitting, and why memorization is not intelligence.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026