Preprocessing & Scaling

Why Scale Features?

Many ML algorithms compute distances (k-NN, SVM) or sum weighted features (linear regression, neural networks). If one feature spans [0, 1000] and another [0, 1], the first will dominate all distance calculations regardless of its real importance. Scaling puts features on equal footing.

🌿

Tree-based models don't need scaling. Decision trees, Random Forests, and Gradient Boosting split on individual feature thresholds — they're scale-invariant. Don't waste time scaling features for XGBoost.

Standard Scaling (Z-score Normalisation)

Transform each feature to have mean 0 and standard deviation 1:

Standard Scaling$$x' = rac{x - \mu}{\sigma} \qquad \mu = ext{feature mean},\;\sigma = ext{feature std}$$

Python

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

np.random.seed(42)
X = np.random.randn(200, 4) * [100, 1, 0.01, 500]   # wildly different scales

X_train, X_test = train_test_split(X, test_size=0.2)

# CRITICAL: fit on train only, transform both
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # fit + transform
X_test_scaled  = scaler.transform(X_test)        # transform only (no re-fit!)

print("Before:", X_train[:3, 0].round(1))        # large numbers
print("After: ", X_train_scaled[:3, 0].round(3)) # near ±1-3

# Manual implementation
mu    = X_train.mean(axis=0)
sigma = X_train.std(axis=0)
X_manual_scaled = (X_train - mu) / sigma

⚠️

The #1 data leakage mistake: Never fit the scaler on the entire dataset. Fit on training data only, then transform test data with those same statistics. Fitting on test data "leaks" test information into training.

Min-Max Normalisation

Scale features to a fixed range, typically [0, 1]:

Min-Max Scaling$$x' = rac{x - x_{\min}}{x_{\max} - x_{\min}}$$

Python

from sklearn.preprocessing import MinMaxScaler

scaler_mm = MinMaxScaler(feature_range=(0, 1))
X_train_mm = scaler_mm.fit_transform(X_train)   # all values in [0,1]
X_test_mm  = scaler_mm.transform(X_test)

# Use case: neural networks with sigmoid/tanh outputs, image pixel values
# Watch out: sensitive to outliers — a single large outlier compresses all others

Robust Scaling

Uses the median and IQR instead of mean/std — resistant to outliers:

Robust Scaling $$x' = rac{x - ext{median}}{Q3 - Q1}$$

Python

from sklearn.preprocessing import RobustScaler

# Introduce outliers
X_outliers = X_train.copy()
X_outliers[0, 0] = 1e6     # extreme outlier in feature 0

rs = RobustScaler()
X_robust = rs.fit_transform(X_outliers)
# The outlier won't distort scaling for the other 199 samples

Choosing the Right Scaler

Use StandardScaler when…

Feature is roughly Gaussian.
Algorithm assumes zero-mean features (PCA, logistic regression, neural nets).
Most common default choice.

Use MinMaxScaler when…

You need values in [0,1] (e.g., image pixels, NN inputs with sigmoid).
Distribution is uniform or bounded.
Beware: sensitive to outliers.

Use RobustScaler when…

Data has significant outliers you can't remove.
Distribution is heavily skewed.

No scaling needed for…

Decision trees, Random Forest, XGBoost, LightGBM.
Features that are already on comparable scales.

Pipelines: The Right Way

Wrap preprocessing + model in a scikit-learn Pipeline to prevent data leakage and make deployment trivial:

Python

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

# Pipeline: scaling is fitted only on train folds inside CV
pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model",  LogisticRegression(max_iter=1000)),
])

cv_scores = cross_val_score(pipe, X, y, cv=5, scoring="roc_auc")
print(f"AUC: {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")

Summary

Always fit scalers on training data only; transform test with the same fitted scaler.
StandardScaler: zero mean, unit variance. Best default for most algorithms.
MinMaxScaler: [0,1] range. Sensitive to outliers.
RobustScaler: median/IQR based. Best when outliers are present.
Tree-based models are scale-invariant — don't waste time scaling for XGBoost.
Use Pipeline to prevent data leakage in cross-validation.