Dimensionality Reduction: PCA, t-SNE & LDA

Why Reduce Dimensions?

High-dimensional data creates three problems: (1) the curse of dimensionality — distances lose meaning; (2) computational cost grows with $d$; (3) humans can't visualise beyond 3D. Dimensionality reduction compresses data while preserving what matters.

PCA — Principal Component Analysis

PCA finds the orthogonal directions (principal components) of maximum variance in the data and projects onto the top-$k$ of them. It's a linear transformation that preserves global structure.

PCA Recipe$$\text{1. Centre: } \mathbf{X}_c = \mathbf{X}-\bar{\mathbf{X}} \quad \text{2. Covariance: } \boldsymbol{\Sigma}=\frac{\mathbf{X}_c^\top\mathbf{X}_c}{m-1} \quad \text{3. Eigenvectors: } \boldsymbol{\Sigma}\mathbf{v}=\lambda\mathbf{v}$$The eigenvectors of $\boldsymbol{\Sigma}$ (principal components) are the directions of maximum variance. The eigenvalues give how much variance each direction explains.

Python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_digits    # 64-dim handwritten digits

digits = load_digits()
X, y   = digits.data, digits.target       # (1797, 64)

# PCA pipeline
pca_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("pca",    PCA()),   # keep all components first
])
X_pca = pca_pipe.fit_transform(X)

# How many components explain 95% of variance?
var_ratio  = pca_pipe.named_steps["pca"].explained_variance_ratio_
cumulative = np.cumsum(var_ratio)
n_95       = np.searchsorted(cumulative, 0.95) + 1
print(f"Components for 95% variance: {n_95} / 64")

# Scree plot
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.bar(range(1, 21), var_ratio[:20]*100, color="#4a8fa8")
ax1.set_title("Scree Plot — Top 20 Components")
ax1.set_xlabel("Principal Component"); ax1.set_ylabel("Explained Variance (%)")

ax2.plot(range(1, len(cumulative)+1), cumulative*100, lw=2, color="#b85c2a")
ax2.axhline(95, color="green", ls="--", label="95% threshold")
ax2.axvline(n_95, color="green", ls=":")
ax2.set_title("Cumulative Explained Variance")
ax2.set_xlabel("Number of Components"); ax2.legend()
plt.tight_layout(); plt.show()

# 2D visualisation
pca2d = Pipeline([("scaler", StandardScaler()), ("pca", PCA(n_components=2))])
X_2d  = pca2d.fit_transform(X)
plt.figure(figsize=(8, 6))
scatter = plt.scatter(X_2d[:,0], X_2d[:,1], c=y, cmap="tab10", s=10, alpha=0.8)
plt.colorbar(scatter, label="Digit class")
plt.title("Digits dataset — PCA 2D projection")
plt.xlabel("PC1"); plt.ylabel("PC2"); plt.tight_layout(); plt.show()

t-SNE — Non-Linear Visualisation

t-SNE (t-distributed Stochastic Neighbour Embedding) is a non-linear method designed for visualisation (typically 2D or 3D). It preserves local neighbourhoods — similar points cluster together. It does not preserve global distances or distances between clusters.

Python

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt
import time

# t-SNE is slow — run PCA first to reduce to 50D (standard practice)
X_pca50 = Pipeline([("sc", StandardScaler()),
                    ("pca", PCA(n_components=50))]).fit_transform(X)

t0 = time.time()
tsne = TSNE(
    n_components=2,
    perplexity=30,       # effective neighbourhood size. Try 5-50. Lower=local, higher=global.
    learning_rate="auto",
    n_iter=1000,
    random_state=42,
    init="pca",          # initialise from PCA (faster + more reproducible than random)
)
X_tsne = tsne.fit_transform(X_pca50)
print(f"t-SNE took {time.time()-t0:.1f}s")

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
for ax, X_2d, title in zip(axes, [X_2d, X_tsne], ["PCA", "t-SNE"]):
    sc = ax.scatter(X_2d[:,0], X_2d[:,1], c=y, cmap="tab10", s=10, alpha=0.8)
    ax.set_title(f"Digits — {title}")
    plt.colorbar(sc, ax=ax, label="Digit")
plt.tight_layout(); plt.show()

⚠️

t-SNE gotchas: (1) Results change every run unless you fix random_state. (2) Cluster sizes and distances between clusters in t-SNE plots are not meaningful. (3) It's not suitable for dimensionality reduction before a downstream ML model — use PCA for that. (4) Run PCA to ~50D first for speed.

LDA — Linear Discriminant Analysis

LDA is a supervised dimensionality reduction method that finds the projection maximising class separability — the ratio of between-class variance to within-class variance.

LDA Objective$$\max_{\mathbf{w}}\;J(\mathbf{w}) = \frac{\mathbf{w}^\top S_B \mathbf{w}}{\mathbf{w}^\top S_W \mathbf{w}}$$$S_B$: between-class scatter matrix. $S_W$: within-class scatter matrix. Can project to at most $K-1$ dimensions ($K$ = number of classes).

Python

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
import matplotlib.pyplot as plt

# LDA: supervised — uses class labels to find best separation
lda  = LDA(n_components=2)  # max K-1 = 9 components, we use 2 for viz
X_lda = lda.fit_transform(X, y)

# Compare all three methods
fig, axes = plt.subplots(1, 3, figsize=(18, 5))
for ax, X_2d, title in zip(axes, [X_2d, X_tsne, X_lda], ["PCA", "t-SNE", "LDA"]):
    sc = ax.scatter(X_2d[:,0], X_2d[:,1], c=y, cmap="tab10", s=10, alpha=0.8)
    ax.set_title(f"Digits — {title}"); plt.colorbar(sc, ax=ax)
plt.suptitle("PCA vs t-SNE vs LDA — Digits Dataset", fontsize=13)
plt.tight_layout(); plt.show()

# LDA as a preprocessing step before classification
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score

pipe_lda = Pipeline([("scaler", StandardScaler()), ("lda", LDA(n_components=9)), ("clf", LogisticRegression(max_iter=500))])
pipe_raw = Pipeline([("scaler", StandardScaler()),                                  ("clf", LogisticRegression(max_iter=500))])
for name, pipe in [("Raw features (64D)", pipe_raw), ("LDA features (9D)", pipe_lda)]:
    acc = cross_val_score(pipe, X, y, cv=5).mean()
    print(f"{name}: {acc:.4f}")

Choosing a Reduction Method

PCAUnsupervised, linear, fast. Use for preprocessing before ML, noise reduction, or visualisation when global structure matters. Start here.

t-SNEUnsupervised, non-linear, visualisation only. Best for exploring cluster structure in 2D/3D. Run PCA to 50D first.

LDASupervised, linear. Maximises class separability. Use when you have labels and want the best linear compression for classification. Max $K-1$ dimensions.

Summary

PCA: orthogonal directions of max variance. Use for preprocessing + visualisation. Always scale first.
t-SNE: preserves local neighbourhoods. Visualisation only. Cluster distances are not meaningful.
LDA: maximises class separation using labels. Supervised. Use as preprocessing before classifiers.
Standard pipeline: StandardScaler → PCA (to 50D) → t-SNE (for plots) or LDA (for classification).