Backpropagation
Backpropagation is the algorithm that computes gradients of the loss with respect to every parameter in the network. It's not magic — it's the chain rule of calculus applied systematically from the output layer back to the input layer.
Chain Rule Through Layers$$\frac{\partial L}{\partial \mathbf{W}^{[l]}} = \frac{\partial L}{\partial \mathbf{A}^{[l]}} \cdot \frac{\partial \mathbf{A}^{[l]}}{\partial \mathbf{Z}^{[l]}} \cdot \frac{\partial \mathbf{Z}^{[l]}}{\partial \mathbf{W}^{[l]}}$$Each factor is a local gradient. Forward pass stores intermediate values; backward pass multiplies them from right to left. No new calculus needed — just systematic bookkeeping.
Python
import numpy as np
# Manual backprop through a 2-layer network: X -> Dense(4, ReLU) -> Dense(1, Sigmoid) -> BCE
np.random.seed(42)
n, d = 100, 5
X = np.random.randn(n, d)
y = (np.random.randn(n) > 0).astype(float)
# He initialisation
W1 = np.random.randn(d, 4) * np.sqrt(2 / d)
b1 = np.zeros(4)
W2 = np.random.randn(4, 1) * np.sqrt(2 / 4)
b2 = np.zeros(1)
def relu(z): return np.maximum(0, z)
def sigmoid(z): return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
lr = 0.05
for epoch in range(1000):
# ── Forward pass
Z1 = X @ W1 + b1 # (n,4)
A1 = relu(Z1) # (n,4)
Z2 = A1 @ W2 + b2 # (n,1)
A2 = sigmoid(Z2) # (n,1)
y_hat = A2.squeeze()
# ── Loss (binary cross-entropy)
loss = -np.mean(y * np.log(y_hat + 1e-9) + (1 - y) * np.log(1 - y_hat + 1e-9))
# ── Backward pass (chain rule)
dL_dA2 = -(y / (y_hat + 1e-9) - (1-y) / (1-y_hat+1e-9)) / n # (n,)
dL_dZ2 = (dL_dA2 * A2.squeeze() * (1 - A2.squeeze())).reshape(-1, 1) # sigmoid deriv
dL_dW2 = A1.T @ dL_dZ2 # (4,1)
dL_db2 = dL_dZ2.sum(axis=0) # (1,)
dL_dA1 = dL_dZ2 @ W2.T # (n,4)
dL_dZ1 = dL_dA1 * (Z1 > 0) # ReLU derivative
dL_dW1 = X.T @ dL_dZ1 # (d,4)
dL_db1 = dL_dZ1.sum(axis=0) # (4,)
# ── Parameter update
W2 -= lr * dL_dW2; b2 -= lr * dL_db2
W1 -= lr * dL_dW1; b1 -= lr * dL_db1
if epoch % 200 == 0:
print(f"Epoch {epoch:4d} Loss: {loss:.4f}")SGD and Mini-Batch Gradient Descent
Batch GDGradient over all $m$ samples. Exact but slow. Impractical for large datasets.
SGD (batch=1)One sample at a time. Very noisy — gradient fluctuates wildly. Can escape local minima. Too slow in practice.
Mini-Batch GD$b$ samples per step (typically 32–256). Best of both worlds. The standard in deep learning. GPU-friendly.
Mini-Batch SGD Update$$\mathbf{W} \leftarrow \mathbf{W} - \alpha \frac{1}{b}\sum_{i\in\text{batch}}\nabla_{\mathbf{W}} L^{(i)}$$
Advanced Optimisers
Python
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow import keras
# Compare optimisers on a real task
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
sc = StandardScaler(); X_tr = sc.fit_transform(X_tr); X_te = sc.transform(X_te)
def make_model():
return keras.Sequential([
keras.layers.Input(shape=(30,)),
keras.layers.Dense(64, activation="relu"),
keras.layers.Dense(32, activation="relu"),
keras.layers.Dense(1, activation="sigmoid"),
])
optimizers = {
"SGD (lr=0.01)": keras.optimizers.SGD(learning_rate=0.01),
"SGD + Momentum": keras.optimizers.SGD(learning_rate=0.01, momentum=0.9),
"RMSprop": keras.optimizers.RMSprop(learning_rate=1e-3),
"Adam": keras.optimizers.Adam(learning_rate=1e-3),
"AdamW": keras.optimizers.AdamW(learning_rate=1e-3, weight_decay=1e-4),
}
histories = {}
for name, opt in optimizers.items():
m = make_model()
m.compile(optimizer=opt, loss="binary_crossentropy", metrics=["accuracy"])
h = m.fit(X_tr, y_tr, validation_data=(X_te, y_te),
epochs=100, batch_size=32, verbose=0)
histories[name] = h.history
colors = ["#4a8fa8","#5c8a58","#b85c2a","#c4891a","#7a5a8a"]
plt.figure(figsize=(10, 5))
for (name, h), col in zip(histories.items(), colors):
plt.plot(h["val_accuracy"], label=name, color=col, lw=2)
plt.xlabel("Epoch"); plt.ylabel("Val Accuracy")
plt.title("Optimiser Comparison — Breast Cancer")
plt.legend(); plt.grid(alpha=0.3); plt.tight_layout(); plt.show()MomentumAccumulates a velocity vector in the direction of persistent gradient. Smooths noisy updates, speeds up convergence. $v \leftarrow \beta v - \alpha\nabla L$.
RMSpropPer-parameter adaptive learning rates. Divides by exponential moving average of squared gradients. Prevents oscillation in ravines.
AdamAdaptive Moment Estimation. Combines Momentum (1st moment) and RMSprop (2nd moment). Bias-corrected. Default choice for most networks.
AdamWAdam + decoupled weight decay regularisation. Better generalisation than Adam in practice. Modern default for transformers.
Learning Rate Scheduling
Python
import tensorflow as tf
# Cosine annealing with warm restarts — state of the art
lr_schedule = keras.optimizers.schedules.CosineDecayRestarts(
initial_learning_rate=1e-3,
first_decay_steps=50, # restart every 50 steps
t_mul=2.0, # double restart period each time
m_mul=0.9, # multiply peak lr by 0.9 each restart
)
# Step decay: halve LR every 30 epochs
def step_decay(epoch, lr):
return lr * 0.5 if epoch > 0 and epoch % 30 == 0 else lr
model = make_model()
model.compile(
optimizer=keras.optimizers.Adam(learning_rate=lr_schedule),
loss="binary_crossentropy", metrics=["accuracy"]
)
model.fit(X_tr, y_tr, epochs=100, batch_size=32, verbose=0,
callbacks=[keras.callbacks.LearningRateScheduler(step_decay)])Summary
- Backpropagation = chain rule applied layer-by-layer from output to input. Stores forward activations, multiplies local gradients backward.
- Mini-batch SGD (batch 32–256) is the standard. Full-batch is too slow; batch=1 is too noisy.
- Adam is the default optimiser. AdamW is preferred when regularisation is important.
- Learning rate scheduling (cosine decay, ReduceLROnPlateau) improves final performance.