Beyond Fully-Connected Networks
MLPs treat every input feature equally — a pixel in the top-left corner of an image has no special relationship to its neighbours, and word position in a sentence carries no meaning. Specialised architectures encode these structural priors directly into the model.
Convolutional Neural Networks (CNNs)
CNNs exploit two properties of images: local correlations (nearby pixels are related) and translation invariance (a cat is a cat regardless of where in the image it appears). They achieve this via learnable convolutional filters.
Convolution Operation$$(\mathbf{I} * \mathbf{K})_{ij} = \sum_{m}\sum_{n} \mathbf{I}_{i+m,\, j+n} \cdot \mathbf{K}_{m,n}$$$\mathbf{I}$: input feature map. $\mathbf{K}$: learnable kernel/filter (e.g., 3×3). Each filter slides across the input, computing dot products at every position.
Conv LayerApplies $F$ learnable filters. Each filter detects one feature (edge, curve, texture). Output: $F$ feature maps.
PoolingMax or average pooling: spatially downsamples feature maps. Reduces computation; adds translation invariance.
Flatten + DenseAfter conv/pool layers: flatten to 1D, pass through Dense layers for final classification/regression.
Python
import tensorflow as tf
from tensorflow import keras
import numpy as np
# ── CNN for MNIST digit classification
(X_tr, y_tr), (X_te, y_te) = keras.datasets.mnist.load_data()
X_tr = X_tr[..., np.newaxis] / 255.0 # (60000,28,28,1) float32
X_te = X_te[..., np.newaxis] / 255.0
cnn = keras.Sequential([
keras.layers.Input(shape=(28, 28, 1)),
# Block 1: extract low-level features (edges)
keras.layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"),
keras.layers.Conv2D(32, kernel_size=3, padding="same", activation="relu"),
keras.layers.MaxPooling2D(pool_size=2), # (14,14,32)
keras.layers.BatchNormalization(),
keras.layers.Dropout(0.25),
# Block 2: extract higher-level features (shapes)
keras.layers.Conv2D(64, kernel_size=3, padding="same", activation="relu"),
keras.layers.Conv2D(64, kernel_size=3, padding="same", activation="relu"),
keras.layers.MaxPooling2D(pool_size=2), # (7,7,64)
keras.layers.BatchNormalization(),
keras.layers.Dropout(0.25),
# Classifier head
keras.layers.Flatten(),
keras.layers.Dense(256, activation="relu"),
keras.layers.Dropout(0.5),
keras.layers.Dense(10, activation="softmax"), # 10 digit classes
])
cnn.compile(
optimizer=keras.optimizers.Adam(1e-3),
loss="sparse_categorical_crossentropy",
metrics=["accuracy"]
)
cnn.summary()
cnn.fit(X_tr, y_tr, validation_split=0.1, epochs=10, batch_size=128,
callbacks=[keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True)])
loss, acc = cnn.evaluate(X_te, y_te, verbose=0)
print(f"Test accuracy: {acc:.4f}") # typically > 99%Data Augmentation for Images
Python
# Keras built-in augmentation layer (runs on GPU, no extra code at inference)
augment = keras.Sequential([
keras.layers.RandomFlip("horizontal"),
keras.layers.RandomRotation(0.1),
keras.layers.RandomZoom(0.1),
keras.layers.RandomTranslation(0.1, 0.1),
], name="augmentation")
# Add at the start of the model
cnn_augmented = keras.Sequential([augment] + cnn.layers[1:])Recurrent Neural Networks (RNNs)
MLPs and CNNs have no memory — each input is processed independently. RNNs process sequences by maintaining a hidden state that propagates information across time steps.
RNN Hidden State Update$$\mathbf{h}_t = \tanh(\mathbf{W}_h \mathbf{h}_{t-1} + \mathbf{W}_x \mathbf{x}_t + \mathbf{b})$$$\mathbf{h}_t$: hidden state at step $t$ (the "memory"). $\mathbf{x}_t$: current input. Problem: gradients vanish through long sequences → simple RNNs can't learn long-range dependencies.
LSTM — Long Short-Term Memory
LSTMs solve the vanishing gradient problem with a cell state $c_t$ — a "conveyor belt" of information that flows through the sequence with only small, controlled modifications at each step, controlled by three gates:
Forget gate $f_t$Decides what to erase from cell state. $f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)$
Input gate $i_t$Decides what new information to write. $i_t = \sigma(W_i[h_{t-1}, x_t] + b_i)$
Output gate $o_t$Decides what to expose as hidden state. $o_t = \sigma(W_o[h_{t-1}, x_t] + b_o)$
Python
import numpy as np
import tensorflow as tf
from tensorflow import keras
# ── Time-series: predict next value from 30-step windows
np.random.seed(42)
t = np.linspace(0, 200, 2000)
signal = np.sin(t) + 0.3 * np.sin(3*t) + 0.1 * np.random.randn(2000)
# Create sliding windows: (X[t:t+30], y[t+30])
SEQ_LEN = 30
X_seq = np.array([signal[i:i+SEQ_LEN] for i in range(len(signal)-SEQ_LEN)])
y_seq = signal[SEQ_LEN:]
X_seq = X_seq[..., np.newaxis] # (N, 30, 1)
split = int(0.8 * len(X_seq))
X_tr, X_te = X_seq[:split], X_seq[split:]
y_tr, y_te = y_seq[:split], y_seq[split:]
# ── LSTM model
lstm_model = keras.Sequential([
keras.layers.Input(shape=(SEQ_LEN, 1)),
keras.layers.LSTM(64, return_sequences=True), # return_sequences=True: pass full output to next LSTM
keras.layers.Dropout(0.2),
keras.layers.LSTM(32), # last LSTM: return final hidden state only
keras.layers.Dense(16, activation="relu"),
keras.layers.Dense(1), # predict next value
])
lstm_model.compile(optimizer="adam", loss="mse", metrics=["mae"])
lstm_model.fit(X_tr, y_tr, validation_split=0.1, epochs=30,
batch_size=64, verbose=0,
callbacks=[keras.callbacks.EarlyStopping(patience=5, restore_best_weights=True)])
mse = lstm_model.evaluate(X_te, y_te, verbose=0)[0]
print(f"LSTM Test MSE: {mse:.6f}")GRU — Gated Recurrent Unit
GRU is a simplified LSTM with just two gates (update + reset). Comparable performance to LSTM on most tasks, faster to train (fewer parameters). Good default when LSTM feels like overkill.
Python
gru_model = keras.Sequential([
keras.layers.Input(shape=(SEQ_LEN, 1)),
keras.layers.GRU(64, return_sequences=True),
keras.layers.GRU(32),
keras.layers.Dense(1),
])
gru_model.compile(optimizer="adam", loss="mse")
gru_model.fit(X_tr, y_tr, epochs=30, batch_size=64, verbose=0,
callbacks=[keras.callbacks.EarlyStopping(patience=5)])Transfer Learning (Practical Tip)
Training CNNs from scratch requires millions of images. Transfer learning uses a model pre-trained on ImageNet (e.g., MobileNetV2, EfficientNet) as a feature extractor — freeze its weights, add your own classification head, and fine-tune on your small dataset.
CopyPython
base = keras.applications.MobileNetV2(weights="imagenet", include_top=False,
input_shape=(128, 128, 3))
base.trainable = False # freeze pretrained weights
model_tl = keras.Sequential([
base,
keras.layers.GlobalAveragePooling2D(),
keras.layers.Dense(128, activation="relu"),
keras.layers.Dropout(0.3),
keras.layers.Dense(num_classes, activation="softmax"),
])Summary
- CNNs: convolutional filters learn local features. Pooling adds translation invariance. Default for images.
- RNNs: hidden state carries memory across time steps. Vanishing gradient limits range.
- LSTMs: cell state + 3 gates solve vanishing gradients. Default for sequences (text, time-series).
- GRUs: lighter LSTM variant. Good default when training speed matters.
- Transfer learning: fine-tune ImageNet models for new vision tasks. Works well with as little as 1000 images.