The Artificial Neuron
A single artificial neuron takes a vector of inputs, computes a weighted sum, adds a bias, and passes the result through a non-linear activation function. This is a direct (loose) analogy to a biological neuron — inputs arrive via dendrites, are summed in the cell body, and fire an output signal if the sum exceeds a threshold.
Single Neuron$$z = \mathbf{w}^\top\mathbf{x} + b \qquad a = g(z)$$$z$: pre-activation (linear combination). $b$: bias. $g$: activation function. $a$: output activation.
The Perceptron
The perceptron (Rosenblatt, 1958) is the earliest trainable neural network. It uses a step activation function and updates weights to fix misclassifications:
Perceptron Update Rule$$\mathbf{w} \leftarrow \mathbf{w} + \alpha(y^{(i)} - \hat{y}^{(i)})\mathbf{x}^{(i)}$$Only updates when wrong. Converges in finite steps if data is linearly separable (Perceptron Convergence Theorem). Fails on XOR — non-linearity requires multiple layers.
Python
import numpy as np
import matplotlib.pyplot as plt
class Perceptron:
def __init__(self, lr=0.1, n_iter=100):
self.lr = lr; self.n_iter = n_iter
def fit(self, X, y):
self.w = np.zeros(X.shape[1])
self.b = 0.0
self.errors_ = []
for _ in range(self.n_iter):
errors = 0
for xi, yi in zip(X, y):
pred = self.predict_single(xi)
update = self.lr * (yi - pred)
self.w += update * xi
self.b += update
errors += int(update != 0)
self.errors_.append(errors)
return self
def predict_single(self, x):
return 1 if self.w @ x + self.b >= 0 else 0
def predict(self, X):
return np.array([self.predict_single(x) for x in X])
# Test on AND gate (linearly separable)
X_and = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y_and = np.array([0, 0, 0, 1])
p = Perceptron(lr=0.1, n_iter=20).fit(X_and, y_and)
print("AND gate predictions:", p.predict(X_and)) # [0, 0, 0, 1]
plt.plot(p.errors_, color="#4a8fa8", marker="o")
plt.xlabel("Epoch"); plt.ylabel("Misclassifications")
plt.title("Perceptron — AND gate convergence")
plt.tight_layout(); plt.show()Activation Functions
Activation functions introduce non-linearity — without them, stacking multiple linear layers is still just one linear layer. They determine the range of neuron outputs and have crucial effects on gradient flow during training.
Python
import numpy as np
import matplotlib.pyplot as plt
z = np.linspace(-5, 5, 300)
# ── Sigmoid: squashes to (0,1). Output interpretable as probability.
# Problem: saturates at extremes → vanishing gradients in deep nets.
sigmoid = 1 / (1 + np.exp(-z))
d_sigmoid = sigmoid * (1 - sigmoid) # max derivative = 0.25 (at z=0)
# ── Tanh: squashes to (-1,1). Zero-centred → faster convergence than sigmoid.
# Still saturates, still has vanishing gradient problem.
tanh = np.tanh(z)
d_tanh = 1 - tanh**2 # max derivative = 1.0 (at z=0)
# ── ReLU: max(0,z). Default for hidden layers. No saturation for z>0.
# Problem: "dying ReLU" — neurons stuck at 0 for all inputs if w→wrong direction.
relu = np.maximum(0, z)
d_relu = (z > 0).astype(float)
# ── Leaky ReLU: fixes dying ReLU by allowing small negative gradient
leaky_relu = np.where(z > 0, z, 0.01 * z)
# ── Softmax: converts vector of scores to probability distribution (sums to 1).
# Used ONLY in output layer for multiclass classification.
def softmax(logits):
e = np.exp(logits - logits.max()) # subtract max for numerical stability
return e / e.sum()
fig, axes = plt.subplots(2, 4, figsize=(16, 7))
pairs = [("Sigmoid", sigmoid, d_sigmoid, "#4a8fa8"),
("Tanh", tanh, d_tanh, "#5c8a58"),
("ReLU", relu, d_relu, "#b85c2a"),
("Leaky ReLU", leaky_relu, np.where(z>0,1,0.01), "#c4891a")]
for i, (name, fn, dfn, col) in enumerate(pairs):
axes[0,i].plot(z, fn, color=col, lw=2.5); axes[0,i].set_title(f"{name}")
axes[1,i].plot(z, dfn, color=col, lw=2.5, ls="--"); axes[1,i].set_title(f"d/dz {name}")
for ax in [axes[0,i], axes[1,i]]:
ax.axhline(0, color="gray", lw=0.5); ax.axvline(0, color="gray", lw=0.5)
ax.grid(alpha=0.2); ax.set_xlabel("z")
plt.suptitle("Activation Functions and Their Derivatives", fontsize=13)
plt.tight_layout(); plt.show()SigmoidOutput layer for binary classification. Avoid in hidden layers of deep nets (vanishing gradient).
TanhZero-centred version of sigmoid. Better for hidden layers than sigmoid, but ReLU dominates in practice.
ReLUDefault for hidden layers. Simple, fast, no saturation for positive inputs. Watch for dying neurons.
Leaky ReLU / ELUFix dying ReLU. Slight improvement in some architectures.
SoftmaxOutput layer for multiclass classification. Converts logits to probability distribution summing to 1.
Weights and Biases
Weights $\mathbf{w}$ control the strength and direction of each input's influence. Bias $b$ shifts the activation threshold — it lets the neuron fire even when all inputs are zero.
Weight initialisation matters. All-zeros: every neuron learns the same thing (symmetry problem). All-ones: exploding gradients. Best practice: He initialisation for ReLU ($\mathcal{N}(0, \sqrt{2/n_{in}})$), Xavier/Glorot for Tanh ($\mathcal{N}(0, \sqrt{2/(n_{in}+n_{out})})$).
Summary
- A neuron computes $a = g(\mathbf{w}^\top\mathbf{x}+b)$. Weights scale inputs; bias shifts the threshold.
- Perceptron: step activation + misclassification update rule. Converges on linearly separable data.
- ReLU: default for hidden layers. Sigmoid/Softmax for output layers (binary/multiclass classification).
- Use He initialisation with ReLU; Xavier/Glorot with Tanh/Sigmoid.