Classification Metrics

Beyond Accuracy

Accuracy is often a misleading metric. On a dataset where 95% of samples are class 0, a model that always predicts class 0 achieves 95% accuracy — but is completely useless. You need metrics that reveal how the model performs on each class.

The Confusion Matrix

The confusion matrix shows the full picture of classification performance, breaking predictions into four categories:

Confusion Matrix$$\begin{pmatrix}\text{TN} & \text{FP}\\ \text{FN} & \text{TP}\end{pmatrix} \qquad \text{Accuracy} = \frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}}$$TP: correctly predicted positive. TN: correctly predicted negative. FP: negative predicted as positive (Type I error). FN: positive predicted as negative (Type II error).

Python

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import (confusion_matrix, classification_report,
                              ConfusionMatrixDisplay)

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2,
                                            random_state=42, stratify=y)

model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_tr, y_tr)
y_pred = model.predict(X_te)
y_prob = model.predict_proba(X_te)[:, 1]

# Confusion matrix
cm = confusion_matrix(y_te, y_pred)
fig, ax = plt.subplots(figsize=(5, 4))
ConfusionMatrixDisplay(cm, display_labels=["Malignant","Benign"]).plot(
    ax=ax, colorbar=False, cmap="Blues")
ax.set_title("Confusion Matrix — Breast Cancer")
plt.tight_layout(); plt.show()

# Full classification report
print(classification_report(y_te, y_pred, target_names=["Malignant","Benign"]))

Precision and Recall

Precision, Recall, F1$$\text{Precision} = \frac{\text{TP}}{\text{TP}+\text{FP}} \qquad \text{Recall} = \frac{\text{TP}}{\text{TP}+\text{FN}} \qquad F_1 = 2\cdot\frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}}$$Precision: "of all positive predictions, how many were correct?" Recall: "of all actual positives, how many did we find?" F1 is their harmonic mean.

🌿

When to prioritise which metric: Use recall when missing a positive is costly — cancer screening (missing cancer = bad). Use precision when a false alarm is costly — spam filter (marking legit email as spam = bad). Use F1 when you need a balance. Use AUC-ROC when you need a threshold-independent ranking score.

Python

from sklearn.metrics import (precision_recall_curve, PrecisionRecallDisplay,
                              average_precision_score, f1_score)

# Precision-Recall curve — best for imbalanced datasets
precision, recall, thresholds = precision_recall_curve(y_te, y_prob)
ap = average_precision_score(y_te, y_prob)

fig, ax = plt.subplots(figsize=(6, 5))
PrecisionRecallDisplay(precision=precision, recall=recall).plot(ax=ax)
ax.set_title(f"Precision-Recall Curve (AP = {ap:.4f})")
plt.tight_layout(); plt.show()

# Find threshold that maximises F1
f1_scores  = 2 * precision[:-1] * recall[:-1] / (precision[:-1] + recall[:-1] + 1e-9)
best_thresh = thresholds[np.argmax(f1_scores)]
y_pred_best = (y_prob >= best_thresh).astype(int)
print(f"Default threshold (0.5) F1: {f1_score(y_te, y_pred):.4f}")
print(f"Optimal threshold ({best_thresh:.3f}) F1: {f1_score(y_te, y_pred_best):.4f}")

ROC Curve and AUC

The ROC (Receiver Operating Characteristic) curve plots True Positive Rate (Recall) vs. False Positive Rate across all classification thresholds. AUC is the area under this curve — a single number summarising overall discriminative ability.

TPR and FPR$$\text{TPR} = \frac{\text{TP}}{\text{TP}+\text{FN}} \qquad \text{FPR} = \frac{\text{FP}}{\text{FP}+\text{TN}}$$AUC = 0.5: random classifier. AUC = 1.0: perfect classifier. AUC is threshold-independent — measures ranking quality. Prefer PR-AUC for heavily imbalanced datasets.

Python

from sklearn.metrics import roc_curve, roc_auc_score, RocCurveDisplay
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

classifiers = {
    "Random Forest":     model,
    "Logistic Reg":      Pipeline([("sc", StandardScaler()), ("lr", LogisticRegression(max_iter=1000))]),
    "SVM (RBF)":         Pipeline([("sc", StandardScaler()), ("svm", SVC(probability=True))]),
}

fig, ax = plt.subplots(figsize=(7, 6))
ax.plot([0,1],[0,1],"k--", label="Random (AUC=0.50)", lw=1)
colors = ["#4a8fa8","#5c8a58","#b85c2a"]

for (name, clf), col in zip(classifiers.items(), colors):
    clf.fit(X_tr, y_tr)
    prob = clf.predict_proba(X_te)[:,1]
    fpr, tpr, _ = roc_curve(y_te, prob)
    auc = roc_auc_score(y_te, prob)
    ax.plot(fpr, tpr, label=f"{name} (AUC={auc:.3f})", color=col, lw=2)

ax.set_xlabel("False Positive Rate"); ax.set_ylabel("True Positive Rate")
ax.set_title("ROC Curves — Breast Cancer Classification")
ax.legend(loc="lower right"); ax.grid(alpha=0.3)
plt.tight_layout(); plt.show()

Multiclass Metrics

Python

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=42)
rf = RandomForestClassifier(n_estimators=100, random_state=42).fit(X_tr, y_tr)

print(classification_report(y_te, rf.predict(X_te),
      target_names=load_iris().target_names))
# macro avg:    treat all classes equally
# weighted avg: weight each class by its support (sample count)

# Multiclass ROC — one-vs-rest
from sklearn.metrics import roc_auc_score
y_prob_multi = rf.predict_proba(X_te)
print(f"Multiclass AUC (OvR, weighted): {roc_auc_score(y_te, y_prob_multi, multi_class='ovr', average='weighted'):.4f}")

Summary

Accuracy is misleading on imbalanced datasets. Always look at the full confusion matrix.
Precision: minimise false alarms. Recall: minimise missed positives. F1: balance of both.
ROC-AUC: threshold-independent rank quality. PR-AUC: better for severe class imbalance.
For multiclass: macro average treats all classes equally; weighted average accounts for class frequency.