Classification: Logistic Regression & KNN

From Regression to Classification

Classification predicts a discrete label rather than a continuous value. The same data, preprocessed the same way — but the model output, loss function, and evaluation metrics all change.

Logistic Regression

Despite the name, logistic regression is a classification algorithm. It models the probability of belonging to the positive class using the sigmoid function:

Logistic Regression$$\hat{p} = \sigma(\mathbf{w}^\top\mathbf{x}+b) = \frac{1}{1+e^{-(\mathbf{w}^\top\mathbf{x}+b)}}$$Output $\hat{p} \in (0,1)$. Predict class 1 if $\hat{p} \geq 0.5$, else class 0. Decision boundary: $\mathbf{w}^\top\mathbf{x}+b=0$ (a hyperplane).

Binary Cross-Entropy Loss$$J(\mathbf{w}) = -\frac{1}{m}\sum_{i=1}^m \left[y^{(i)}\log\hat{p}^{(i)} + (1-y^{(i)})\log(1-\hat{p}^{(i)})\right]$$Convex loss — gradient descent always finds the global minimum (if data is not perfectly separable).

Python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.pipeline import Pipeline

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

pipe_lr = Pipeline([
    ("scaler", StandardScaler()),
    ("model",  LogisticRegression(C=1.0, max_iter=1000, random_state=42)),
    # C = 1/lambda: SMALLER C = MORE regularisation
])
pipe_lr.fit(X_train, y_train)

print(classification_report(y_test, pipe_lr.predict(X_test),
      target_names=["malignant","benign"]))
print(f"ROC-AUC: {roc_auc_score(y_test, pipe_lr.predict_proba(X_test)[:,1]):.4f}")

# Multiclass: One-vs-Rest (OvR) is the default
# For softmax multinomial, use: LogisticRegression(multi_class='multinomial', solver='lbfgs')

K-Nearest Neighbours (KNN)

The simplest classification algorithm: classify a new point by looking at the $k$ closest training examples (its "neighbourhood") and taking a majority vote. No explicit training — all computation is at inference time.

KNN Classification$$\hat{y}(\mathbf{x}) = \arg\max_{c}\sum_{i \in \mathcal{N}_k(\mathbf{x})} \mathbf{1}[y^{(i)}=c]$$$\mathcal{N}_k(\mathbf{x})$: the $k$ training points nearest to $\mathbf{x}$ under chosen distance metric.

Python

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

X, y = make_moons(n_samples=300, noise=0.25, random_state=42)

# Find best k via cross-validation
k_range = range(1, 31)
cv_scores = []
for k in k_range:
    pipe = Pipeline([("scaler", StandardScaler()),
                     ("knn",   KNeighborsClassifier(n_neighbors=k))])
    cv_scores.append(cross_val_score(pipe, X, y, cv=5).mean())

best_k = k_range[np.argmax(cv_scores)]
print(f"Best k = {best_k}  (CV accuracy = {max(cv_scores):.4f})")

# Visualise decision boundary for k=1 vs best_k
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
scaler = StandardScaler().fit(X)
X_s = scaler.transform(X)
x_min, x_max = X_s[:,0].min()-0.5, X_s[:,0].max()+0.5
y_min, y_max = X_s[:,1].min()-0.5, X_s[:,1].max()+0.5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200), np.linspace(y_min, y_max, 200))

for ax, k, title in zip(axes, [1, best_k], [f"k=1 (overfit)", f"k={best_k} (best)"]):
    knn = KNeighborsClassifier(n_neighbors=k).fit(X_s, y)
    Z = knn.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
    ax.contourf(xx, yy, Z, alpha=0.3, cmap=ListedColormap(["#e07070","#70a870"]))
    ax.scatter(X_s[:,0], X_s[:,1], c=y, cmap=ListedColormap(["#c04040","#408040"]), s=20, edgecolors="white", lw=0.3)
    ax.set_title(title); ax.set_xlabel("Feature 1"); ax.set_ylabel("Feature 2")
plt.tight_layout(); plt.show()

Distance Metrics

Euclidean (L2)$\sqrt{\sum(a_i-b_i)^2}$ — straight line. Default. Sensitive to feature scale → always StandardScale first.

Manhattan (L1)$\sum|a_i-b_i|$ — grid distance. More robust to outliers in individual features.

Cosine$1 - \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}$ — angle between vectors. Ignores magnitude. Best for text/embeddings.

Minkowski$(\sum|a_i-b_i|^p)^{1/p}$ — generalises L1 ($p=1$) and L2 ($p=2$).

Logistic Regression vs. KNN

Logistic Regression

Linear decision boundary only.
Fast inference: $O(n)$ per prediction.
Interpretable coefficients.
Works well when classes are linearly separable. Needs scaling.

KNN

Non-linear, flexible boundary.
Slow inference: $O(m \cdot n)$ — searches all training points.
No "training" phase; all data stored.
Degrades in high dimensions (curse of dimensionality). Needs scaling.

Summary

Logistic Regression: $\hat{p}=\sigma(\mathbf{w}^\top\mathbf{x})$. Minimises cross-entropy. Linear boundary.
KNN: majority vote among $k$ nearest neighbours. No training; all computation at query time.
Both require feature scaling. Use Pipeline to keep it clean.
Logistic regression is preferred when interpretability or speed matters. KNN for small datasets with non-linear boundaries.