Exploratory Data Analysis

What is EDA?

Exploratory Data Analysis (EDA) is your first conversation with the data. You're looking for: the shape of distributions, relationships between features, outliers, data quality issues, and whether the signal you expect is actually there. Good EDA prevents bad models.

🌿

EDA First Principle: Never clean before you look. Cleaning removes evidence of what's broken. Explore first, then clean, then explore again to verify.

Quick Dataset Profile

Python

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Load the Titanic dataset (classic EDA dataset)
df = pd.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv")

# ── Basic overview
print("Shape:", df.shape)
print("
Data types:")
print(df.dtypes)
print("
Null counts:")
print(df.isnull().sum())
print("
Basic stats:")
print(df.describe())
print("
Target balance (survived):")
print(df["survived"].value_counts(normalize=True).round(3))

Visualising Distributions

Python

import matplotlib.pyplot as plt
import seaborn as sns

# Use a clean, readable style
sns.set_theme(style="whitegrid", palette="muted", font_scale=1.1)
fig, axes = plt.subplots(2, 3, figsize=(14, 8))
fig.suptitle("Titanic — Distribution Analysis", fontsize=14, fontweight="bold")

# Histogram for continuous variable
axes[0,0].hist(df["age"].dropna(), bins=30, color="#4a8fa8", edgecolor="white")
axes[0,0].set_title("Age Distribution"); axes[0,0].set_xlabel("Age")

# KDE (smooth distribution estimate) overlaid
sns.histplot(df["fare"], bins=40, kde=True, ax=axes[0,1], color="#5c8a58")
axes[0,1].set_title("Fare Distribution")

# Box plot — shows median, IQR, and outliers
sns.boxplot(x="pclass", y="fare", data=df, ax=axes[0,2], palette="Set2")
axes[0,2].set_title("Fare by Passenger Class")

# Count plot for categorical
sns.countplot(x="embarked", hue="survived", data=df, ax=axes[1,0], palette="Set1")
axes[1,0].set_title("Embarkation Port vs. Survival")

# Violin plot — richer than boxplot
sns.violinplot(x="sex", y="age", data=df, ax=axes[1,1], palette="muted")
axes[1,1].set_title("Age by Sex")

# Survival rate by class
survival_by_class = df.groupby("pclass")["survived"].mean()
axes[1,2].bar(survival_by_class.index, survival_by_class.values, color="#b85c2a")
axes[1,2].set_title("Survival Rate by Class")
axes[1,2].set_xlabel("Passenger Class"); axes[1,2].set_ylabel("Survival Rate")

plt.tight_layout()
plt.savefig("distributions.png", dpi=150, bbox_inches="tight")
plt.show()

Correlation Analysis

Python

import seaborn as sns, matplotlib.pyplot as plt

# Pearson correlation matrix for numeric columns
corr = df.select_dtypes("number").corr()

fig, ax = plt.subplots(figsize=(9, 7))
mask = np.triu(np.ones_like(corr, dtype=bool))   # upper triangle mask
sns.heatmap(
    corr, mask=mask, annot=True, fmt=".2f",
    cmap="RdYlBu_r", center=0, vmin=-1, vmax=1,
    square=True, linewidths=0.5, ax=ax
)
ax.set_title("Pearson Correlation Matrix")
plt.tight_layout(); plt.show()

# Pair plot for selected features (sample for speed)
sample = df[["age","fare","survived","pclass"]].dropna().sample(200)
sns.pairplot(sample, hue="survived", palette={0:"#e07070", 1:"#70a870"}, plot_kws={"alpha":0.5})
plt.suptitle("Pairplot — Titanic Features", y=1.02)
plt.show()

Identifying Outliers

Two principal methods:

Python

import numpy as np, pandas as pd

# ── Method 1: IQR (Interquartile Range) — robust, preferred
def iqr_outliers(series, k=1.5):
    Q1, Q3 = series.quantile([0.25, 0.75])
    IQR = Q3 - Q1
    lower, upper = Q1 - k * IQR, Q3 + k * IQR
    return series[(series < lower) | (series > upper)]

fare_outliers = iqr_outliers(df["fare"])
print(f"IQR method — Fare outliers: {len(fare_outliers)} ({len(fare_outliers)/len(df)*100:.1f}%)")

# ── Method 2: Z-score — assumes Gaussian, sensitive
from scipy import stats
z_scores = np.abs(stats.zscore(df["fare"].dropna()))
outlier_z = df["fare"].dropna()[z_scores > 3]
print(f"Z-score method — Fare outliers: {len(outlier_z)}")

# Visualise with boxplot + stripplot
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
df["fare"].plot.box(ax=ax1, vert=False); ax1.set_title("Fare — Boxplot (note outliers)")
np.log1p(df["fare"]).plot.box(ax=ax2, vert=False); ax2.set_title("log1p(Fare) — Boxplot")
plt.tight_layout(); plt.show()

Missing Value Analysis

Python

import matplotlib.pyplot as plt

# Missing value heatmap
null_pct = df.isnull().mean().sort_values(ascending=False)
null_pct = null_pct[null_pct > 0]

fig, ax = plt.subplots(figsize=(8, 4))
null_pct.plot.barh(ax=ax, color="#b85c2a")
ax.set_xlabel("Fraction Missing")
ax.set_title("Missing Values by Column")
ax.xaxis.set_major_formatter(lambda x, _: f"{x*100:.0f}%")
plt.tight_layout(); plt.show()

# Is missingness random or systematic?
df["age_missing"] = df["age"].isnull().astype(int)
print(df.groupby("age_missing")[["survived","pclass"]].mean())
# If stats differ by missingness, the data is Missing Not At Random (MNAR)

⚠️

MCAR vs. MAR vs. MNAR: Missing Completely At Random (safe to drop), Missing At Random (impute carefully), Missing Not At Random (the missing-ness is informative — keep a flag!). Always test whether missingness correlates with other variables before deciding what to do.

Summary

Explore before cleaning — raw data contains evidence of what's wrong.
Use histograms + KDE for distributions, boxplots for outliers, heatmaps for correlations.
IQR method for outliers (robust). Z-score assumes Gaussian.
Always check whether missing values are random — if not, add a binary missingness indicator.
Pairplots reveal feature interactions and potential class separability at a glance.