Pandas for Data Science

Why Pandas?

Real-world data is messy, heterogeneous, and labelled. NumPy works on uniform arrays; Pandas adds column names, row labels, mixed types, and a rich API for data wrangling — the day-to-day reality of ML engineering.

Core Data Structures

Series1-D labelled array. Like a NumPy array with an index. Think: one column of a spreadsheet.

DataFrame2-D labelled table of Series. Think: the spreadsheet itself. Columns can have different dtypes.

Python

import pandas as pd
import numpy as np

# ── Series
s = pd.Series([10, 20, 30, 40], index=["a","b","c","d"])
print(s["b"])          # 20  (label-based)
print(s.iloc[1])       # 20  (position-based)
print(s[s > 15])       # b:20  c:30  d:40

# ── DataFrame — from dict
df = pd.DataFrame({
    "name":    ["Alice", "Bob",  "Carol", "Dan"],
    "age":     [28,       35,    22,       41  ],
    "salary":  [75000,    92000, 58000,    110000],
    "dept":    ["Eng",    "Mkt", "Eng",    "Eng" ],
})

# Inspect
print(df.shape)         # (4, 4)
print(df.dtypes)        # object, int64, int64, object
print(df.head(2))       # first 2 rows
print(df.describe())    # count, mean, std, min, quartiles, max
print(df.info())        # dtypes + non-null counts

Selection & Filtering

Python

# Column selection
df["age"]                    # Series
df[["name","salary"]]        # DataFrame

# Row selection
df.loc[0]                    # row with label 0
df.loc[1:2, "name":"salary"] # rows 1-2, cols name to salary (inclusive)
df.iloc[0:2, 0:3]            # positional: rows 0-1, cols 0-2

# Boolean filtering
senior    = df[df["age"] > 30]
eng_high  = df[(df["dept"] == "Eng") & (df["salary"] > 70000)]
top3      = df.nlargest(3, "salary")

# Assign a new column
df["seniority"] = df["age"].apply(lambda x: "senior" if x >= 35 else "junior")
df["bonus"]     = df["salary"] * 0.1          # vectorised

GroupBy: Split-Apply-Combine

The most powerful Pandas pattern. Split the data into groups, apply an aggregation function, combine the results.

Python

import pandas as pd

# Avg salary & headcount per department
dept_stats = df.groupby("dept")["salary"].agg(["mean","count","std"])
print(dept_stats)

# Multiple columns
summary = df.groupby("dept").agg(
    avg_salary  = ("salary", "mean"),
    max_salary  = ("salary", "max"),
    avg_age     = ("age",    "mean"),
    headcount   = ("name",   "count"),
)

# Custom aggregation
dept_salary_range = df.groupby("dept")["salary"].agg(
    lambda x: x.max() - x.min()
)

# Transform: broadcast group-level stats back to original rows
df["dept_avg_salary"] = df.groupby("dept")["salary"].transform("mean")

Pivot Tables

Python

import pandas as pd, numpy as np

sales = pd.DataFrame({
    "month":    ["Jan","Jan","Feb","Feb","Mar","Mar"],
    "product":  ["A",  "B",  "A",  "B",  "A",  "B"],
    "revenue":  [1200, 800, 1500, 950, 1100, 1300],
})

# Pivot: rows=month, cols=product, values=revenue
pivot = sales.pivot_table(
    values  = "revenue",
    index   = "month",
    columns = "product",
    aggfunc = "sum",        # default is mean
    fill_value = 0          # fill NaN with 0
)
print(pivot)

# Reverse: melt wide → long format (common for plotting)
melted = pivot.reset_index().melt(id_vars="month", var_name="product", value_name="revenue")

Reading & Writing Data

Python

import pandas as pd

# CSV
df = pd.read_csv("data.csv", parse_dates=["date"], index_col="id")
df.to_csv("output.csv", index=False)

# Excel
df = pd.read_excel("data.xlsx", sheet_name="Sheet1")
df.to_excel("output.xlsx", index=False)

# Parquet (fast, compressed — preferred for large datasets)
df = pd.read_parquet("data.parquet")
df.to_parquet("output.parquet", index=False)

# From a URL
df = pd.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/titanic.csv")

Performance Tips

Use vectorised opsPrefer df["col"] * 2 over df["col"].apply(lambda x: x*2). The former is C-level; the latter is Python-level.

Use category dtypeConvert low-cardinality string columns: df["dept"] = df["dept"].astype("category"). Cuts memory 5–10× for large DataFrames.

Avoid iterrows()iterrows() is 100× slower than vectorised operations. Almost always replaceable with .apply(), .transform(), or direct array ops.

Read with dtypesPass dtype= to read_csv() to avoid loading everything as float64, saving memory.

Summary

DataFrame: 2-D labelled table. Series: 1-D labelled array.
Use .loc[] for label-based, .iloc[] for positional selection.
GroupBy: split → apply aggregation → combine. The workhorse of data analysis.
Pivot tables: reshape data from long to wide format (and back with melt()).
Use category dtype, vectorised ops, and Parquet for performance.