pandas Essentials — DataFrames for Real Data
Intermediatepandas DataFrames load, filter, transform and aggregate tabular data — read_csv, boolean filters, groupby/agg, and merge cover the analytics loop every engineer eventually needs.
Overview
pandas is Excel + SQL inside Python. A DataFrame is a table (columns = Series); one line reads a CSV, one expression filters rows, groupby aggregates like SQL GROUP BY, and merge joins like SQL JOIN. Backend engineers meet pandas in ETL scripts, report generation, and data debugging; data roles live in it. The five-operation core here — load, inspect, filter, derive, group — handles the overwhelming majority of practical tasks.
Load → Inspect → Filter → Derive
read_csv infers types; head/info/describe orient you; boolean expressions filter rows; new columns derive from old ones. loc selects by condition + column.
import pandas as pd
df = pd.DataFrame({
"name": ["Asha", "Ravi", "Neha", "Kiran"],
"branch": ["CS", "IT", "CS", "ME"],
"cgpa": [8.7, 7.2, 9.4, 6.8],
"backlogs": [0, 1, 0, 3],
})
# In real life: df = pd.read_csv("students.csv")
print(df.head()) # first rows
print(df.shape) # (4, 4)
df.info() # dtypes + nulls — ALWAYS check first
# Filtering — & | with parentheses (not and/or!)
eligible = df[(df.cgpa >= 7.0) & (df.backlogs == 0)]
print(eligible.name.tolist()) # ['Asha', 'Neha']
# Derived column
df["grade"] = df.cgpa.apply(lambda c: "A" if c >= 8.5 else "B")
# loc: rows by condition, specific columns
print(df.loc[df.branch == "CS", ["name", "cgpa"]])
# Sort + top-k
print(df.sort_values("cgpa", ascending=False).head(2))groupby, merge & Missing Data
groupby().agg() is SQL GROUP BY; merge() is JOIN (how= inner/left). isna/fillna/dropna handle the missing values every real dataset has.
import pandas as pd
df = pd.DataFrame({
"branch": ["CS", "IT", "CS", "ME", "IT"],
"cgpa": [8.7, 7.2, 9.4, 6.8, 8.1],
"placed": [True, False, True, False, True],
})
# GROUP BY branch: count, average, placement rate
summary = df.groupby("branch").agg(
students=("cgpa", "size"),
avg_cgpa=("cgpa", "mean"),
placement_rate=("placed", "mean"),
).round(2)
print(summary)
# students avg_cgpa placement_rate
# branch
# CS 2 9.05 1.0
# IT 2 7.65 0.5
# ME 1 6.80 0.0
# JOIN two tables
packages = pd.DataFrame({"branch": ["CS", "IT"], "avg_lpa": [12.5, 8.0]})
merged = df.merge(packages, on="branch", how="left") # ME gets NaN
# Missing data
print(merged.avg_lpa.isna().sum()) # 1
merged["avg_lpa"] = merged.avg_lpa.fillna(0)
# Out: merged.to_csv("report.csv", index=False)Key Points to Remember
- 1DataFrame = table; Series = column; read_csv/to_csv for I/O
- 2Filter with boolean expressions — & and | with parentheses, never and/or
- 3groupby().agg() ≈ GROUP BY; merge(on=, how=) ≈ JOIN
- 4Check df.info() and isna() first — real data always has holes
Interview Questions
Sign in to ask AriaFilter a DataFrame on two conditions — why do and/or fail here?
Show the pandas equivalent of SELECT branch, AVG(cgpa) ... GROUP BY branch.
How do you handle missing values — three strategies and when each applies?
Ask Aria about pandas Essentials — DataFrames for Real Data
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.