Profiling & Optimization — Measure Before You Tune
Advancedtimeit for micro-benchmarks, cProfile to find where a real program spends time — then apply the standard wins: builtins and comprehensions, set/dict lookups, functools.cache, and NumPy for numeric loops.
Overview
The first rule of Python performance is that intuition is usually wrong — measure first. timeit answers "which of these two snippets is faster" with proper repetition; cProfile answers "where do my 30 seconds actually go" for a whole program, and the top of its cumtime column is the only place worth optimizing. Most real speedups then come from a short standard playbook: push loops into C (builtins, comprehensions, str.join, NumPy), replace O(n) list membership with O(1) sets, cache repeated pure computation with functools.cache, and stop re-doing work inside hot loops. Rewriting in Rust/C is the last resort, not the first.
Measure: timeit and cProfile
timeit runs a snippet thousands of times for stable numbers — never benchmark with a single time.time() pair. cProfile ranks functions by cumulative time; optimize only the top entries.
import timeit
# Which is faster? Never guess — timeit it.
concat = timeit.timeit(
's = ""\nfor w in words: s += w',
setup='words = ["x"] * 1000',
number=2000,
)
join = timeit.timeit(
'"".join(words)',
setup='words = ["x"] * 1000',
number=2000,
)
print(f"+= loop : {concat:.3f}s")
print(f"join : {join:.3f}s") # typically ~10x faster
# Whole-program profiling:
# python -m cProfile -s cumtime app.py | head -20
# ncalls tottime cumtime function
# 1000 0.02 8.41 fetch_report ← optimize THIS
# 50000 3.90 3.90 parse_row
# cumtime = time including callees; start at the top, ignore the rest.
# Line-level detail: pip install line_profiler → @profile + kernprof -lvThe Standard Wins
Four changes cover most Python speedups: O(1) membership with sets, C-speed loops via builtins/comprehensions, functools.cache for repeated pure calls, and hoisting invariant work out of hot loops. For numeric arrays, NumPy vectorization is a 10-100x class of win.
from functools import cache
# 1. set/dict membership: O(1) vs list O(n)
blocked = {"9876543210", "9123456789"} # set, not list
print("9876543210" in blocked) # instant even with 1 crore entries
# 2. Builtins & comprehensions run the loop in C
nums = range(1_000_000)
total = sum(n * n for n in nums) # beats a manual += loop
squares = [n * n for n in nums] # beats append() in a for loop
# 3. Cache repeated pure computation
@cache
def fib(n):
return n if n < 2 else fib(n - 1) + fib(n - 2)
print(fib(300)) # instant — exponential → linear
# 4. Hoist invariants out of hot loops
import re
PIN = re.compile(r"^41\d{4}$") # compile ONCE, outside the loop
codes = ["413001", "411045", "560001"]
valid = [c for c in codes if PIN.match(c)]
# Numeric arrays? Vectorize:
# import numpy as np
# arr = np.array(list(nums)); total = int((arr * arr).sum()) # C speedKey Points to Remember
- 1Measure first: timeit for snippets, cProfile -s cumtime for programs
- 2Optimize only the top of the profile — everything else is noise
- 3Sets/dicts for membership, builtins/comprehensions for loops, cache for pure repeats
- 4NumPy vectorization for numeric arrays; native rewrites are the last resort
Interview Questions
Sign in to ask AriaA Python API endpoint takes 3 seconds. Describe your exact steps to find the bottleneck.
Why is "".join(list) faster than += concatenation in a loop?
What does functools.cache do, and when is it wrong to use it?
Ask Aria about Profiling & Optimization — Measure Before You Tune
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.