multiprocessing — True Parallelism for CPU-Bound Work
IntermediateEach process gets its own interpreter and its own GIL — ProcessPoolExecutor spreads pure-Python computation across all cores, at the cost of process startup and pickling data between processes.
Overview
multiprocessing sidesteps the GIL by running separate Python processes, one interpreter each, scheduled by the OS onto different cores. ProcessPoolExecutor gives the same map/submit API as its thread twin, so switching a workload from threads to processes is often a one-line change. The costs are real: each process must be started (slow spawn on Windows/macOS), and every argument and result is pickled and shipped across process boundaries — so sending huge data back and forth can eat the speedup. The if __name__ == "__main__" guard is mandatory: with the spawn start method, children re-import your module, and unguarded pool creation forks bombs.
ProcessPoolExecutor — Same API, Real Cores
The identical CPU-bound task that threads could not speed up scales nearly linearly across processes. The __main__ guard is not optional on Windows and macOS.
import time
from concurrent.futures import ProcessPoolExecutor
def heavy(n): # pure-Python CPU work
return sum(i * i for i in range(n))
if __name__ == "__main__": # REQUIRED: children re-import this module
jobs = [10_000_000] * 4
start = time.perf_counter()
seq = [heavy(n) for n in jobs]
print(f"sequential : {time.perf_counter() - start:.1f}s") # ~4.0s
start = time.perf_counter()
with ProcessPoolExecutor() as pool: # defaults to cpu_count() workers
par = list(pool.map(heavy, jobs))
print(f"4 processes: {time.perf_counter() - start:.1f}s") # ~1.1s on 4+ cores
assert seq == parThe Costs — Pickling, Startup, and When Not to Bother
Arguments and results cross processes by pickling. Ship small inputs and small outputs; keep big data inside the worker (e.g., pass a file path, not the file contents). Lambdas and locally-defined functions cannot be pickled — workers must be module-level functions.
from concurrent.futures import ProcessPoolExecutor
def word_count(path): # ship the PATH, not the file contents
with open(path, encoding="utf-8") as f:
return path, sum(len(line.split()) for line in f)
if __name__ == "__main__":
files = ["jan.log", "feb.log", "mar.log", "apr.log"]
with ProcessPoolExecutor() as pool:
for path, words in pool.map(word_count, files):
print(path, words)
# Gotchas:
# pool.map(lambda x: x * 2, data) ✗ PicklingError — no lambdas
# chunksize matters for many small tasks:
# pool.map(f, million_items, chunksize=1000) # batch to cut IPC overhead
# Choosing the tool:
# asyncio → thousands of concurrent I/O waits, single process
# threading → dozens of I/O waits, simplest mental model
# multiprocessing → CPU-bound pure Python
# none of these → NumPy/pandas already use C (and often multiple cores)Key Points to Remember
- 1One interpreter + one GIL per process → real parallel CPU execution
- 2if __name__ == "__main__": guard is mandatory with the spawn start method
- 3Everything crossing process boundaries is pickled — keep inputs/outputs small
- 4Workers must be module-level functions; lambdas and closures fail to pickle
Interview Questions
Sign in to ask AriaThreads vs processes vs asyncio in Python — when do you pick each?
Why does multiprocessing need the __main__ guard on Windows?
Your ProcessPool job is slower than the sequential version. List three likely reasons.
Ask Aria about multiprocessing — True Parallelism for CPU-Bound Work
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.