Python — Cheat Sheet

Every value — integers, functions, classes, modules, even types themselves — is an object with an identity, a type, and attributes. The consequences are practical. Functions can be assigned to variables, passed as arguments and returned, which is what makes decorators and callbacks natural rather than a special feature. Classes are objects too, instances of type, which is what metaclasses manipulate. It also explains the assignment model. A variable is a name bound to an object, not a box containing a value. Assignment binds a name; it does not copy. So two names can refer to the same object, and mutating through one is visible through the other — which is the source of most surprising aliasing bugs. id() gives the identity, and is tests identity while == tests equality. Confusing them is a common mistake, and it is why comparing to None uses is. The small-integer cache makes this visible: 256 is 256 is True because CPython caches small integers, while 257 is 257 may be False. That is an implementation detail, not a language guarantee, and relying on it is a bug.

Immutable objects cannot be changed after creation — int, float, str, tuple, frozenset, bytes. Mutable ones can — list, dict, set, bytearray, and most custom classes. It matters in several places. Function arguments: passing a mutable object lets the function modify the caller's object, since both names refer to the same thing. Passing an immutable one cannot, because any apparent modification rebinds the local name. Dictionary keys and set members must be hashable, and mutable built-ins are not — which is why a list cannot be a key but a tuple can. A tuple containing a list is itself unhashable, since hashability is recursive. Default arguments are evaluated once at definition time, so a mutable default is shared across every call, which is the classic bug. And equality of an immutable value is stable, so caching and interning are safe. The practical guidance is to prefer immutable types for anything shared, and to be deliberate about returning a mutable internal — returning self._items hands the caller the ability to modify your object.

Default arguments are evaluated once, when the function is defined, not on each call. So a default of [] creates one list that is shared by every invocation. A function that appends to that default accumulates across calls: the first call returns one item, the second returns two, and nobody expects it. It is the single most common Python gotcha and it appears in production code regularly, usually as a cache that never resets or a list that grows forever. The fix is to default to None and create the object inside the function body, which gives a fresh one per call. The reason it works this way is consistency: the def statement is executed, and its default expressions are evaluated at that point like any other expression. Evaluating them per call would require storing the expression rather than the value, which is a different language design — one that Python deliberately did not take. It is not always a bug: the same mechanism is occasionally used deliberately for memoisation or to bind a value at definition time. But that is rare enough that a mutable default should be treated as a mistake until proven otherwise, and linters flag it.

is tests identity — whether two names refer to the same object. == tests equality, which calls __eq__ and can be defined however the type wants. Use is only for singletons: None, True, False. The idiom is x is None, and it is preferred over == None because it is faster and cannot be subverted by a class defining a strange __eq__. The trap is small-value caching. CPython interns small integers and some strings, so a is b can be True for values that were computed separately, and False for slightly larger ones. Code that appears to work with is on integers breaks when the values grow past 256. That behaviour is an implementation detail, not a guarantee, and it differs between interpreters. Python 3.8 added a SyntaxWarning for is comparisons against literals, precisely because people write it by mistake. The related point is that == on a custom class defaults to identity unless __eq__ is defined, so two objects with identical attributes are unequal by default — which surprises people coming from languages with structural equality. Dataclasses generate __eq__ for you.

Name resolution searches four scopes in order: Local, Enclosing, Global, Built-in. Local is the current function. Enclosing is any outer function, which is what makes closures work. Global is module level. Built-in is the interpreter's namespace. The rule that surprises people is that assigning to a name anywhere in a function makes it local for the entire function, including before the assignment. So reading a global and then assigning to it raises UnboundLocalError on the read, which looks contradictory until you know the rule. global declares that assignments refer to the module-level name. nonlocal, added in Python 3, refers to the nearest enclosing function scope, which is how a closure can modify a variable in its enclosing function rather than only read it. The other thing worth knowing is what does not create a scope. Loops, if statements and with blocks do not — a variable assigned inside a for loop is still visible after it. Only functions, classes, modules and comprehensions create scopes. Comprehensions having their own scope in Python 3 is why their loop variable does not leak, unlike Python 2.

A shallow copy creates a new container whose elements are references to the same objects as the original. A deep copy recursively copies the objects too. So copying a list of lists shallowly gives you a new outer list, but the inner lists are shared — mutating one is visible through both. That is the bug people hit when they copy a nested structure and are surprised the original changed. list(), slicing with [:], and copy.copy() are all shallow. copy.deepcopy() is deep. Deep copy is expensive, since it walks the entire object graph, and it handles cycles by tracking what it has seen. It also copies things you may not want copied — a deep copy of an object holding a database connection tries to copy the connection. A class can control this with __copy__ and __deepcopy__. The pragmatic guidance is that if you find yourself reaching for deepcopy regularly, the design is probably relying on mutation where immutability would be simpler. An immutable structure needs no copying at all, which is why dataclasses with frozen=True and tuples avoid the question entirely.

Double-underscore methods define how a class participates in language protocols. They are the hooks the interpreter calls for operators and built-in functions. The ones that matter most in practice. __init__ initialises an instance; __new__ actually creates it, and is only needed for immutable types or singletons. __repr__ should give an unambiguous representation for developers — ideally something that could reconstruct the object. __str__ is the human-readable form and falls back to __repr__. Defining __repr__ is high value and often skipped, and it is what makes debugging and logging useful. __eq__ and __hash__ must be consistent: objects that compare equal must hash equal. Defining __eq__ without __hash__ makes the class unhashable in Python 3, which is a deliberate safety measure. __enter__ and __exit__ make a context manager. __iter__ and __next__ make an iterator. __len__, __getitem__ and __contains__ make a container behave like one. __call__ makes an instance callable. The design idea is that Python's built-in behaviour is protocol-based, so your types can participate fully rather than being second class.

__new__ creates and returns the instance. __init__ initialises the instance that __new__ returned. __new__ is a static method that receives the class; __init__ receives the already-created object. The order is that __new__ runs first, and __init__ runs only if __new__ returned an instance of that class. You almost never need __new__. The cases where you do: subclassing an immutable type such as str, int or tuple, because by the time __init__ runs the value is already fixed and cannot be changed. Implementing a singleton or an object cache, where you want to return an existing instance rather than a new one. And metaclass work. The common mistake is overriding __new__ and forgetting to return the instance, or forgetting to call super().__new__, which produces confusing failures because __init__ then never runs. For the singleton case, the honest observation is that a module-level instance is simpler and more Pythonic than overriding __new__, since modules are already singletons. Reaching for __new__ to implement one is usually importing a pattern from another language where it was necessary.

Any object can be used in a boolean context. Python calls __bool__ if defined, otherwise __len__, and treats a length of zero as false. With neither, the object is true. So falsy values are: False, None, zero of any numeric type, and empty containers — empty string, list, tuple, dict, set, and range. The practical idiom is if items: rather than if len(items) > 0:, which is both more idiomatic and works for any container. The trap is that this conflates "empty" with "absent". A function returning either a list or None can be tested with if result:, but that treats an empty list the same as None — which is often wrong. When the distinction matters, test explicitly with is None. The same applies to zero: a config value of 0 is falsy, so if timeout: silently falls back to a default when the caller explicitly asked for zero. That is a real bug pattern, and the fix is if timeout is not None. Numpy arrays raise on truthiness testing precisely because the answer is ambiguous for multiple elements.

Primarily by reference counting: every object tracks how many references point at it, and when the count reaches zero it is deallocated immediately. That gives deterministic cleanup — an object goes away as soon as the last reference does, which is why with blocks and __del__ behave predictably in CPython. Reference counting cannot handle cycles: two objects referring to each other keep each other alive even when nothing else does. So a generational garbage collector runs periodically to detect and collect cycles. The collector is generational because most objects die young, so newer generations are scanned more often. gc.collect() forces a pass, and gc.disable() is occasionally used in latency-sensitive code. The practical implications. Reference cycles are created easily by a parent-child structure where the child holds a reference back, and weakref is how you break them. CPython also does not always return freed memory to the operating system — small object arenas are reused — so RSS can stay high after a large workload completes. And __del__ on objects in a cycle historically prevented collection, though that was fixed in Python 3.4.

Duck typing means an object's suitability is determined by the methods it has, not by its declared type. If it has read(), it can be used where a readable file is expected, regardless of its class. That is why Python code typically accepts anything supporting the operations it uses, rather than checking types. Type hints do not change runtime behaviour — the interpreter ignores them — but they let a static checker verify consistency. The tension is that a hint of a concrete class narrows what is accepted, undermining duck typing. The resolution is structural typing with Protocol from typing. A Protocol defines the methods required, and any class with those methods satisfies it without inheriting from anything. That expresses duck typing in the type system, which is exactly what was missing. The alternative, abstract base classes, is nominal — a class must register or inherit — which is stricter and less Pythonic. The practical advice is to hint parameters with the most general thing that works: Iterable rather than list, Mapping rather than dict, and a Protocol when you need specific methods. Hinting concrete types on parameters is the common mistake.

The assignment expression, :=, assigns and returns a value in one expression. The cases where it genuinely helps: avoiding a duplicate call in a condition, as in if (match := pattern.search(line)): which binds the result and tests it without calling search twice or assigning on a separate line. In a while loop reading until a sentinel. And in a comprehension where you need an intermediate value, avoiding computing it twice. The cases where it hurts readability are those where it is used purely for compactness. Nesting assignments inside complex expressions makes code harder to follow, and Python deliberately kept assignment as a statement for decades to avoid the C family's if (x = y) bug class. The practical guideline is that it is worth it when it removes a genuine duplication or a clumsy pre-loop read, and not worth it when it merely saves a line. It was contentious enough that the debate contributed to Guido stepping down as BDFL, which is a piece of context worth knowing if the question is asked conversationally.

*args collects extra positional arguments into a tuple. **kwargs collects extra keyword arguments into a dict. The names are convention; the asterisks do the work. At a call site the same syntax unpacks: f(*items) spreads a sequence into positional arguments, f(**mapping) spreads a dict into keyword arguments. Python 3 added two markers that constrain how parameters may be passed. A bare * in the signature makes everything after it keyword-only, which is valuable for optional flags — it prevents callers passing a bare True whose meaning is unreadable at the call site. A / makes everything before it positional-only, added in 3.8. That lets you rename a parameter later without breaking callers who passed it by keyword, which matters for library APIs. The practical guidance: make boolean and configuration parameters keyword-only, because f(data, verbose=True) reads and f(data, True) does not. And use positional-only sparingly, mainly in libraries where you want freedom to rename. The built-ins have used positional-only semantics for a long time, which is why you cannot call len(obj=x).

On import, Python checks sys.modules for an already-imported module and returns it if present — so a module is executed only once per process, which is what makes module-level state effectively a singleton. If not cached, it searches sys.path in order, finds the module, executes it top to bottom, and stores the resulting module object. The consequences worth knowing. Module-level code runs on first import, so expensive work or side effects at import time slow every startup and can create surprising ordering dependencies. Circular imports fail when module A imports B which imports A, and A is only partially executed — the name B needs does not exist yet. The fixes are to move the import inside a function, to restructure so the dependency is one-directional, or to import the module rather than a name from it. Relative imports with dots refer to package-relative locations and only work inside a package. And the difference between running a file as a script and importing it changes __name__ and the package context, which is why python -m matters for packages.

A list is a dynamic array of pointers to objects, not a linked list. Indexing and assignment by index are O(1). Appending is amortised O(1), because the array over-allocates and only occasionally reallocates and copies. Inserting or deleting at an arbitrary position is O(n), since everything after it shifts. Membership testing with in is O(n), because it scans. That is the one people get wrong: a loop containing if x in some_list is quadratic, and converting the list to a set makes it linear. It is the most common accidental O(n²) in Python. len() is O(1) since the length is stored. Sorting is O(n log n) using Timsort, which is adaptive — it exploits existing runs of sorted data, so partially sorted input is much faster than the bound suggests. The over-allocation means a list uses more memory than the number of elements implies. For large homogeneous numeric data, array or numpy is dramatically more compact, since a list stores pointers to boxed objects rather than raw values. collections.deque gives O(1) at both ends, which a list does not.

A dict is a hash table. Lookup, insertion and deletion are O(1) average, degrading to O(n) in the pathological case of many collisions. Keys must be hashable, meaning they implement __hash__ and their hash does not change — which is why mutable built-ins cannot be keys. Since Python 3.7, insertion order is a language guarantee. It was an implementation detail in 3.6 as a side effect of the compact dict layout, and was then promoted to a guarantee. That layout also reduced memory use considerably by separating a dense array of entries from a sparse index array. The practical consequences. Membership testing on a dict or set is O(1) versus O(n) for a list, which is the single most valuable performance fact in Python. dict.get with a default avoids a KeyError and is cleaner than checking membership first. collections.defaultdict handles accumulate-into-a-list patterns. And dict.setdefault exists but is less readable than defaultdict for most uses. Sets are the same machinery without values, so the same complexity applies.

When the collection is fixed and its positions have meaning, rather than being a homogeneous sequence that might grow. The practical reasons: tuples are immutable, so they can be dictionary keys and set members, and they are safe to share without defensive copying. They are slightly smaller and faster to construct. But the stronger reason is semantic. A list signals "many of the same thing, order may change, length may change". A tuple signals "a fixed structure where position matters" — a coordinate, a database row, a return value with several parts. The convention that follows: functions returning multiple values return a tuple, and unpacking at the call site is idiomatic. For anything with more than two or three fields, a NamedTuple or a dataclass is better than a bare tuple, because accessing by index becomes unreadable and adding a field silently breaks every unpacking site. The caveat worth knowing: a tuple is only immutable at the top level. A tuple containing a list can have that list mutated, and the tuple is then unhashable — which is why hash() raises on it.

defaultdict takes a factory called when a missing key is accessed, inserting the produced value automatically. The common use is accumulation. Grouping items by a key with a plain dict requires checking whether the key exists and creating a list if not; with defaultdict(list) you just append. Counting with defaultdict(int) works the same way. It is cleaner than dict.setdefault, which creates the default object on every call whether needed or not, and clearer than a try/except KeyError. The behaviour to be aware of is that merely reading a missing key inserts it. So checking whether a key exists with d[key] adds it, and iterating after a series of lookups shows keys you never meant to create. That surprises people and can cause memory growth. Use the in operator or .get() to test without inserting. For counting specifically, collections.Counter is better still — it has most_common, supports arithmetic between counters, and returns zero for missing keys without inserting them. And a defaultdict does not serialise to JSON differently, but it does compare equal to a plain dict with the same contents.

collections.deque is a double-ended queue implemented as a doubly linked list of blocks, giving O(1) append and pop at both ends. A list gives O(1) at the end but O(n) at the front, because inserting or removing at index 0 shifts every element. So using list.pop(0) in a loop is quadratic — a common and easily missed performance bug in queue-shaped code. deque fixes that, which makes it the right structure for queues, for breadth-first search, and for any sliding window over a stream. It also supports maxlen, which makes it a fixed-size ring buffer: appending past the limit discards from the other end automatically. That is a neat way to keep the last N items without manual trimming — useful for recent-events buffers and moving averages. rotate() shifts elements around, which is occasionally handy. The trade-off is that indexing into the middle is O(n) rather than O(1), so it is not a list replacement. If you index randomly, use a list; if you add and remove at the ends, use a deque. It is also thread-safe for appends and pops.

Both are unordered collections of unique hashable elements with O(1) membership testing. A set is mutable; a frozenset is not. The practical consequence is that a frozenset is itself hashable, so it can be a dictionary key or an element of another set. A regular set cannot. That matters when you need to key something by a group — mapping a set of permissions to a role, or deduplicating collections of items. Sets support the mathematical operations directly: union with |, intersection with &, difference with -, symmetric difference with ^. Those are much clearer and much faster than the equivalent loops, and using them is one of the easy wins in Python code — replacing a nested loop that checks membership with a single intersection. The things to remember: sets are unordered, so iteration order is arbitrary and must not be relied on. Elements must be hashable, so a set of lists is impossible. And the empty set is set(), not {}, which creates an empty dict — a small trap. For deduplication while preserving order, dict.fromkeys is the idiom.

A dataclass generates __init__, __repr__ and __eq__ from annotated class attributes, removing the boilerplate of a data-holding class. Compared to a plain class it is less code and the generated methods are correct — particularly __repr__, which people skip and then regret when debugging. Compared to a NamedTuple, a dataclass is mutable by default and is a normal class, so it can have methods, inheritance and default factories naturally. A NamedTuple is a tuple, so it is immutable, indexable, unpackable and slightly lighter, and it works where a tuple is expected. frozen=True makes a dataclass immutable and hashable, which is the right choice for value objects. slots=True, added in 3.10, removes the per-instance dict and saves substantial memory for many instances. field(default_factory=list) is how you give a mutable default safely, avoiding the shared-default bug. The rough guidance: dataclass for most structured data, frozen for value objects, NamedTuple when you need tuple behaviour or maximum lightness, and Pydantic when you need runtime validation and parsing rather than just structure.

Pass a key function to sorted() or list.sort(). The key is called once per element and the results are compared, which is the decorate-sort-undecorate pattern built in. That is why key beats the old cmp approach: a comparison function is called O(n log n) times, while a key function is called n times. For multiple criteria, return a tuple: sorting by department then by name is key=lambda p: (p.department, p.name). Tuples compare element-wise, so this works naturally. Mixing ascending and descending is the awkward case, since reverse applies to everything. For numbers you can negate; for strings you cannot, so the idiom is to sort twice — Python's sort is stable, so sorting by the secondary key first and then by the primary gives the correct combined order. operator.itemgetter and attrgetter are faster than an equivalent lambda because they are implemented in C, and they read well. sort() sorts in place and returns None; sorted() returns a new list and accepts any iterable. Returning the result of .sort() is a classic mistake that silently produces None.

A list comprehension builds the entire list in memory immediately. A generator expression produces values lazily, one at a time, holding only the current state. The syntax differs only in the brackets — square for a list, parentheses for a generator. The practical difference is memory. Summing a computation over ten million items with a list comprehension allocates ten million objects; with a generator expression it allocates one at a time. For large or unbounded data the generator is the only viable option. When a generator is passed as the sole argument to a function, the parentheses can be omitted — sum(x*x for x in data) — which is the idiomatic form. The cases where a list is right: you need to iterate more than once, since a generator is exhausted after one pass; you need len() or indexing; or the collection is small and you want the simplicity. The trap is exhaustion. Passing a generator to something that iterates twice silently gets an empty second pass, with no error. That is a real bug source, and it is why functions taking an iterable should document whether they consume it once.

Since Python 3.9, the | operator merges two dicts producing a new one, and |= merges in place. That is the clearest form. Before that, {**a, **b} unpacking was the idiom, which also works and is still common. dict.update() mutates the left-hand dict rather than producing a new one, which is what you want when updating in place and not what you want when you need a copy. In all cases, later keys win — b overrides a for duplicate keys. That is the behaviour to state, since it is the only real semantic question. The important limitation is that all of these are shallow. Merging two dicts whose values are themselves dicts replaces the nested dict entirely rather than merging it, so a configuration merge with nested sections loses everything in the section that was overridden. That catches people building layered configuration. A deep merge has to be written by hand or taken from a library, and it requires a decision about how to combine lists — replace or concatenate — which is why there is no obvious built-in. collections.ChainMap is the alternative for lookup layering without copying.

Counter is a dict subclass for counting hashable items. Constructing one from an iterable counts occurrences in a single pass. The useful parts beyond counting: most_common(n) returns the top n by count, which is the thing you would otherwise write with sorted() and a lambda. Missing keys return zero rather than raising, so you can query freely without checking. Counters support arithmetic — adding two counters sums their counts, subtracting removes them, and & and | give minimum and maximum per key. That makes set-like operations on multisets natural. The typical applications: word frequency, tallying events by type, finding duplicates, and comparing two collections by content rather than order — two strings are anagrams if their Counters are equal, which is both the clearest and one of the fastest ways to express it. The behaviour worth noting: subtraction keeps zero and negative counts, while the subtract() method and the - operator differ in this. And elements() expands back into an iterable, repeating each item by its count. It is a small part of the standard library that removes a surprising amount of hand-written code.

heapq implements a binary min-heap over a plain list. Push and pop are O(log n), and the smallest element is always at index 0. The uses: a priority queue, and finding the k largest or smallest items in a stream. nlargest and nsmallest are provided, and for small k they beat sorting the whole collection. The idiom for a max-heap is to negate the values, since only a min-heap is provided. For objects, push tuples of (priority, item), and include a tiebreaker such as a counter, because if priorities tie Python compares the next tuple element and objects may not be comparable — that raises TypeError and is a classic bug. bisect performs binary search on a sorted list. bisect_left and bisect_right find insertion points in O(log n), and insort inserts while maintaining order — though the insertion itself is O(n) because of the shift. The practical uses: maintaining a sorted collection with fast lookup, and mapping a value to a bucket, such as converting a score to a grade with one call instead of a chain of comparisons. Both are underused given how often the patterns appear.

A closure is a function that captures variables from its enclosing scope and keeps them alive after that scope has returned. The gotcha is that closures capture the variable, not its value at creation time. So creating functions in a loop and capturing the loop variable gives every function the final value. The classic case is [lambda: i for i in range(3)] — calling all three returns 2, 2, 2, not 0, 1, 2. Every lambda refers to the same i, which is 2 when the loop ends. The fix is to bind the value at definition time with a default argument: lambda i=i: i. Defaults are evaluated when the function is defined, so each gets its own value. functools.partial does the same thing more explicitly. This appears in real code whenever callbacks or handlers are created in a loop — event handlers, retry wrappers, per-column formatters — and it is genuinely confusing the first time because each function looks independent. nonlocal is the related keyword: it lets a closure rebind a variable in the enclosing scope rather than only read it, which is how a counter-in-a-closure works.

A decorator is a callable that takes a function and returns a replacement. The @ syntax is sugar: @log above a definition means the function is passed to log and the name is rebound to whatever log returns. The usual implementation is a wrapper function defined inside, which does something before and after calling the original, and is returned. The detail that matters is functools.wraps. Without it, the wrapper replaces the original's __name__, __doc__, __module__ and signature, so introspection, documentation tools and debuggers all report the wrapper instead. @wraps copies that metadata across, and omitting it is the single most common decorator mistake. A decorator taking arguments needs one more level: a function that takes the arguments and returns the actual decorator. Three nested functions, which is why they read awkwardly. Decorators stack bottom-up: the one closest to the def is applied first. They are the natural mechanism for cross-cutting concerns — timing, caching, retry, authorisation, logging — because they add behaviour without modifying the function. The caution is that heavy stacking makes stack traces deep and control flow hard to follow.

It memoises a function, storing results keyed by the arguments and returning the cached value on a repeat call, evicting least-recently-used entries past maxsize. It is excellent for pure, expensive, repeatedly-called functions — recursive computations, configuration parsing, lookups that hit a slow source. Where it is unsafe or wrong. The function must be pure. Caching something with side effects means the side effects stop happening on cache hits, which is a subtle and nasty bug. Arguments must be hashable, so a function taking a list or dict cannot be cached without conversion. It holds strong references to arguments and results, so caching a method keyed on self keeps every instance alive forever — a real memory leak, and a common one because decorating a method looks harmless. Unbounded caching with maxsize=None grows without limit. And it is per-process, so in a multi-worker deployment each worker has its own cache, which matters for consistency. functools.cache in 3.9 is lru_cache(maxsize=None), and cached_property caches on the instance, which avoids the leak for that case.

A method is a function accessed through an instance or class, with the descriptor protocol binding it. Accessing instance.method returns a bound method that passes the instance as the first argument automatically, which is why self appears in the definition but not at the call site. staticmethod removes that binding entirely — the function receives exactly the arguments passed, with no implicit first parameter. It is a function that lives in the class namespace for organisational reasons. Often a module-level function would be equally good, and reaching for staticmethod is sometimes a habit imported from Java. classmethod binds the class rather than the instance as the first argument, conventionally named cls. Its main use is alternative constructors: a from_json or from_row that builds an instance. Because it receives the actual class, it works correctly with subclasses — a subclass calling the alternative constructor gets an instance of the subclass, which a staticmethod hard-coding the class name would not. That subclass correctness is the real argument for classmethod over staticmethod for factories, and it is the part usually missed.

A decorator factory taking the retry count, the backoff and the exceptions to catch, returning a decorator whose wrapper loops. The elements that matter. Catch specific exceptions, not everything — retrying a TypeError caused by a bug wastes time and hides the error. Retry on transient failures only. Exponential backoff with jitter, because fixed-interval retries from many callers synchronise into a thundering herd against a service that is already struggling. A cap on total attempts and ideally on total elapsed time, so a retry cannot outlive the caller's own deadline. Re-raise the last exception after exhausting attempts rather than returning None, or the caller cannot distinguish failure from a legitimate empty result. Use functools.wraps so the decorated function keeps its identity. Log each retry with the attempt number, or a service silently retrying looks like a slow one. The honest addition is that tenacity already does all of this well, including async support and rich stop and wait conditions, so writing your own is worth doing once as an exercise and reaching for the library in production. And retries must only wrap idempotent operations.

A context manager defines setup and teardown for a with block, guaranteeing the teardown runs even if the body raises. The class form implements __enter__, which returns the value bound by as, and __exit__, which receives the exception type, value and traceback if one occurred. Returning a truthy value from __exit__ suppresses the exception, which should be done deliberately and rarely. The simpler form is contextlib.contextmanager on a generator: code before yield is setup, the yielded value is what as binds, and code after is teardown. Wrapping the yield in try/finally is what makes teardown run on exception, and omitting that is the common mistake — without it, an exception in the body skips the cleanup entirely. The uses beyond files and locks: temporarily changing state and restoring it, timing a block, database transactions, and suppressing exceptions with contextlib.suppress. contextlib.ExitStack handles a dynamic number of context managers, which is what you need when the count is not known at compile time. And async context managers use __aenter__ and __aexit__ with async with.

Functions are objects, so they can be stored, passed and returned. That makes higher-order functions — those taking or returning functions — natural rather than exotic. The everyday uses: key functions for sorting, callbacks, and the strategy pattern expressed as a plain function rather than a class hierarchy. functools.partial pre-binds arguments, returning a new callable with fewer parameters. It is useful for adapting a general function to an interface expecting a narrower one — supplying a configured logger, or fixing a base URL — and it is clearer than a lambda because it preserves introspection and reads as specialisation rather than as an anonymous wrapper. It is also the correct fix for the late-binding closure problem, since it binds values at creation. functools.reduce exists but is rarely the clearest option; an explicit loop or sum, min and max usually read better, which is why reduce was moved out of builtins in Python 3. The broader point is that Python supports functional composition without being a functional language, and the idiomatic use is light — a key function here, a partial there — rather than building deep combinator chains.

A lambda is an expression producing an anonymous single-expression function. Use it where a small function is needed inline and naming it would add nothing — a sort key, a simple map or filter, a default factory. Do not use it where a named function would be clearer, which is more often than lambdas appear in real code. The specific cases to avoid. Assigning a lambda to a name is pointless — def gives the same thing with a proper name in tracebacks, and PEP 8 says so explicitly. A lambda with complex logic squeezed into one expression, using conditional expressions and tuple tricks, is harder to read than three lines of def. Lambdas as event handlers created in a loop hit the late-binding trap. And a lambda cannot contain statements, so anything needing a try block or an assignment cannot be one. The replacements worth knowing: operator.itemgetter and attrgetter are faster and clearer than the equivalent lambda for sort keys, and methodcaller covers calling a method. The test is whether the reader has to pause. If the lambda takes a moment to parse, name it.

A plain function decorator generally works on methods too, because a method is just a function at class-definition time — self arrives as the first positional argument and passes through *args unchanged. So the usual wrapper signature of (*args, **kwargs) handles both without special cases. Where it breaks down: if the decorator needs to inspect or use self, it cannot assume args[0] is an instance, because on a plain function it is not. If the decorator caches, keying on all arguments includes self, which keeps instances alive — the lru_cache-on-a-method leak. And if the decorator returns something that is not a function — a class instance implementing __call__ — the descriptor protocol does not apply, so it is not bound to the instance and self is not passed. Fixing that requires implementing __get__ on the decorator class, which is where descriptors become necessary. The practical guidance is to write decorators as functions returning functions, use functools.wraps, and accept *args and **kwargs. That covers functions, methods, classmethods and staticmethods without branching — provided the decorator is applied in the right order relative to classmethod, which must be outermost.

It implements single-dispatch generic functions: a function whose implementation is selected by the type of its first argument. You decorate a base implementation with @singledispatch, then register type-specific versions with @base.register. Calling the function dispatches to the matching registration, falling back to the base. The motivation is replacing a chain of isinstance checks. A serialiser handling several types with if/elif becomes a set of independent registrations, and adding a type means adding a function rather than editing the chain — the Open/Closed argument applied to a function. It also allows registering handlers for types you do not own, which subclassing cannot. singledispatchmethod does the same for methods, dispatching on the first argument after self. The limitations: dispatch is on one argument only, so multiple dispatch needs a different approach. Registration is global to that function, so two libraries registering for the same type conflict. And in modern code, structural pattern matching with match handles many of the same cases more visibly. It is genuinely useful for extensible serialisation and formatting, and largely unknown.

They are for tooling and for people, not for the interpreter. A static checker — mypy, pyright — catches whole classes of error before execution: passing the wrong type, forgetting that a function can return None, misspelling an attribute. On a large codebase that is substantial value, and it scales with size in a way that tests do not. Editors use them for completion and inline errors, which is a real productivity difference. And they are documentation that cannot go stale silently, because the checker verifies it. Some libraries do use them at runtime: Pydantic validates and coerces from annotations, FastAPI derives request parsing and OpenAPI schemas from them, and dataclasses read them to generate methods. So in those contexts they are load-bearing. The practical advice: hint public function signatures first, since that is where the value concentrates. Use the most general parameter types that work — Iterable rather than list — and specific return types. Optional means "may be None" and should be explicit rather than implied by a default. And adopt gradually; mypy is designed for partial coverage.

Catch specific exceptions. A bare except catches everything including KeyboardInterrupt and SystemExit, so Ctrl-C stops working — except Exception is the broad-but-sane version, and even that should be rare outside a top-level handler. Use else for the code that runs when no exception occurred, keeping the try block to only the statements that can fail. That prevents accidentally catching an exception from unrelated code. Use finally, or better a context manager, for cleanup. Chain exceptions with raise NewError(...) from original, which preserves the cause in the traceback. Raising without from inside an except block still records the context automatically, but from expresses intent and lets you suppress it with from None when the original is noise. Never swallow silently. except: pass discards information and is how failures go unnoticed for months. Define custom exception types for your domain, inheriting from a single base for the package so callers can catch broadly or narrowly. And prefer asking forgiveness over permission — try the operation rather than checking first — since that is idiomatic and avoids a race between the check and the use.

Python allows multiple inheritance and resolves attribute lookup using the C3 linearisation, which produces a method resolution order — a single ordered list of classes to search. C3 guarantees that a class appears before its parents, and that the order of parents in the class definition is preserved. If no consistent order exists, the class definition itself raises a TypeError rather than failing mysteriously later. You can inspect it with ClassName.__mro__. super() does not mean "the parent class" — it means "the next class in the MRO of the actual instance". That distinction is the key to understanding cooperative multiple inheritance: in a diamond, the shared base is called once rather than twice, because super() follows the linearisation rather than the static hierarchy. For that to work, every class in the chain must call super() and accept compatible arguments, which is what makes cooperative inheritance fragile — one class that does not cooperate breaks the chain. The practical advice is to keep hierarchies shallow and to use multiple inheritance mainly for mixins that are narrow and orthogonal. Deep diamonds are technically supported and hard to reason about.

A property makes an attribute access run a method. @property defines the getter, and @name.setter defines the setter. The idiomatic use is to start with a plain public attribute and convert it to a property later if validation or computation becomes necessary — without changing any calling code, because the access syntax is identical. That is why Python does not have the Java convention of getters and setters everywhere: they add ceremony for a flexibility that properties provide when actually needed. So writing get_x and set_x methods in Python is a code smell rather than good practice. Good uses: validating on assignment, computing a derived value, and making an attribute read-only by defining only a getter. The caution is that a property hides work behind what looks like an attribute access. If the getter performs a database query or an expensive computation, callers will use it in a loop assuming it is free. For anything expensive, a method named explicitly is more honest — or functools.cached_property, which computes once and stores the result on the instance. And properties are class-level descriptors, so they do not work on instances.

An object defining __get__, __set__ or __delete__ is a descriptor, and placing one as a class attribute means attribute access on instances goes through it. A data descriptor defines __set__ or __delete__ and takes priority over the instance dict. A non-data descriptor defines only __get__ and is overridden by an instance attribute of the same name. That priority rule explains real behaviour: a property cannot be shadowed by setting the instance attribute, while a cached_property can — which is exactly how it caches, by writing into the instance dict so subsequent lookups skip the descriptor. Descriptors are the machinery behind properties, methods, classmethod, staticmethod and slots. Every method access is a descriptor call producing a bound method, which is why self is passed automatically. You rarely write one directly. The cases where you would: a reusable validated attribute applied across many classes, a typed field, or ORM-style column definitions — which is exactly what Django and SQLAlchemy do. __set_name__, added in 3.6, tells a descriptor its own attribute name at class creation, which removes the need to repeat the name.

__slots__ declares a fixed set of attributes, and the class then stores them in a compact array rather than a per-instance dictionary. The benefit is memory. A normal instance carries a dict, which has significant overhead; slots can cut per-instance memory substantially — often by half or more for small objects. With millions of instances that is the difference between fitting in memory and not. Attribute access is also marginally faster. The costs: you cannot add attributes not declared in slots, which is sometimes a feature and sometimes an obstacle. There is no __dict__ unless you add it explicitly, which breaks code that inspects or patches instances — including some serialisation and mocking tools. Multiple inheritance with slots on both parents is restricted. And a subclass without slots reintroduces the dict, silently losing the benefit. The practical guidance is to use it for classes instantiated in very large numbers, and not to reach for it by default — the memory saving is irrelevant for a handful of objects and the constraints are real. Dataclasses support slots=True since 3.10, which makes it a one-word change.

A metaclass is the class of a class. Since classes are objects, they have a type, and by default that type is type. A metaclass customises class creation — it runs when the class is defined, not when instances are created. The honest answer to when you need one is: almost never. Tim Peters' observation applies — if you are not sure you need a metaclass, you do not. The legitimate uses are frameworks: registering subclasses automatically, validating that a class defines required attributes, transforming class attributes into something else. ORMs use them to turn column declarations into descriptors, and ABCMeta uses one to enforce abstract methods. The reason to avoid them is that they are invisible. Someone reading a class has no indication that its behaviour was altered at creation, which makes debugging genuinely hard. The lighter alternatives cover most cases. __init_subclass__, added in 3.6, hooks subclass creation without a metaclass and is far more discoverable. Class decorators transform a class after definition and are explicit at the point of use. Both should be preferred.

An ABC, from the abc module, defines methods that subclasses must implement. Instantiating a subclass that has not implemented an abstractmethod raises TypeError. It is nominal typing: a class must inherit from the ABC, or be explicitly registered with it, to be considered a subtype. A Protocol, from typing, is structural. Any class with the right methods satisfies it, with no inheritance and no registration. It is duck typing expressed in the type system, and it is checked statically rather than at runtime. The practical difference: use an ABC when you own the hierarchy and want to enforce implementation at runtime, and when a shared base implementation is useful. Use a Protocol when you want to accept anything with the right shape — including third-party classes you cannot modify. Protocol is usually the better fit for function parameters, because it does not demand that callers inherit from your type. runtime_checkable makes isinstance work with a Protocol, but only checks method presence, not signatures, so it is weaker than it looks. The collections.abc module provides ABCs for the container protocols, which are worth inheriting from for correct behaviour.

A single leading underscore means "internal, do not rely on this". It is purely convention — nothing prevents access — but it signals that the name is not part of the public interface and may change. A double leading underscore triggers name mangling: __x inside class C becomes _C__x. The purpose is not privacy but avoiding accidental collision in subclasses, particularly for mixins where two classes might independently use the same attribute name. It is frequently misused as "more private", which produces awkward access and confusing errors without adding real protection. A trailing underscore avoids clashing with a keyword — class_ or id_. Double leading and trailing underscores are reserved for the language's own protocols; you should use the existing ones rather than inventing new ones. A bare underscore is the convention for a value you are deliberately ignoring, as in unpacking, and it is also the last-result variable in the REPL. The practical guidance: single underscore for internal, double only when you genuinely need mangling for subclass safety, and rely on the convention rather than trying to enforce privacy — Python deliberately trusts the caller.

Define __eq__ comparing the attributes that determine identity, returning NotImplemented for unsupported types rather than False, so Python can try the reflected operation. Then define __hash__ consistently: objects that compare equal must have equal hashes. Hash the same tuple of attributes that __eq__ compares. Python 3 helps by setting __hash__ to None when you define __eq__ without __hash__, making the class unhashable rather than silently inconsistent. That is a deliberate safety measure and catches the mistake at use rather than at debugging time. The crucial requirement is that the hashed attributes must not change while the object is in a set or used as a dict key. Mutating them strands the entry in the wrong bucket, and the object becomes unfindable in a container it is demonstrably in. So hashable objects should be immutable, at least in the fields that participate. The easy path is a frozen dataclass, which generates both correctly and prevents mutation. For a mutable entity, define __eq__ on an identifier and either omit hashing or hash that identifier — never all fields.

The general argument is the same as anywhere — inheritance couples you to the parent's implementation, composition does not — but Python has some specifics. Python's duck typing weakens one of inheritance's usual justifications. You do not need a common base class for polymorphism; any object with the right methods works. So inheriting purely to satisfy a type is unnecessary. Multiple inheritance makes mixins practical, which is a legitimate use of inheritance for orthogonal behaviour — a LoggingMixin adding logging to any class. The caution is that mixins depend on the MRO and on cooperative super() calls, and deep mixin stacks become hard to reason about. Delegation is easy in Python via __getattr__, which forwards unknown attribute access to a wrapped object — that gives composition much of inheritance's convenience without the coupling, though it can obscure what is available. The practical position: prefer composition, use inheritance for genuine is-a relationships and for narrow mixins, and prefer Protocols over base classes when you only need a shared interface. Reaching for a base class to share a couple of helper methods is usually better served by a module-level function.

super() returns a proxy that dispatches attribute lookup to the next class in the MRO of the instance's actual type — not to the statically declared parent. That distinction matters. In a diamond hierarchy, super() in a class does not necessarily call its own declared parent; it calls whatever comes next in the linearisation for the object being constructed. That is what makes cooperative multiple inheritance work and why a shared base runs once rather than twice. The zero-argument form, super(), works in Python 3 because the compiler supplies the class and instance implicitly. It only works inside a class body; outside one you need the explicit two-argument form. For it to work correctly, every class in the chain must call super() and the signatures must be compatible — a class that omits it silently truncates the chain, so later classes never run their initialisation. That is the fragility of cooperative inheritance, and it is why **kwargs is often used in mixin __init__ methods to pass through arguments they do not understand. Calling the parent class explicitly by name works for single inheritance and breaks diamonds, which is why super() is preferred.

An iterable defines __iter__ returning an iterator. An iterator defines __next__ returning the next value and raising StopIteration when exhausted, and also defines __iter__ returning itself. So every iterator is an iterable, but not every iterable is an iterator. A list is iterable and not an iterator — calling iter() on it produces a fresh iterator each time, which is why you can loop over a list repeatedly. An iterator is consumed. Once exhausted it stays exhausted, which is why a generator cannot be iterated twice and why passing one to a function that loops over it more than once silently gets nothing the second time. The practical implementation: for a container, define __iter__ as a generator function that yields the elements. That gives a fresh iterator per call with almost no code. If you need the object itself to be an iterator — a stream, a cursor — define both __iter__ returning self and __next__. The simplest way to make a class iterable is often to define __getitem__ with integer indices, which Python falls back to, though __iter__ is clearer.

__repr__ produces an unambiguous representation aimed at developers. It is what the REPL shows, what appears in a list when you print it, what shows in a debugger, and what goes into log messages of collections. The convention is that it should look like valid Python that would recreate the object — ClassName(field=value, other=value) — and where that is impractical, an angle-bracketed description with the identifying details. __str__ is the human-readable form used by print and str(). If it is not defined, str falls back to repr, which is why defining repr alone is often sufficient. The practical argument for always defining it is debugging. The default repr shows only the class name and a memory address, so a list of your objects in a log or a debugger tells you nothing — and that is precisely the moment you need to know what they contain. Include the fields that identify the object, not every field, and be careful not to include secrets, since repr output ends up in logs and tracebacks. Dataclasses generate a reasonable one automatically, which is one of their better small benefits.

A function containing yield is a generator function. Calling it does not run the body — it returns a generator object. Each call to next() runs until the next yield, produces that value, and suspends, preserving all local state. The suspension is the key idea: the function's frame is kept alive between calls, so locals, the instruction pointer and the call stack position survive. The benefits are memory and composability. A generator over ten million records holds one at a time rather than materialising a list. And generators chain: one can consume another, forming a pipeline where data flows through without intermediate collections. They also allow infinite sequences, since nothing is computed until requested. The things to know. A generator is an iterator, so it is exhausted after one pass — iterating twice silently yields nothing the second time. It has no length and no indexing. And an exception raised inside propagates out of the next() call, which can be confusing because the traceback points into the generator body from an apparently unrelated loop. return inside a generator raises StopIteration with the value attached.

yield from delegates to another iterable, yielding all its values as if they had been yielded directly. The simple use is flattening: instead of looping over a sub-generator and yielding each item, you write one line. That is clearer and slightly faster. The more important part is that it establishes a transparent two-way channel. Values sent into the outer generator with send() are forwarded to the inner one, exceptions thrown in are propagated, and the inner generator's return value becomes the result of the yield from expression. That full delegation is what made generator-based coroutines practical before native async, and it is why the old asyncio style used yield from before await existed. await is essentially the same mechanism with clearer syntax and stricter rules. The practical uses today: recursive generators, such as walking a tree where each node yields from its children; and composing generator pipelines where a stage delegates part of its work. Without it, recursion in generators requires an explicit loop at every level, and the send and throw plumbing has to be written by hand.

Iterate over the file object directly. Python file objects are iterators over lines, reading in buffered chunks rather than loading everything, so for line in f: is already lazy. The mistake is f.readlines() or f.read(), which materialise the whole file. On a multi-gigabyte log that is an immediate memory problem. For processing pipelines, chain generators: a generator that parses each line, feeding one that filters, feeding one that transforms. Nothing accumulates, and the memory footprint is one record regardless of file size. For binary or fixed-size chunks, read in blocks with a loop, or use iter with a sentinel to make it an iterator. For CSV, csv.reader is already a lazy iterator. For JSON, the standard library is not streaming — json.load reads everything — so large JSON needs ijson or newline-delimited JSON, which is why NDJSON is the common format for large exports. Use mmap when you need random access to a large file without reading it all. And always use a with block, so the file is closed even on exception, which matters more in long-running processes than in scripts.

itertools provides memory-efficient iterator building blocks, implemented in C. The ones that earn their place. chain concatenates iterables without building a combined list. islice takes a slice of any iterator, including an infinite one, which is how you take the first n of a generator. groupby groups consecutive equal elements — note consecutive, so the input usually needs sorting first, which is the mistake everyone makes once. zip_longest pads the shorter iterable instead of stopping early, which matters when truncation would be silent data loss. tee splits one iterator into several, though it buffers, so it is not free. product, permutations and combinations replace nested loops for combinatorics and read far better. count and cycle produce infinite sequences, useful with islice or zip. accumulate gives running totals. pairwise, added in 3.10, yields overlapping pairs — perfect for comparing consecutive elements, which is otherwise a fiddly index dance. The general value is that these express intent directly and avoid intermediate lists, and the recipes section of the docs is worth reading once.

A generator is one kind of iterator — the one produced by a generator function or a generator expression. An iterator is any object implementing __next__ and __iter__. You can write one as a class, maintaining state in attributes and raising StopIteration when done. The generator form is almost always preferable because the language handles the state machine for you. A class-based iterator over a tree requires an explicit stack; a generator uses recursion with yield from and is a few lines. Generators also support send, throw and close, which class-based iterators do not unless you implement them. The cases where a class is better: when you need the iterator to have other behaviour or attributes — a cursor that can report progress, or one that can be reset; when you need to be able to iterate the same object more than once, which means __iter__ should return a new iterator each time rather than self. That last distinction is the practical one. If your class defines __iter__ as a generator function, it is a reusable iterable. If it returns self, it is a single-use iterator.

send() resumes a generator while passing a value in, which becomes the result of the yield expression inside. So yield is bidirectional: it produces a value outward and can receive one back. A generator written as x = yield result both emits result and waits to receive x. The pattern this enables is a coroutine — a generator that consumes values pushed into it rather than producing them on demand. A running-average accumulator, or a sink at the end of a pipeline, can be written this way. The protocol detail is that a generator must be primed with next() or send(None) before the first meaningful send, because it has to run to the first yield to be able to receive anything. Forgetting that raises TypeError, and it is the standard first mistake. In practice, send is rarely used directly today. It was the foundation of generator-based coroutines before async and await, and native coroutines have taken over that role with clearer semantics. So the honest answer is that it matters mainly for understanding how asyncio works underneath, and occasionally for a push-based pipeline.

A comprehension is generally preferred in Python for readability. [f(x) for x in items if cond(x)] states the transformation and the filter in one readable line, whereas the equivalent map and filter composition reads inside out. map and filter return lazy iterators in Python 3, so they are memory-efficient — but so is a generator expression, which is equally lazy and more readable. Where map is genuinely competitive is when you already have a named function and no filtering: map(str.strip, lines) is clean and slightly faster than the comprehension, because it avoids building a Python-level frame per element. Where map is worse is when it forces a lambda, since map(lambda x: x * 2, items) is both slower and less readable than the comprehension. filter with None as the function removes falsy values, which is a compact idiom worth knowing. The practical guidance: comprehension by default, generator expression when the result is consumed once or is large, and map only with an existing function reference. And avoid deeply nested comprehensions — beyond two levels or with a conditional, a loop is clearer.

For dicts and sets, Python raises RuntimeError: dictionary changed size during iteration. That is a deliberate guard, because the internal iteration state becomes invalid. For lists, there is no error and the behaviour is silently wrong. Removing elements while iterating shifts subsequent items into positions already passed, so the loop skips elements. Removing every element in a list by iterating and removing typically leaves half of them, which is a classic bug that looks like a logic error rather than an iteration error. The fixes: iterate over a copy — for x in list(items) — and modify the original. Or build a new collection with a comprehension, which is usually cleaner and expresses the intent better. Or iterate in reverse if removing by index, since the shift then affects only already-visited positions. For dicts, iterate over list(d.keys()) if you need to delete during iteration. The broader guidance is that building a new collection is almost always clearer than mutating during iteration, and it avoids the whole category. A filtering comprehension replaces a remove-while-iterating loop and is obviously correct.

The standard approach is itertools.islice in a loop: repeatedly take the next n items until the slice comes back empty. That works for any iterable, including generators and streams, and it does not require knowing the length or supporting indexing — which slicing a list would. Python 3.12 added itertools.batched, which does exactly this and is the answer to reach for on modern versions. The naive alternative of slicing a list by index works only for sequences and requires the whole thing in memory. The implementation detail that catches people is the terminating condition: islice on an exhausted iterator returns an empty result rather than raising, so the loop must check for empty and break. The use cases are everywhere — batching database inserts, chunking API requests to respect a payload limit, processing a large file in groups. The related pattern is chunking with overlap, for sliding windows, which itertools does not provide directly; a deque with maxlen is the neat implementation, appending each item and yielding the deque once it is full.

A chain of list comprehensions materialises every intermediate stage. Reading a file, parsing, filtering and transforming with four list comprehensions holds four full copies at peak. The same chain with generator expressions holds one item per stage — the pipeline is a set of coroutines pulling from each other, so total memory is constant regardless of input size. That is the difference between a script that handles a hundred-megabyte file and one that does not. The trade-offs. Generators are single-pass, so if a later stage needs to iterate twice you must materialise. You cannot take len() or index into them. And debugging is harder, because nothing exists until it is consumed — printing an intermediate generator shows an object, not data. There is also a per-item overhead: pulling through several generator stages costs more per element than a single tight loop, so for small collections a list comprehension can be faster. The practical guidance is generators for anything large or streaming, lists for small collections and when you need random access, and materialising deliberately with list() at the point where you genuinely need the whole thing.

The Global Interpreter Lock is a mutex ensuring only one thread executes Python bytecode at a time in CPython. It prevents true parallel execution of Python code across threads. So a CPU-bound workload gets no speedup from threading — and often gets slower, because of lock contention and context switching. What it does not prevent is concurrency for I/O. The GIL is released during blocking I/O — file reads, network calls, database queries — so threads waiting on I/O do not block each other. That is why threading works well for I/O-bound work. It is also released by some C extensions, which is why NumPy operations on large arrays can use multiple cores despite the GIL. The reasons it exists: it makes CPython's memory management simple and fast for single-threaded code, and it makes writing C extensions much easier. Removing it has been attempted repeatedly and historically made single-threaded performance worse. PEP 703 adds an optional free-threaded build in 3.13, which is the most serious attempt yet, though it is experimental and carries a single-threaded performance cost. The practical rule: threads for I/O, processes for CPU.

Threading for I/O-bound work where the libraries are blocking. The GIL is released during I/O, so threads overlap waiting. Good for a moderate number of concurrent operations — hundreds, not tens of thousands, because each thread carries real stack memory. Multiprocessing for CPU-bound work. Each process has its own interpreter and its own GIL, so they genuinely run in parallel. The cost is process startup and that data must be pickled to cross the boundary, which makes it unsuitable for small tasks or large shared state. Asyncio for I/O-bound work at high concurrency, where you can use async libraries throughout. A single thread handles thousands of concurrent operations because each is a lightweight coroutine rather than an OS thread. That is why it suits network servers and clients making many concurrent calls. The decision points: is the work CPU or I/O bound, and are async-compatible libraries available? A blocking database driver in an async application blocks the whole event loop, which defeats the purpose — so asyncio is an all-or-nothing commitment in a way threading is not. concurrent.futures gives one interface over threads and processes.

A single-threaded event loop runs coroutines cooperatively. A coroutine runs until it awaits something, at which point it yields control back to the loop, which runs another ready coroutine. When the awaited operation completes, the loop resumes the original. The word cooperative is the crucial part. Nothing preempts a coroutine — it must yield voluntarily by awaiting. So a coroutine performing a long CPU computation, or calling a blocking library, freezes the entire loop and every other task with it. That is the most common asyncio bug: mixing a synchronous database driver or requests into async code, which serialises everything while looking concurrent. Under the hood, async def creates a coroutine function whose call returns a coroutine object; await drives it. The loop uses the OS multiplexing primitive — epoll, kqueue — to know which file descriptors are ready, which is what lets one thread manage thousands of sockets. asyncio.to_thread or run_in_executor offloads blocking work to a thread pool, which is the escape hatch when a library has no async version. And awaiting is not concurrency by itself — sequential awaits run sequentially.

Awaiting one coroutine after another runs them sequentially. Each await suspends until that operation completes before the next begins, so three one-second calls take three seconds. asyncio.gather schedules them concurrently and waits for all, so the same three calls take about one second. That distinction is the single most common asyncio misunderstanding — code that is syntactically async but semantically sequential, gaining nothing. gather returns results in the order the coroutines were passed, regardless of completion order. The error handling detail matters: by default, the first exception propagates immediately and the other tasks keep running unawaited, which can produce warnings about never-retrieved exceptions. return_exceptions=True collects exceptions as results instead, letting you handle them individually. asyncio.TaskGroup, added in 3.11, is the modern preferred form. It cancels remaining tasks when one fails and ensures nothing outlives the block, which is structured concurrency and avoids the orphaned-task problem gather has. asyncio.as_completed yields results as they finish rather than in order, which is right when you want to start processing the fastest responses immediately.

It blocks the entire event loop. Every other coroutine stops making progress until it returns, because there is only one thread and nothing can preempt it. So a synchronous HTTP call, a blocking database query, a file read, or a CPU-heavy computation inside a coroutine converts your concurrent server into a sequential one — while every appearance of the code suggests otherwise. The symptoms are confusing: throughput far below expectation, latency that scales with concurrent requests, and health checks timing out under load, with no obvious blocking in the code because it is inside a library. The fixes. Use an async library — aiohttp or httpx instead of requests, asyncpg or the async SQLAlchemy driver instead of a sync one. Where no async version exists, offload with asyncio.to_thread, which runs it in a thread pool and awaits the result. For CPU-bound work, use a process pool via run_in_executor, since a thread would still hold the GIL. The diagnostic tool is the loop's debug mode, which logs callbacks taking longer than a threshold — that is what finds the blocking call quickly.

A coroutine is what an async def call returns. It is inert — creating one runs no code. Awaiting it runs it in the current context. A Task wraps a coroutine and schedules it on the event loop to run concurrently. asyncio.create_task returns one, and the coroutine begins making progress at the next suspension point without you awaiting it. A Future is a lower-level placeholder for a result that will exist eventually. Task is a subclass of Future. The practical distinction is that awaiting a coroutine directly is sequential, while creating a task starts it concurrently and you await it later — or not at all, though not awaiting means exceptions go unnoticed. The pitfall is that a task holds only a weak reference from the loop, so a task whose reference you discard can be garbage collected mid-execution. The documented fix is to keep a reference in a set until it completes, which surprises people and is a genuine source of vanishing work. TaskGroup handles that correctly and is the better modern approach. Futures are mostly what you meet when bridging to callback-based code.

asyncio.Semaphore. Acquire it before the operation and release after, which caps how many coroutines are in that section simultaneously. The reason you need it is that gather over ten thousand coroutines starts all of them, which can exhaust file descriptors, overwhelm the target service, or run out of memory. Unbounded concurrency is not a feature. The idiom is a semaphore with the limit, and each worker using async with sem: around the actual call, then gathering all of them. They are all created but only the permitted number proceed. The alternatives: a queue with a fixed number of worker tasks consuming from it, which gives more control and natural back-pressure. Or a library like aiometer that provides the pattern directly. For HTTP specifically, the client usually has its own connection pool limit, which provides some bounding — but it queues rather than rejecting, so it does not prevent memory growth from ten thousand pending coroutines. The general principle mirrors thread pools: unbounded concurrency converts a throughput problem into a resource exhaustion problem, and the limit should be chosen from what the downstream can absorb.

multiprocessing runs work in separate OS processes, each with its own interpreter and GIL, giving genuine parallelism for CPU-bound work. The costs are significant and often underestimated. Process startup is expensive — on Windows and macOS the default is spawn, which imports the module fresh in each child, so module-level work runs again per process. On Linux fork is cheaper but copies the address space lazily and interacts badly with threads. Data must be pickled to cross the boundary, so arguments and results are serialised and copied. Passing a large DataFrame to workers can cost more than the computation. And anything unpicklable — a lambda, an open file, a database connection — cannot be passed at all. Memory is multiplied, since each process has its own copy of the interpreter and data. The practical guidance: use a Pool so processes are reused rather than created per task; make tasks coarse enough that the overhead is amortised; and avoid sharing state — use the return value rather than shared memory where possible. For numeric work, vectorising with NumPy often beats multiprocessing entirely.

It provides a uniform interface over thread and process pools: ThreadPoolExecutor and ProcessPoolExecutor share the same API, so switching between them is a one-word change. submit schedules a callable and returns a Future; map applies a function over an iterable. as_completed yields futures as they finish. The value is that it removes most of the boilerplate of managing threads or processes directly — no manual thread creation, no join loops, no queue plumbing — and the executor as a context manager guarantees shutdown. It is the right abstraction for embarrassingly parallel work: apply this function to these inputs. It is not the right abstraction when tasks need to coordinate, communicate, or have complex lifecycles. The detail worth knowing is exception handling: an exception in a worker is stored on the Future and re-raised when you call result(). If you never call result — for instance using submit and never checking — the exception is silently swallowed. That is a real source of invisible failures, and it is why as_completed with result() inside is the safer pattern. map re-raises on iteration, which is more forgiving.

The GIL makes individual bytecode operations atomic, which means some operations are safe by accident — appending to a list, or setting a dict key, will not corrupt the structure. But compound operations are not safe. counter += 1 is a read, an add and a write, and the GIL can be released between them, so increments are lost. The same applies to check-then-act patterns: testing whether a key exists and then setting it is a race. So the tools are the usual ones. threading.Lock for mutual exclusion, and using it as a context manager so it is always released. RLock when the same thread may acquire it twice. Event, Condition and Semaphore for coordination. queue.Queue is the best answer for most producer-consumer work, because it handles the locking internally and gives a clean hand-off rather than shared mutable state. threading.local gives per-thread storage, avoiding sharing altogether. The strongest advice is to avoid shared mutable state. Pass data through queues, or have each thread own its data and combine results at the end. Relying on GIL-atomicity for correctness is fragile and stops being true on free-threaded builds.

It is a queue designed for coroutines — put and get are awaitable, so a coroutine waiting on an empty queue yields to the event loop rather than blocking the thread. That makes it the right structure for producer-consumer patterns in async code. queue.Queue would block the loop entirely, since its get is synchronous. The standard pattern is a set of worker tasks looping on await queue.get(), processing, and calling task_done, with producers putting items in. await queue.join() waits until everything is processed. A maxsize gives back-pressure: producers awaiting put on a full queue naturally slow to the consumers' rate, which is what prevents unbounded memory growth when production outpaces consumption. The shutdown detail is the fiddly part. Workers looping forever never exit, so the convention is to put a sentinel value per worker, or to cancel the tasks after join completes. Forgetting leaves tasks pending at shutdown and produces warnings. It is also the cleanest way to bound concurrency with a fixed number of workers, as an alternative to a semaphore over unbounded tasks.

asyncio.timeout, added in 3.11, is the modern form: an async context manager that cancels everything inside it if the deadline passes, raising TimeoutError. asyncio.wait_for wraps a single awaitable and is the older equivalent. Cancellation works by raising CancelledError inside the coroutine at its next suspension point. That is important: cancellation is cooperative, so a coroutine stuck in a blocking call or a tight CPU loop cannot be cancelled, because it never yields. The handling rule is that CancelledError should generally not be swallowed. Catching it to do cleanup and then re-raising is correct; catching and continuing breaks cancellation semantics and leaves the task running when the caller believes it stopped. In Python 3.8+ it inherits from BaseException rather than Exception specifically so that except Exception does not catch it by accident. Use try/finally, or a context manager, for cleanup that must run on cancellation. asyncio.shield protects an inner operation from cancellation, which is occasionally right for a write that must complete — but it is easy to misuse and leave work running after the caller has gone.

A threaded server allocates a thread per request. Threads are preemptive, so a slow handler does not stop others, and blocking libraries work unchanged. The limit is memory and context switching — each thread has a stack measured in megabytes, so thousands of concurrent connections is expensive. An async server handles all requests on one thread with coroutines. Each is cheap, so tens of thousands of concurrent connections are feasible, which is why async suits long-lived connections and high-concurrency I/O. The cost is the all-or-nothing requirement: every library in the request path must be async, or one blocking call stalls every request on that worker. For a typical CRUD service where each request does a couple of database queries and returns, threading is often perfectly adequate and much simpler to reason about — and Python's dominant deployment model of several processes each with a thread pool works well. Async wins clearly for many concurrent outbound calls, WebSockets, streaming, and proxy-like workloads. In practice the deployment is both: multiple worker processes for CPU parallelism, each running an async loop or a thread pool.

Calling a coroutine without awaiting it. It creates a coroutine object and never runs, producing a RuntimeWarning that is easy to miss — the code silently does nothing. Awaiting sequentially when you meant concurrently, so async code runs no faster than synchronous code. Using blocking libraries inside coroutines, which stalls the whole loop. Creating tasks and discarding the reference, letting them be garbage collected mid-flight. Swallowing CancelledError, breaking cancellation. Unbounded gather over a huge list, exhausting file descriptors or memory. Running CPU-bound work in a coroutine rather than offloading it. Mixing event loops — calling asyncio.run inside a running loop raises, and creating a second loop causes confusing failures. Forgetting that async context managers and iterators need async with and async for. And not closing resources on cancellation, so sockets leak when a request times out. The common root is treating async as a keyword you sprinkle on rather than a different execution model. The debug mode of the loop, plus a linter that flags un-awaited coroutines, catches most of these early.

First establish that it is genuinely CPU-bound and profile to find where the time goes, because the intuition is often wrong. Then, in rough order of effort. Algorithmic improvement, which usually beats everything else. A dict lookup replacing a list scan changes the complexity class. Vectorise with NumPy if the work is numeric. Moving a loop into array operations pushes it into C and is frequently an order of magnitude or more, without any parallelism. Use multiprocessing to parallelise across cores, accepting the pickling and startup costs. concurrent.futures.ProcessPoolExecutor is the easy interface. Move the hot path to C via Cython, or compile with Numba, both of which target exactly this. Try PyPy, whose JIT can give large speedups for pure Python, though C extension compatibility is the constraint. And rewrite the hot function in Rust with PyO3, which is increasingly the chosen route for libraries. The framing worth giving is that Python is glue: the answer is usually to move the hot loop out of Python rather than to make Python faster.

PEP 703 introduces an optional build of CPython without the GIL, so threads can execute Python bytecode genuinely in parallel. That removes the main reason to reach for multiprocessing: CPU-bound threading becomes viable, without pickling overhead or separate address spaces. The costs and caveats are substantial, which is why it is opt-in and experimental. Single-threaded performance is worse, because reference counting must become thread-safe — biased reference counting and deferred counting mitigate it but do not eliminate the cost. C extensions must be updated to declare compatibility; those that relied on the GIL for implicit locking are unsafe, and much of the ecosystem is not yet ready. And code that was accidentally correct because of GIL atomicity — relying on a dict update being uninterrupted — becomes racy. That is a real migration hazard, because such code has no visible locking to review. The practical position for now is that it matters for library authors and for people with genuinely parallel CPU workloads, and most applications should wait for the ecosystem. It is worth knowing about as a direction rather than as something to adopt.

collections for defaultdict, Counter, deque and namedtuple — the structures that remove the most hand-written code. itertools for lazy iterator composition. functools for lru_cache, wraps, partial and singledispatch. pathlib for filesystem paths, which replaces os.path string manipulation with an object model that is far harder to get wrong on different platforms. datetime with timezone handling, and zoneinfo since 3.9 for IANA timezones without a third-party dependency. dataclasses for structured data. enum for named constants, which beats module-level strings for type safety and readability. typing for hints, and contextlib for context manager helpers. logging, which is worth learning properly rather than using print — the hierarchy, handlers and formatters. json, csv, and re for the obvious. secrets rather than random for anything security-related, which is a genuinely important distinction. subprocess for running commands, with the shell=False default understood. And unittest.mock, which is used even in pytest projects. The general point is that a large fraction of code written in application projects duplicates something already in the standard library.

Store and compute in UTC, and convert to local time only at the presentation boundary. Use timezone-aware datetimes, not naive ones. A naive datetime has no timezone attached, so comparing or subtracting a naive and an aware one raises, and comparing two naive ones from different zones silently gives wrong answers. datetime.now(timezone.utc) is correct; datetime.utcnow() is not, because it returns a naive datetime that merely happens to hold UTC values — which is exactly the trap, and it is deprecated in 3.12 for that reason. Use zoneinfo for IANA timezones, which handles daylight saving transitions properly. Fixed UTC offsets do not, so storing "+05:30" loses the ability to compute correctly across a transition. For a future event tied to a wall-clock time in a place — a meeting next year — store the local time plus the timezone name, not a UTC instant, because if the timezone rules change the meeting must move with them. For durations, use timedelta rather than arithmetic on components. And parse with explicit formats or a strict parser rather than guessing, since ambiguous formats produce silently wrong dates.

Get a logger per module with logging.getLogger(__name__), which gives a hierarchy matching your package structure and lets you set levels per subsystem. Configure handlers and levels once, at application startup, not in library modules. A library should attach no handlers — that is the application's decision — and adding a NullHandler prevents the "no handler found" warning. Use the level appropriately: DEBUG for development detail, INFO for significant events, WARNING for recoverable problems, ERROR for failures. Levels are how consumers filter, and using them consistently is what makes that possible. Use lazy formatting — logger.info("processed %s", item) rather than an f-string — so the formatting cost is skipped when the level is disabled. With an f-string it is always paid. logger.exception inside an except block includes the traceback automatically, which is what you want and is frequently missed. For a service, structured logging as JSON is far more useful than free text, because it is queryable. And include a correlation or request ID in every line, propagated through the request, or you cannot reconstruct what happened.

os.path treats paths as strings and provides functions to manipulate them. pathlib provides a Path object with methods and operators. The practical differences. Joining with the / operator reads far better than nested os.path.join calls. Properties like .name, .stem, .suffix and .parent replace a family of functions. Methods like .exists(), .read_text() and .mkdir(parents=True) put the operation on the object. It handles platform differences by construction, so you are not manipulating separators by hand — which is where string-based path code breaks on Windows. It also makes intent clearer in type hints: a parameter annotated Path is unambiguous, while str could be anything. The compatibility story is good — Path implements os.PathLike, so it can be passed to anything accepting a path, including open() and most third-party libraries. Where a library insists on a string, str(path) converts. pathlib is the recommendation for new code. The remaining case for os.path is performance in a very hot loop, since Path objects have more overhead, and that rarely matters.

Read configuration from the environment, following the twelve-factor approach, so the same artefact runs in every environment with different settings. Do not commit secrets. A .env file for local development is fine if it is gitignored; the values in deployed environments should come from the platform's secret management. Validate configuration at startup rather than at first use. A missing or malformed setting should fail the process immediately with a clear message, not produce a confusing error deep in a request three hours later. Pydantic Settings does this well — declaring the settings as a typed model gives parsing, validation and defaults in one place. Type coercion matters: environment variables are strings, so a boolean read naively is always truthy — os.environ.get("DEBUG") returns "False" which is a non-empty string and therefore true. That is a classic bug. Keep a single settings object rather than reading os.environ throughout, so configuration is discoverable and testable. And never log the settings object wholesale, since it contains secrets — define a repr that redacts them.

random uses the Mersenne Twister, a fast pseudo-random generator designed for simulation and statistics. It is deterministic given its seed, and its internal state can be reconstructed from a modest number of outputs. secrets uses the operating system's cryptographically secure source, suitable for anything where predictability is a security problem. So tokens, password reset links, session identifiers, API keys, salts and one-time codes must use secrets. Using random for any of these is a real vulnerability — an attacker who observes some outputs can predict the rest. secrets provides token_hex, token_urlsafe, token_bytes, choice and compare_digest. compare_digest is worth knowing separately: it compares strings in constant time, so an attacker cannot infer how many leading characters were correct from the response timing. Comparing a token with == leaks that information, which is a genuine timing attack surface. random remains correct for sampling, shuffling test data, jitter in retry backoff, and simulation. The rule is simple: if being able to guess the value would be bad, use secrets.

subprocess.run with a list of arguments and shell=False, which is the default. Passing a list means the arguments are handed to the process directly, with no shell involved, so nothing in them can be interpreted as shell syntax. Passing a string with shell=True runs it through the shell, and any user-controlled content becomes a command injection. That is the whole security answer: never build a command string from untrusted input with shell=True. Beyond safety: use check=True so a non-zero exit raises rather than being silently ignored, which is the most common bug — a failing command that the script never notices. capture_output=True with text=True gives stdout and stderr as strings. Set a timeout, or a hung child hangs your process indefinitely. The pipe deadlock is worth knowing: if you use Popen with pipes and the child produces more output than the pipe buffer while you are writing to it, both block. communicate() handles this correctly, which is why it exists. And prefer a library over shelling out where one exists — calling curl instead of using an HTTP client is a common and avoidable dependency.

venv creates an isolated environment with its own interpreter and site-packages, so projects do not share dependencies. pip installs packages into it. requirements.txt records what to install, and pip freeze captures the resolved set — but pip historically had no real dependency resolver, and requirements.txt does not distinguish direct dependencies from transitive ones. Poetry adds a project file declaring direct dependencies with constraints, a lock file pinning the full resolved tree with hashes, and environment management. That gives reproducible installs, which requirements.txt alone does not. pip-tools is the lighter alternative, compiling a locked requirements file from a declared one. uv is the newer Rust-based tool doing installation, resolution, environment and Python version management, dramatically faster than pip — often by an order of magnitude — and largely drop-in compatible. The practical recommendation now: pyproject.toml as the declaration, a lock file committed, and uv or Poetry to manage it. Bare pip and requirements.txt still work and are common, but reproducibility is weaker. The underlying point is separating what you asked for from what got installed.

f-strings, for almost everything. They are evaluated inline, are the fastest option, and read best because the expression is where the value appears. The format specifiers are worth knowing: alignment, width, precision, thousands separators, and since 3.8 the = suffix which prints both the expression and its value — f"{count=}" — which is excellent for debugging. str.format remains useful when the template is not a literal: a message string loaded from configuration or a translation file cannot be an f-string, since f-strings are evaluated at their definition point. Percent formatting is legacy, with one important exception: logging calls should use the percent style with arguments passed separately, so formatting is deferred and skipped when the level is disabled. Template strings from the string module are the safe choice when the template comes from an untrusted source, since they support only simple substitution and cannot execute arbitrary expressions. The security note that matters: never build SQL with an f-string. Use parameterised queries. The same applies to shell commands and to any other context where injection is possible.

The standard json module is correct and adequate for most use, but it is pure Python for encoding and comparatively slow. The things to know about it: it cannot serialise datetime, Decimal, UUID or dataclass instances by default, so you supply a default function or a custom JSONEncoder. Sets and bytes are not serialisable at all. It loads the entire document into memory, so very large JSON needs a streaming parser such as ijson, or newline-delimited JSON where each line is an independent object — which is why NDJSON is the common format for large exports. orjson and ujson are faster alternatives; orjson is the usual recommendation, handling datetime and dataclasses natively and being substantially quicker on both encode and decode. For API work, Pydantic handles serialisation and validation together, which is usually what you actually want — the standard library validates nothing, so json.loads on untrusted input gives you an arbitrary structure that you must then check by hand. And float precision is worth remembering: JSON numbers are floats, so monetary values should be strings or integer minor units rather than floats.

Less ceremony and better failure output. A pytest test is a plain function with a plain assert. unittest requires a class inheriting TestCase and specific assertion methods — assertEqual, assertIn — which is more to write and to remember. pytest rewrites assert statements so a failure shows the actual values on both sides, including the diff for collections. That introspection is the single biggest practical difference: a failing assert tells you what the values were, rather than just that they differed. Fixtures are more flexible than setUp and tearDown: they are composable, have explicit scopes, and are requested by parameter name so a test declares what it needs. Parametrisation is built in, so one test function covers many cases with clear per-case reporting. The plugin ecosystem is large — coverage, mocking, async support, database fixtures. And pytest runs unittest tests, so migration is incremental. The case for unittest is that it is in the standard library, which matters when adding a dependency is genuinely constrained. That is the only real argument, and it is rarely decisive.

A fixture is a function decorated with @pytest.fixture that provides a resource to tests. A test requests it by naming it as a parameter, and pytest supplies it. Fixtures compose: one can request another, so setup is built up in layers rather than duplicated. Using yield inside a fixture splits setup from teardown, with the teardown running after the test even on failure — the context manager pattern applied to test resources. The scopes control lifetime and reuse. function, the default, creates a fresh instance per test, which gives the best isolation. class, module and package widen it. session creates one for the entire run. The trade-off is speed against isolation. A database container is expensive to start, so session scope is right — but then tests share state and must clean up after themselves or become order-dependent, which is the classic flaky-test cause. The usual compromise is a session-scoped container with a function-scoped transaction that rolls back, giving fast setup and per-test isolation. conftest.py makes fixtures available to a whole directory without importing.

unittest.mock provides Mock and MagicMock, objects that record calls and return configurable values, plus patch, which temporarily replaces an attribute for the duration of a test. The rule that catches everyone is where to patch. You patch where the name is looked up, not where it is defined. If a module does from x import fetch, you patch mymodule.fetch, not x.fetch — patching the origin has no effect because the importing module already holds its own reference. That single rule accounts for most "the mock is not being used" confusion. What to mock: external services, the network, the clock, randomness, and anything slow or non-deterministic. What not to mock: your own domain objects and internal collaborators, because that couples the test to the current structure and it breaks on refactoring while passing when behaviour is wrong. Use autospec or spec, so calling the mock with a signature the real object does not have raises rather than silently succeeding. Without it, a mock accepts anything, and a test can pass against a function whose signature has changed. Prefer dependency injection over patching where you can.

Do not call datetime.now() directly in the code under test — inject the time source. The simplest form is a parameter with a default: a function taking now=None and computing it if absent, so tests can pass a fixed value. Slightly cleaner is passing a clock callable, or a small Clock object, so the dependency is explicit. That makes testing expiry, scheduling, date boundaries and timezone transitions trivial and deterministic. The alternative is patching. freezegun is the popular library, patching datetime globally within a context so all time queries return a fixed value. time-machine is a faster equivalent. Both work well and require no production code change, which is their appeal for existing codebases. The argument against patching is that it hides the dependency — the function still has an invisible coupling to the clock — and global patching can interact badly with libraries that cache time. The same reasoning applies to randomness, UUID generation, and any other ambient source of variability: inject it, or the code has an untestable dependency that does not appear in its signature.

@pytest.mark.parametrize runs the same test function with different inputs, each reported as a separate test case. The value is coverage without duplication. Testing a validator against ten inputs is one function and a list, rather than ten near-identical functions or a loop. The advantage over a loop inside one test is reporting: each case is a separate result, so a failure names the specific input rather than just the test, and a failure does not stop the remaining cases from running. Ids can be supplied so failures are readable rather than showing raw values. Stacking two parametrize decorators produces the cross product, which is a compact way to test a matrix of combinations — and a fast way to accidentally generate hundreds of tests, so it is worth being deliberate. Fixtures can be parametrised too, which runs every test using that fixture against each variant — the standard way to run a suite against several database backends. The related tool is Hypothesis for property-based testing, which generates inputs rather than enumerating them and shrinks failures to a minimal case. It finds edge cases that hand-written parameters miss.

With pytest-asyncio, which lets a test be an async function and runs it on an event loop. Mark it with @pytest.mark.asyncio, or configure asyncio_mode to auto so every async test is collected without the marker. anyio is the alternative and supports trio as well as asyncio. The practical points. Fixtures that need to be async — a database connection, an HTTP client — must themselves be async fixtures, and mixing sync and async fixtures incorrectly is a common source of confusing errors. Mocking needs AsyncMock rather than Mock, since awaiting a regular Mock fails. unittest.mock.patch detects async functions and uses AsyncMock automatically in recent versions, but being explicit avoids surprises. Testing timeouts and cancellation is worth doing deliberately, since those paths are where async bugs concentrate — and they are almost never covered. The event loop scope matters: by default each test gets a fresh loop, so a session-scoped async fixture holding a connection bound to one loop breaks. Aligning the fixture and loop scopes is the fix, and getting it wrong produces "attached to a different loop" errors that are hard to interpret.

Usually I/O in unit tests — a real database, real HTTP calls, real filesystem — where a fake would do. The first fix is separating fast unit tests from slower integration tests, with markers so developers can run the fast set constantly and the full set in CI. A suite that takes ten minutes stops being run. Fixture scope is the next lever. Creating a database container or an application instance per test is enormously wasteful; session scope with per-test transaction rollback gives isolation at a fraction of the cost. Run in parallel with pytest-xdist, which distributes across cores. That requires tests to be independent, which they should be anyway — parallelism exposes hidden shared state, which is a useful side effect. Profile the suite with --durations to find the worst offenders, since the time is usually concentrated in a handful of tests. Avoid sleeps. A test that waits a second for something to settle is both slow and flaky; poll for the condition with a timeout instead. And check for expensive module-level work at import, which is paid on every run.

A test that passes sometimes and fails sometimes without the code changing. The usual causes: shared state between tests, so the result depends on order; real time — a test that fails at midnight or on a leap day; timing assumptions, waiting a fixed duration for an async operation; unmanaged randomness; network dependencies; and concurrency in the code under test. The correct response is to find and fix the cause, because a flaky test is often reporting a real race in the code rather than a test problem. The diagnostic tools: run the test repeatedly, run the suite in random order with pytest-randomly to expose order dependence, and run in parallel to expose shared state. What not to do is add a retry and move on. Retrying hides the signal, and if the flakiness reflects a genuine race the bug ships. If the cause genuinely cannot be fixed quickly, quarantine the test — mark it so it runs but does not fail the build, with a ticket — rather than leaving it failing intermittently, because a suite that is sometimes red trains everyone to ignore red. That erosion of trust is the real cost.

Measure before optimising, because intuition about where time goes is usually wrong. cProfile is the standard deterministic profiler, giving call counts and cumulative time per function. Its output is best read through snakeviz or a similar visualiser, since the raw table is hard to interpret. The caveat is overhead: it instruments every call, so it distorts timing and is unsuitable for production. For production, a sampling profiler such as py-spy is the right tool. It attaches to a running process without modifying it, has negligible overhead, and can produce flame graphs — which is how you find a hot path in a live service. line_profiler gives per-line timing within a function, which is what you need once you have narrowed to one function. memory_profiler and tracemalloc handle memory rather than time, and tracemalloc is in the standard library and good for finding leaks by comparing snapshots. timeit is for microbenchmarks of small snippets, and it handles the setup and repetition correctly. The method matters more than the tool: profile a realistic workload, find the top few entries, and fix those rather than optimising everything.

Membership testing against a list inside a loop, which is O(n) per check and quadratic overall. Converting to a set is usually the single biggest available win. String concatenation in a loop with +=, which creates a new string each time. Use join over a list, or io.StringIO. Repeated attribute lookups in a tight loop, since each one is a dictionary lookup; hoisting to a local is a real gain in hot code. Calling a function that recomputes the same result — the case for lru_cache. Using list.pop(0) or insert(0, x) in a queue pattern, which shifts every element; deque fixes it. Building intermediate lists in a pipeline where generators would stream. Unnecessary deepcopy. N+1 queries in ORM code, which is a database problem expressed in Python. And doing element-wise numeric work in a Python loop rather than vectorising with NumPy, which is often an order of magnitude. The pattern is that most of these are algorithmic or structural rather than micro-optimisations, which is why profiling first matters — the wins are large and specific, not spread thinly.

Python does not leak in the C sense — the collector reclaims unreachable objects — so a leak means objects are still reachable when they should not be. The usual causes: a module-level cache or list that only grows; an lru_cache on a method, keeping every instance alive through self; registering callbacks or observers and never unregistering; and reference cycles involving objects with finalisers. The diagnostic approach is to take tracemalloc snapshots at intervals and compare, which shows which allocation sites are growing. gc.get_objects with a Counter by type shows which types are accumulating, which usually identifies the structure immediately. objgraph can render reference chains showing what is keeping an object alive, which answers the actual question. The fixes: bound caches with a maxsize, use weakref for back-references and observer registries so they do not keep targets alive, and unregister explicitly in teardown or a context manager. The non-leak explanation worth checking first is that CPython does not always return freed memory to the OS — arenas are reused — so flat-then-high RSS after a large workload may be fragmentation rather than a leak.

When you are doing element-wise numeric work over more than a few thousand items. A Python list of numbers stores pointers to boxed integer or float objects scattered across the heap. A NumPy array stores raw values contiguously, so it uses a fraction of the memory and has vastly better cache behaviour. The bigger win is vectorisation: an operation over an array runs as a single C loop rather than as a Python loop with per-element interpreter overhead. That is routinely one to two orders of magnitude. So the guidance is to express the computation as array operations rather than looping. If you find yourself writing a for loop over a NumPy array, there is usually a vectorised form. Broadcasting lets operations on differently-shaped arrays work without explicit loops, which is powerful and worth learning properly. The costs: a dependency, a different mental model, and arrays are homogeneous and fixed-type so they do not replace lists generally. Small arrays also carry construction overhead that can make NumPy slower than a list for a handful of elements. And it releases the GIL for large operations, so it parallelises where pure Python cannot.

Mutable default arguments, which are almost always a bug. Bare except clauses and except Exception used where a specific type belongs, plus any silently swallowed exception. Membership tests against lists inside loops. String concatenation in loops. f-strings in logging calls, which defeat lazy formatting. f-strings building SQL or shell commands, which is an injection. random used where secrets is required. Missing context managers on files, connections and locks. lru_cache on methods, which leaks instances. Generators consumed twice. Modifying a collection while iterating it. Naive datetimes, and utcnow specifically. Missing type hints on public functions, and hints that are too concrete on parameters. Module-level side effects that run on import. Broad mocking that couples tests to structure. And from module import * anywhere. Most of these are catchable automatically. Ruff covers a large fraction of the list, mypy covers the typing, and Bandit covers the security items — so the review conversation should be about design, with the mechanical checks in CI.

Using the language's own constructs rather than translating from another language. Concretely: iterating over a collection directly rather than over range(len(x)). Using enumerate when you need the index and zip to walk two sequences together. Comprehensions rather than building a list with append in a loop. Unpacking rather than indexing into a tuple. Context managers for anything with cleanup, rather than try/finally by hand. EAFP — try the operation and handle the exception — rather than checking first, which avoids a race and is usually faster. Truthiness for emptiness rather than comparing lengths, while remembering that it conflates empty with None. Properties instead of getter and setter methods. Generators for streaming rather than building lists. The standard library instead of reimplementing Counter, defaultdict or groupby. And naming and structure that follow PEP 8, since consistency is a large part of readability. The underlying point is that idiomatic code is not stylistic preference — it is usually shorter, faster, and less likely to contain the specific bugs the idiom exists to avoid. Code that reads as translated Java tends to carry Java's workarounds for problems Python does not have.