Generators — yield & Lazy Pipelines
IntermediateA function with yield returns a generator that produces values on demand, pausing between them — constant-memory processing and composable data pipelines.
Overview
yield transforms a function: calling it runs NO code, it returns a generator object; each next() runs until the next yield and FREEZES there — locals intact. This lazy evaluation means a generator can represent a billion items, an infinite stream, or a huge file in a few bytes of memory. Chain generators and you get a pipeline where one item flows end-to-end at a time — the architecture of every ETL script and the mental model behind async/await later.
Pause & Resume
Watch execution order: nothing prints until next() is called; the function sleeps between yields. return (or falling off the end) raises StopIteration.
def demo():
print("A: started")
yield 1
print("B: resumed")
yield 2
print("C: finishing")
g = demo() # NOTHING prints — no code ran
print(next(g)) # A: started -> 1
print(next(g)) # B: resumed -> 2
# next(g) # C: finishing -> StopIteration
# Infinite generator — impossible as a list
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
from itertools import islice
print(list(islice(fibonacci(), 8))) # [0,1,1,2,3,5,8,13]
# yield from — delegate to a sub-generator
def flatten(nested):
for item in nested:
if isinstance(item, list):
yield from flatten(item) # recursion, lazily
else:
yield item
print(list(flatten([1, [2, [3, 4]], 5]))) # [1,2,3,4,5]Pipelines — One Item at a Time, End to End
Each stage consumes the previous one lazily. Memory stays constant no matter the input size — this exact shape processes logs, CSVs, and API pages in production.
def read_lines(path):
with open(path, encoding="utf-8") as f:
for line in f:
yield line.strip()
def parse(lines):
for line in lines:
if line and not line.startswith("#"):
name, marks = line.split(",")
yield name, int(marks)
def only_passed(rows, cutoff=40):
for name, marks in rows:
if marks >= cutoff:
yield name, marks
# Compose: nothing executes until iteration begins
pipeline = only_passed(parse(read_lines("marks.txt")))
for name, marks in pipeline: # one line flows through all stages
print(name, marks)
# The generator-expression shorthand for simple stages:
# passed = ((n, m) for n, m in rows if m >= 40)Key Points to Remember
- 1Calling a generator function runs nothing — iteration does
- 2State (locals + position) persists between next() calls; generators are one-shot
- 3yield from delegates to sub-generators (recursive flattening, refactoring)
- 4Pipelines of generators process unlimited data in constant memory
Interview Questions
Sign in to ask AriaWhat happens when you call a function containing yield? When does its code run?
Generator vs list for processing a huge file — memory and speed trade-offs?
What does yield from do? Show recursive flattening.
Ask Aria about Generators — yield & Lazy Pipelines
Your personal AI tutor — ask anything about this concept
Revision Status
Personal Notes
Sign in to save personal notes for this topic.
Discussion
Sign in to join the discussion.