Home/Learn/FastAPI/Streaming & SSE — Pushing Data as It Happens

Streaming & SSE — Pushing Data as It Happens

Advanced

Background & WebSockets

StreamingResponse sends bytes as a generator yields them — huge CSV exports without RAM spikes, and Server-Sent Events (text/event-stream) for one-way push: the protocol behind every LLM token stream.

Overview

Sometimes the response should not wait to be complete. StreamingResponse takes a (async) generator and flushes each yielded chunk to the client immediately — a million-row CSV export streams with constant memory instead of building a giant string, and a slow pipeline shows progress instead of a spinner. Server-Sent Events is the standard framing for one-way push over plain HTTP: content type text/event-stream, each message as "data: ...\n\n", consumed in the browser by EventSource with automatic reconnection built in. SSE is exactly how ChatGPT-style token streaming works — including Aria's chat — and the interview comparison is crisp: SSE is one-way, plain HTTP, proxy-friendly, auto-reconnecting; WebSockets are two-way but heavier machinery. Client needs to talk back continuously → WebSocket; server pushing updates/tokens/progress → SSE is simpler and usually right.

StreamingResponse — Constant-Memory Large Responses

The generator yields chunks; the client starts receiving on the first yield. The CSV export that OOM-killed the naive version becomes a flat, boring memory line.

Yield chunks, flush immediately — 200MB export, 1-row memory

import csv, io
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

# ✗ The naive export: builds ALL rows in RAM, ships once
# @app.get("/export")  → 4 lakh students × 500B = 200MB string + timeout risk

# ✓ Streaming: constant memory, first bytes arrive immediately
@app.get("/students/export.csv")
async def export_students():
    async def rows():
        buf = io.StringIO()
        writer = csv.writer(buf)
        writer.writerow(["id", "name", "branch", "cgpa"])       # header
        yield buf.getvalue(); buf.seek(0); buf.truncate(0)

        async for student in iter_students_in_batches(1000):    # keyset pages!
            writer.writerow([student.id, student.name,
                             student.branch, student.cgpa])
            yield buf.getvalue(); buf.seek(0); buf.truncate(0)

    return StreamingResponse(
        rows(),
        media_type="text/csv",
        headers={"Content-Disposition": "attachment; filename=students.csv"},
    )
# Memory: one row's worth, forever. The DB pagination inside is the
# keyset pattern from the Databases chapters — OFFSET would re-scan.

# Same tool proxies big files without buffering:
#   return StreamingResponse(s3_object.iter_chunks(),
#                            media_type="application/pdf")

SSE — the LLM Token Stream Protocol

text/event-stream + "data: ...\n\n" framing + EventSource on the client. This is verbatim how AI chat UIs stream tokens; note the proxy-buffering header that silently breaks SSE when forgotten.

data: frames over text/event-stream — and the nginx buffering trap

import asyncio, json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.get("/chat/stream")
async def chat_stream(prompt: str):
    async def events():
        # real version: async for chunk in llm.stream(prompt)
        answer = "Binary search halves the range each step, so it is O(log n)."
        for token in answer.split(" "):
            payload = json.dumps({"token": token + " "})
            yield f"data: {payload}\n\n"            # ← SSE frame: data: ...\n\n
            await asyncio.sleep(0.05)                # tokens as they generate
        yield "event: done\ndata: {}\n\n"           # named event = end signal

    return StreamingResponse(
        events(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",    # nginx: DON'T buffer — without this the
        },                                # proxy collects 30s of tokens, sends once,
    )                                     # and your "stream" becomes a lump

# Browser — EventSource reconnects automatically on drops:
#   const es = new EventSource("/chat/stream?prompt=" + q)
#   es.onmessage = (e) => append(JSON.parse(e.data).token)
#   es.addEventListener("done", () => es.close())

# ── SSE vs WebSocket — the decision in four lines ──
#   direction:   SSE server→client only   ·  WS both ways
#   transport:   plain HTTP (proxies/LBs happy)  ·  upgraded protocol
#   reconnect:   EventSource built-in     ·  you write it
#   use SSE for: LLM tokens, notifications, progress, live scores
#   use WS for:  chat with typing, games, collaborative editing
# (This SSE pattern is exactly how Aria streams answers in production.)

Key Points to Remember

1StreamingResponse flushes generator chunks immediately — constant memory for huge responses
2SSE = text/event-stream with "data: ...\n\n" frames; EventSource auto-reconnects
3X-Accel-Buffering: no (and proxy config) — buffering proxies silently kill streams
4One-way push (tokens, progress, notifications) → SSE; two-way interaction → WebSocket

Interview Questions

How does ChatGPT-style token streaming work end to end — server framing, client API, proxy pitfalls?

HardGoogle

Export 40 lakh rows as CSV from FastAPI without OOM — walk through the full solution.

HardFractal

SSE vs WebSocket for a live cricket scoreboard — pick and defend.

MediumDream11

Ask Aria about Streaming & SSE — Pushing Data as It Happens

Your personal AI tutor — ask anything about this concept

Revision Status

Personal Notes

Discussion

Loading discussion…

WebSockets — Real-Time, Two-Way Connections

TestClient — Testing Endpoints with pytest