Functions, Limits & Continuity

Overview

Calculus is the mathematics of change, and it starts with the limit: the value a function approaches as its input creeps toward some point, even if it never quite arrives. This is the idea that makes derivatives possible (a derivative is a limit of a slope over a shrinking interval). Continuity — a function with no gaps, jumps, or holes — matters enormously in deep learning because training relies on nudging inputs a tiny bit and seeing a correspondingly tiny change in output. If a function jumped around wildly, gradient descent could not follow it downhill. This is exactly why the activation functions inside networks are chosen to be (mostly) smooth, and why non-smooth points (like ReLU's kink at zero) get special handling. You do not need epsilon-delta proofs; you need the picture of "approaching a value smoothly".

A limit: the value a function heads toward

import numpy as np

def f(x):
    return np.sin(x) / x        # undefined exactly at 0

for x in [0.1, 0.01, 0.001]:
    print(x, round(f(x), 6))
# heads toward 1 as x -> 0, even though f(0) is 0/0
# 0.1   0.998334
# 0.01  0.999983
# 0.001 1.0

Continuity enables learning by small nudges

import numpy as np

# A tiny input nudge -> a tiny output change (continuous/smooth):
sigmoid = lambda x: 1 / (1 + np.exp(-x))
x = 1.0
print(round(sigmoid(x + 1e-4) - sigmoid(x), 8))   # ~2e-5, tiny & smooth

# A step function is discontinuous -> the same nudge can do nothing... then jump:
step = lambda x: (x > 0).astype(float) if isinstance(x, np.ndarray) else float(x > 0)
print(step(0.00005) - step(-0.00005))             # a full jump of 1.0

Key Points to Remember

• 1A limit is the value a function approaches as the input nears a point
• 2A derivative is defined as a limit of slopes over a shrinking interval
• 3Continuity = no jumps; small input change → small output change
• 4Networks use smooth activations so gradient descent can follow the loss downhill