Sampling, Estimation & Confidence Intervals

Overview

You rarely have the whole population; you have a sample, and you use it to estimate quantities about the whole (a mean, an accuracy, a click-through rate). Because a different sample would give a slightly different estimate, every estimate carries uncertainty. A confidence interval expresses that uncertainty as a range ("accuracy is 84% ± 3%") rather than a single misleading number. This matters enormously in ML evaluation: reporting that model A scores 84.1% and model B scores 83.9% is meaningless if the confidence intervals overlap heavily — the difference is noise. Related ideas — sampling bias (your sample must represent the population), the standard error (how estimate variability shrinks with more data), and resampling methods like bootstrapping — are what separate rigorous evaluation from cherry-picked results. This is the statistical backbone of trustworthy AI claims.

Estimate + standard error: uncertainty shrinks with n

import numpy as np
rng = np.random.default_rng(0)

pop_std = 1.0
for n in [25, 100, 400]:
    se = pop_std / np.sqrt(n)          # standard error of the mean
    print(f"n={n:>3}  standard error={se:.3f}")
# n= 25  0.200
# n=100  0.100   -> 4x data -> 2x tighter
# n=400  0.050

Bootstrap a confidence interval for model accuracy

import numpy as np
rng = np.random.default_rng(0)

correct = rng.random(200) < 0.84        # 200 test predictions, ~84% correct
boot = [rng.choice(correct, size=len(correct), replace=True).mean()
        for _ in range(2000)]
lo, hi = np.percentile(boot, [2.5, 97.5])
print(f"accuracy 95% CI: [{lo:.3f}, {hi:.3f}]")   # e.g. [0.79, 0.89]

Key Points to Remember

• 1We estimate population quantities from finite samples — every estimate has uncertainty
• 2Standard error of the mean = std/√n; uncertainty shrinks slowly with data
• 3A confidence interval reports a range, not a single fragile number
• 4Overlapping CIs mean a metric difference may be noise — essential for honest model comparison