Probability Distributions & the Normal Distribution

Overview

A probability distribution is the full picture of an uncertain quantity — every possible value and how likely it is. You will meet a handful repeatedly: the Bernoulli/Binomial for yes-no events, the categorical for multi-class outcomes (exactly what a softmax layer outputs), the uniform for "equally likely", and above all the normal (Gaussian) distribution, the bell curve defined by a mean and a standard deviation. The normal is everywhere because of the Central Limit Theorem: sums of many independent random effects tend toward a bell curve, so it is a natural model for noise and measurement error. Practically, neural-net weights are initialized from carefully scaled normals, dropout and augmentation inject noise, and generative models learn to map simple normal noise into complex data. Recognising which distribution fits a situation is a core modelling skill.

The normal distribution: mean and spread

import numpy as np
rng = np.random.default_rng(0)

samples = rng.normal(loc=0.0, scale=1.0, size=10000)   # mean 0, std 1
print(round(samples.mean(), 3), round(samples.std(), 3))  # ~0 ~1

# ~68% of a normal lies within 1 std, ~95% within 2 std:
within1 = np.mean(np.abs(samples) < 1)
print(round(within1, 3))    # ~0.68

Central Limit Theorem: why bell curves are everywhere

import numpy as np
rng = np.random.default_rng(0)

# Start from a very non-normal (uniform) distribution:
means = [rng.uniform(0, 1, size=30).mean() for _ in range(5000)]
means = np.array(means)
print(round(means.mean(), 3))   # ~0.5
print(round(means.std(), 3))    # small -> tight bell around 0.5

Key Points to Remember

• 1A distribution lists every possible value and its probability
• 2Softmax outputs a categorical distribution over classes
• 3The normal is set by mean & std; 68/95/99.7% within 1/2/3 std
• 4Central Limit Theorem: sums/averages of many effects → normal; models noise & weight init