Cost/Loss Functions & Convexity

Overview

Training is optimization, and optimization needs an objective: the loss (or cost) function, a single number measuring how wrong the model currently is. Choosing the right loss is a modelling decision — mean squared error for regression, cross-entropy for classification — because it defines what "good" means. Just as important is the shape of the loss surface. A convex function is bowl-shaped: it has exactly one minimum, and gradient descent is guaranteed to find it (linear/logistic regression enjoy this). Deep networks are non-convex — their loss surface is a rugged landscape with many local minima and saddle points — which is why training is more art (initialization, learning-rate schedules, momentum) than guarantee. Understanding convex vs non-convex tells you why some models "just train" and others need careful tuning.

Common losses define what "wrong" means

import numpy as np

# Regression: mean squared error
y, yhat = np.array([3.0, 5.0, 2.0]), np.array([2.5, 5.0, 4.0])
mse = np.mean((y - yhat)**2)
print("MSE:", round(mse, 3))                       # 0.75

# Classification: binary cross-entropy
p = np.array([0.9, 0.1, 0.8]); t = np.array([1, 0, 1])
bce = -np.mean(t*np.log(p) + (1-t)*np.log(1-p))
print("BCE:", round(bce, 3))                       # low = confident & correct

Convex (one bowl) vs non-convex (many traps)

import numpy as np

convex     = lambda x: x**2                       # single minimum at 0
nonconvex  = lambda x: x**2 + 10*np.sin(x)        # many local minima

xs = np.linspace(-6, 6, 13)
print("convex mins are unique; nonconvex has several dips:")
print(np.round(nonconvex(xs), 1))   # multiple valleys -> local minima traps

Key Points to Remember

• 1The loss function is the single number training minimises — it defines "good"
• 2MSE for regression; cross-entropy for classification
• 3Convex losses have one global minimum; gradient descent is guaranteed to reach it
• 4Deep networks are non-convex — initialization, schedules, and momentum matter