How Neural Networks Work
IntermediateA neural network is a mathematical function loosely inspired by the structure of biological brains. It consists of layers of simple computational units (neurons) connected by weighted links. Feed it an input (an image, text, sound), it performs millions of weighted multiplications and nonlinear transformations, producing an output (a class label, a generated word, a probability). Training adjusts the weights using backpropagation so the network's outputs match desired answers. With enough data and depth ("deep learning"), neural networks can learn to recognise faces, translate languages, generate images, and play games at superhuman levels.
Think of a neural network like a committee of voters
Imagine 1,000 committee members voting on whether a photo is a cat. Each member specialises in one tiny detail: "pointy ears?", "whiskers?", "slit pupils?", "fur texture?". Each votes yes or no with a certain confidence. Their votes are weighted (the "whiskers expert" counts more than the "colour expert" since colour doesn't define cats). The final answer is the weighted sum of all votes. Training the network is like adjusting how much each expert's vote counts, based on whether the committee as a whole kept getting the answer wrong.
Step by Step
Key Concepts
Neuron (Node)
The basic computational unit of a neural network. Receives multiple numerical inputs, computes their weighted sum plus a bias, applies an activation function, and passes the result to the next layer. Inspired by biological neurons but far simpler — just a parameterised mathematical function.
Weight
A learnable parameter that determines how much influence one neuron's output has on the next neuron. Analogous to synaptic strength in biological brains. A network with millions of neurons has billions of weights. Training is the process of finding weight values that minimise prediction error.
Deep Learning
Machine learning using neural networks with many layers (typically 10-1000+ layers). "Deep" refers to the depth (number of layers). Deep networks automatically learn hierarchical feature representations from raw data — eliminating the need for hand-engineered features. Enabled by GPUs, large datasets, and algorithmic advances from ~2012 onwards.
Convolutional Neural Network (CNN)
A neural network architecture specialised for grid-structured data (images, video). Convolutional layers apply learnable filters across the input, detecting local patterns regardless of position. Pooling layers reduce spatial dimensions. CNNs power most computer vision applications: face recognition, medical imaging, autonomous driving perception.
Transformer
The neural network architecture behind modern language models (GPT, BERT, Gemini). Uses attention mechanisms to weigh the relevance of every position in the input when computing each output. Unlike RNNs, transformers process all positions in parallel, enabling training on massive text datasets. Also used in vision (Vision Transformer) and biology (AlphaFold for protein structure).
Dropout
A regularisation technique that randomly zeroes out a fraction of neurons during each training step. Forces the network to learn redundant representations — no single neuron becomes too critical. Prevents overfitting by acting like training an ensemble of many different networks simultaneously. Dropout is typically disabled during inference.
Batch Normalisation
A technique that normalises the activations of each layer to have zero mean and unit variance during training. Dramatically speeds up training by allowing higher learning rates and reducing sensitivity to weight initialisation. Also acts as a regulariser. One of the key innovations (2015) that enabled training of very deep networks.
Key Facts
- AlexNet (2012) had 60 million parameters and 8 layers. GPT-4 is estimated to have ~1.8 trillion parameters across ~120 layers — a 30,000x increase in 12 years.
- Training GPT-4 is estimated to have used approximately 25,000 A100 GPUs for around 90-100 days, consuming ~50 GWh of electricity — comparable to the annual energy use of 4,500 average Indian homes.
- A single A100 GPU performs 312 teraflops (312 trillion floating point operations per second) — neural network training is embarrassingly parallel, which is why GPUs replaced CPUs for AI.
- DeepMind's AlphaFold 2 (2020) used neural networks to predict protein 3D structure from amino acid sequence — solving a 50-year-old biology grand challenge and predicting structures for virtually all 200 million known proteins.
- A human retinal ganglion cell connects to about 10,000 other neurons; a neuron in GPT-4's attention layers effectively attends to every other position in a 32,768-token context — a fundamentally different connectivity pattern.
- Neural networks were first proposed by McCulloch and Pitts in 1943 and largely abandoned in the 1970s as intractable. The backpropagation algorithm was popularised in 1986 and enabled the modern deep learning era decades later.
Real-World Applications
Computer Vision
CNNs power facial recognition (used in Aadhaar verification, phone unlock), medical image analysis (detecting cancer in radiology scans), defect detection in manufacturing, and all object recognition in autonomous vehicles.
Natural Language Processing
Transformer-based networks power translation (Google Translate: 100+ languages), sentiment analysis, question answering, document summarisation, code generation (GitHub Copilot), and conversational AI (ChatGPT, Gemini).
Scientific Discovery
AlphaFold revolutionised structural biology. Neural networks discover new antibiotics, predict climate patterns, accelerate particle physics analysis, and identify exoplanets in telescope data — science applications that classical methods couldn't tackle.
Generative AI
Diffusion models (Stable Diffusion, DALL-E) and GANs use neural networks to generate photorealistic images, videos, and audio from text descriptions. Generative AI is transforming creative industries: design, advertising, film, music production.
Game Playing and Robotics
DeepMind's AlphaGo (2016) and AlphaZero beat world champions at Go, Chess, and Shogi using neural networks with reinforcement learning. Boston Dynamics robots use neural networks for balance and locomotion control over unstructured terrain.
Frequently Asked Questions
Are neural networks modelled on the human brain?
Loosely inspired, but fundamentally different. The neuron analogy (inputs, weights, activation) is the only meaningful similarity. Biological neurons communicate with discrete spikes; artificial neurons use continuous values. Biological learning involves complex synaptic chemistry; artificial learning uses gradient descent. Deep neural networks are more accurately described as differentiable function approximators than brain models.
Why do neural networks need so much data?
Neural networks have millions to billions of parameters to learn. With too little data, networks memorise training examples rather than learning generalisable patterns (overfitting). The rule of thumb: you need at least 10 examples per parameter, though modern techniques (transfer learning, data augmentation, self-supervised learning) dramatically reduce labelled data requirements. GPT-3 learned from ~300 billion tokens; the average human learns from roughly equivalent input by age 18.
What is the difference between a neural network and a decision tree?
Decision trees are interpretable: you can trace exactly why a prediction was made (follow the branches). Neural networks are opaque "black boxes" — predictions emerge from millions of interacting parameters with no simple human-readable explanation. Decision trees work well on structured tabular data; neural networks excel on unstructured data (images, text, audio). Tree-based ensembles (XGBoost, Random Forest) often outperform neural networks on tabular data.
Can neural networks explain their reasoning?
Generally no — this is the interpretability/explainability problem. Techniques like SHAP values, LIME, attention visualisation, and gradient-based saliency maps provide partial explanations of which inputs most influenced a prediction, but these are post-hoc approximations, not true causal reasoning chains. For high-stakes decisions (medical diagnosis, loan approval), explainability is a critical regulatory and ethical requirement that current neural networks struggle to fully meet.