The Mathematics Behind Neural Network Training
In the early 1990s, a German student named Sepp Hochreiter analyzed why deep neural networks were so hard to train. In his 1991 diploma thesis, he identified a mathematical culprit: as error signals travel backward through many layers, they're multiplied again and again by small numbers, and they fade toward zero.
With the popular sigmoid activation, each layer can multiply the signal by at most 0.25. Through 10 layers, that's 0.25¹⁰ ≈ 0.000001. The earliest layers barely learn at all. This vanishing gradient problem held back deep learning for years, and its solution is a good way into the mathematics of training.
Training Isn't Finding the Best Weights
It sounds like training should find the optimal set of weights. In practice it doesn't, and it doesn't need to. The loss landscape of a large network has countless valleys. Training finds weights that are good enough and, crucially, that generalize to new data.
In fact, the weights that fit the training data perfectly are often worse on new examples. Much of the mathematics of training is about not over-optimizing.
The Training Loop
Every neural network, from a tiny classifier to a large language model, trains with the same four-step loop:
- Forward pass: compute predictions from inputs
- Loss: measure how wrong the predictions are
- Backward pass: compute the gradient of the loss with respect to every weight
- Update: nudge each weight to reduce the loss
Repeat millions of times.
Step 1: The Forward Pass
Each layer applies a matrix, adds a bias, and applies a nonlinearity:
h = f(Wx + b)
Stacking layers gives the full network. Batches of examples are processed together as matrices. Try a layer by hand with the matrix multiplication calculator.
Step 2: The Loss Function
For predicting numbers, a common loss is mean squared error:
L = (1/n) Σ (yᵢ − ŷᵢ)²
For classification, it's cross-entropy:
L = −log(probability assigned to the correct class)
The loss is a single number summarizing performance over a batch, and it's the only signal training follows.
Step 3: Backpropagation
To improve, we need ∂L/∂w for every weight w: how much the loss changes if that weight changes slightly.
Backpropagation applies the chain rule from calculus, layer by layer, starting from the loss and moving backward. For a weight in layer 1 of a 3-layer network:
∂L/∂W₁ = ∂L/∂h₃ × ∂h₃/∂h₂ × ∂h₂/∂h₁ × ∂h₁/∂W₁
This product is exactly where vanishing gradients come from. If each factor is small, the product shrinks exponentially with depth. See the chain rule on the calculus formulas page.
Step 4: Updating Weights
The basic update is gradient descent:
w ← w − η × ∂L/∂w
Where η is the learning rate. Using a small random batch instead of all data at once is stochastic gradient descent (SGD). See Gradient Descent Explained With Simple Mathematics.
Adam
Most modern networks use Adam, introduced by Diederik Kingma and Jimmy Ba in 2014. It keeps two running averages for each weight:
m = β₁·m + (1 − β₁)·g (average gradient, "momentum")
v = β₂·v + (1 − β₂)·g² (average squared gradient)
w ← w − η · m / (√v + ε)
The default values β₁ = 0.9 and β₂ = 0.999 work well across many problems. Dividing by √v gives each weight its own effective learning rate: weights with consistently large gradients take smaller steps.
Making Deep Networks Trainable
Better Activations
The sigmoid function squashes everything into (0, 1), and its derivative never exceeds 0.25. ReLU, f(x) = max(0, x), has a derivative of exactly 1 for positive inputs, so gradients pass through without shrinking.
Careful Initialization
If starting weights are too large, signals explode; too small, they vanish. In 2010, Xavier Glorot and Yoshua Bengio proposed scaling initial weights by the layer size so that signal variance stays roughly constant. In 2015, Kaiming He and colleagues adapted this for ReLU networks, using a variance of 2/n for n inputs.
Residual Connections
ResNets (He et al., 2015) add each layer's input to its output: h = x + F(x). The derivative always includes a direct "+1" path, so gradients can flow through very deep networks. Transformers use the same idea.
Preventing Overfitting
- Validation data: track loss on held-out examples; stop when it starts rising
- Dropout: introduced by Nitish Srivastava, Geoffrey Hinton and colleagues in 2014, it randomly sets a fraction of activations to zero during training, so the network can't rely on any single neuron
- Weight decay: add a penalty λΣw² to the loss, keeping weights small
- More data: often the most effective regularizer of all
An Insider Reference: Estimating the Cost of Training
How much computation does training take? A widely used approximation for transformer language models, popularized by OpenAI's 2020 scaling-law research, is:
Training compute ≈ 6 × N × D
Where N is the number of parameters and D is the number of training tokens. The 6 comes from roughly 2 operations per parameter per token in the forward pass and 4 in the backward pass.
For a 7-billion-parameter model trained on 2 trillion tokens:
6 × 7×10⁹ × 2×10¹² = 8.4 × 10²² operations
The backward pass costs about twice the forward pass, which is why training is so much more expensive than running a model. Work with powers of ten in the scientific calculator.
Two Concepts Worth Knowing
Epoch
An epoch is one full pass through the training data. Large language models often see most of their data only about once; smaller models may train for dozens of epochs.
Learning Rate Schedule
A learning rate schedule changes η during training, often increasing it briefly at the start ("warmup") and then decaying it. Large steps explore early; small steps settle later.
Quick Answer: How Are Neural Networks Trained?
Neural networks are trained by repeating four steps: a forward pass computes predictions, a loss function measures error, backpropagation uses the chain rule to compute gradients for every weight, and an optimizer such as SGD or Adam updates the weights to reduce the loss. Techniques like ReLU, careful initialization, residual connections and dropout make deep networks train reliably.
Try Them Yourself
- Matrix Multiplication Calculator: run a forward pass
- Calculus Formulas: the chain rule behind backpropagation
- Logarithm Calculator: cross-entropy loss values
- Hyperbolic Tangent Calculator: a classic activation function
- Scientific Calculator: estimate training compute
- The Math Behind Neural Networks — Without the Hype: how a single neuron works
Compute 0.25 raised to the 5th, 10th and 20th powers. Those numbers are why deep networks couldn't learn with sigmoids, and why a simple max(0, x) changed everything.