Why Derivatives Matter in Machine Learning
A large language model can have hundreds of billions of adjustable numbers. During training, it needs to know, for every single one, whether nudging it up or down will make the model a little better.
You could find out by nudging each weight and re-running the model. For a 175-billion-parameter model, that means about 175 billion full model runs, just to take one training step. It would never finish.
Instead, the model computes all of those answers at once, in roughly the time of a few model runs. The tool that makes this possible is the derivative, computed by a technique called automatic differentiation.
Computing All Derivatives at Once Costs About the Same as Computing One
It seems like knowing the effect of a billion different weights should take a billion times more work than knowing the effect of one. It doesn't.
Using the chain rule in reverse order, a network can compute the derivative of its loss with respect to every weight in one backward pass, whose cost is only a small multiple (typically around two to three times) of the forward pass. This single mathematical fact is the reason deep learning is practical.
What a Derivative Measures
The derivative of a function tells you how much its output changes when you nudge its input:
f′(x) = lim (h → 0) [f(x + h) − f(x)] / h
In machine learning, the function is the loss (how wrong the model is), and the input is a weight. The derivative ∂L/∂w answers:
If I increase this weight slightly, does the error go up or down, and by how much?
A negative derivative means increasing the weight reduces error. A large derivative means the loss is very sensitive to that weight. See the definitions on the calculus formulas page.
A Worked Example
A one-weight model predicts ŷ = w·x. For one training example with x = 2 and target y = 3, the squared-error loss is:
L(w) = (w·x − y)²
At w = 0.5, the prediction is 1, so L = (1 − 3)² = 4.
Using the chain rule:
dL/dw = 2(w·x − y) · x = 2(−2)(2) = −8
The derivative is −8: increasing w will reduce the loss quickly. Gradient descent updates w ← w − η·(−8), moving w upward toward the right answer, w = 1.5.
Three Ways to Compute Derivatives
1. Numerical Differentiation
Estimate by nudging:
dL/dw ≈ [L(w + h) − L(w − h)] / (2h)
With h = 0.001, this gives −7.99999…, very close to −8. But it has two big problems. It needs two model evaluations per weight, which is hopeless for billions of weights. And choosing h is tricky: too large is inaccurate, too small causes floating-point rounding errors. It's mainly used to check other methods.
2. Symbolic Differentiation
Apply calculus rules to formulas, as you would by hand or with a computer algebra system. It's exact, but expressions can grow enormously as rules are applied repeatedly through many layers.
3. Automatic Differentiation
Automatic differentiation (autodiff) records the sequence of simple operations a program performs (additions, multiplications, activations) and applies the chain rule to each, numerically. It's exact up to floating-point precision, and in reverse mode it computes the derivative with respect to every input in one backward sweep. Backpropagation is reverse-mode autodiff applied to neural networks.
An Insider Reference: Linnainmaa's 1970 Thesis
The general method of reverse-mode automatic differentiation was described by Finnish mathematician Seppo Linnainmaa in his 1970 master's thesis at the University of Helsinki. He was studying how rounding errors accumulate in computer calculations, and showed how to compute the sensitivity of a result to every intermediate value efficiently.
Related ideas appeared independently in control theory and elsewhere, and David Rumelhart, Geoffrey Hinton and Ronald Williams made backpropagation famous for training neural networks in 1986. Today, frameworks like PyTorch and JAX implement autodiff so that researchers write only the forward computation. The derivatives come for free.
Derivatives of Activation Functions
Every nonlinearity in a network needs a derivative, and those derivatives shape how well learning works.
| Activation | Formula | Derivative |
|---|---|---|
| Sigmoid | σ(x) = 1/(1 + e⁻ˣ) | σ(x)(1 − σ(x)), at most 0.25 |
| Tanh | tanh(x) | 1 − tanh²(x), at most 1 |
| ReLU | max(0, x) | 0 for x < 0, 1 for x > 0 |
The sigmoid's maximum derivative of 0.25 explains the vanishing gradient problem: multiplied through 10 layers, 0.25¹⁰ ≈ 0.000001. ReLU's derivative of 1 for positive inputs lets signals pass through deep networks. Explore tanh values with the hyperbolic tangent calculator.
Partial Derivatives and the Gradient
A real model has many weights, so we take partial derivatives: how the loss changes with one weight while holding the others fixed. Collected into a vector, they form the gradient:
∇L = (∂L/∂w₁, ∂L/∂w₂, …, ∂L/∂wₙ)
The gradient points in the direction of steepest increase in loss. Training steps the opposite way. See Gradient Descent Explained With Simple Mathematics.
Two Concepts Worth Knowing
Chain Rule
The chain rule says the derivative of a composition is the product of derivatives: if y = f(g(x)), then dy/dx = f′(g(x)) · g′(x). Neural networks are long chains of compositions, so the chain rule is everything.
Sensitivity
A derivative is a sensitivity: how responsive an output is to an input. Beyond training, sensitivities help explain which input features most influenced a model's prediction.
Quick Answer: Why Are Derivatives Important in Machine Learning?
Derivatives measure how a model's error changes when each weight changes. Training uses these derivatives, collected into a gradient, to adjust weights in the direction that reduces error. Automatic differentiation computes derivatives for billions of weights efficiently using the chain rule in reverse, which is what makes training large neural networks possible.
Try Them Yourself
- Calculus Formulas: derivative rules and the chain rule
- Hyperbolic Tangent Calculator: an activation and its slope
- Logarithm Calculator: the e⁻ˣ inside the sigmoid
- Scientific Calculator: check a derivative numerically
- Why Is the Derivative of x² Equal to 2x?: derivatives from first principles
- The Mathematics Behind Neural Network Training: derivatives in the training loop
Take our one-weight model and compute the loss at w = 0.499 and w = 0.501. Divide the difference by 0.002. You've just checked a gradient the way ML engineers debug their own code.