Gradient Descent Explained With Simple Mathematics

Imagine you're standing on a hillside in thick fog, and you want to reach the bottom of the valley. You can't see more than a meter. What do you do?

You feel the ground under your feet, figure out which way slopes downhill, and take a step that way. Then you repeat. Eventually, you reach low ground.

That's gradient descent, the algorithm that trains nearly every modern AI model, from spam filters to large language models. The French mathematician Augustin-Louis Cauchy described the method in 1847. Let's work through it by hand.

A Bigger Step Can Take You Further From the Answer

It seems like bigger steps should get you downhill faster. Up to a point, they do. But past a certain size, each step overshoots the bottom and lands higher up on the other side. The next step overshoots even more, and the process diverges, getting worse forever.

The step size, called the learning rate, is one of the most important numbers in machine learning, and we'll see exactly where the tipping point is.

The Setup: A Simple Valley

Let's minimize this function:

f(x) = (x − 3)²

It's a parabola with its lowest point at x = 3, where f(x) = 0. We'll pretend we don't know that and find it with gradient descent.

We need the slope at any point, which is the derivative:

f′(x) = 2(x − 3)

• At x = 0, the slope is −6: the function decreases as x increases, so go right
• At x = 5, the slope is +4: the function increases as x increases, so go left

See the derivative rules on the calculus formulas page.

The Update Rule

x_new = x − η × f′(x)

Subtract the slope (times a step size η) to move downhill. If the slope is negative, subtracting it moves x to the right. If positive, to the left.

Worked Example: η = 0.1

Start at x = 0:

Each step closes 20% of the remaining gap to 3. The steps get smaller automatically because the slope flattens near the bottom. After 10 steps, x ≈ 2.68; after 30, it's within about 0.004 of the answer.

Three Learning Rates, Three Outcomes

For this function, each update multiplies the distance from 3 by (1 − 2η):

x_new − 3 = (1 − 2η)(x − 3)

The tipping point is where |1 − 2η| = 1, which is η = 1. Beyond it, every step makes things worse. Real functions don't have such tidy formulas, which is why practitioners experiment with learning rates.

Two Dimensions: Following the Gradient

With two variables, the slope becomes a vector called the gradient, made of partial derivatives:

f(x, y) = x² + 10y²
∇f = (2x, 20y)

The gradient points uphill in the steepest direction, so gradient descent steps the opposite way:

(x, y) ← (x, y) − η × (2x, 20y)

This valley is stretched: 10 times steeper in y than in x. With η = 0.09, starting from (10, 1), y bounces back and forth across the valley (1 → −0.8 → 0.64 → −0.51) while x creeps slowly toward 0 (10 → 8.2 → 6.72 → 5.51). This zigzagging is a classic weakness of plain gradient descent, and it's why optimizers add momentum, an idea Boris Polyak introduced in 1964, to smooth out the path.

An Insider Reference: From Cauchy to Stochastic Descent

Cauchy's 1847 paper proposed the method to solve systems of equations arising in astronomy. For the next century, it remained a numerical tool.

In 1951, statisticians Herbert Robbins and Sutton Monro published a method for finding solutions when you can only measure noisy estimates. That idea underlies stochastic gradient descent (SGD): instead of computing the gradient on all data (expensive), estimate it from a small random batch. Each step is noisier, but vastly cheaper, and the noise can even help escape poor regions.

In 2014, Yann Dauphin and colleagues argued that in very high-dimensional problems like neural networks, the main obstacle isn't getting stuck in bad local minima, but slowing down near saddle points, spots that curve up in some directions and down in others. That insight helped explain why simple gradient methods work so well on huge networks.

Two Concepts Worth Knowing

Derivative

The derivative f′(x) measures how fast f changes at x: the slope of the tangent line. Gradient descent only needs local slope information, never a view of the whole landscape. See Why Is the Derivative of x² Equal to 2x?.

Convex Function

A convex function curves upward everywhere, like a bowl, so any local minimum is the global minimum. Our (x − 3)² is convex. Neural network losses aren't, which makes training more of an art.

Quick Answer: What Is Gradient Descent?

Gradient descent is an algorithm that minimizes a function by repeatedly stepping in the direction opposite its gradient (slope): x ← x − η·f′(x). The learning rate η sets the step size: too small is slow, too large overshoots or diverges. It's the core method used to train machine learning models.

Try Them Yourself

• Calculus Formulas: derivatives and partial derivatives
• Square Numbers List: values of the loss (x − 3)²
• Scientific Calculator: iterate the update rule yourself
• Algebra Formulas: parabolas and their vertices
• The Mathematics Behind Machine Learning: where gradient descent fits
• The Mathematics Behind Neural Network Training: gradient descent at scale

Minimize f(x) = (x + 2)² starting from x = 5 with η = 0.25. Before you calculate, predict how many steps it will take to get within 0.1 of the answer.