Math Tools Math Tools
Share:
← Back to Blog

The Math Behind Neural Networks — Without the Hype

The Math Behind Neural Networks — Without the Hype

By Math Tools ·

The Math Behind Neural Networks — Without the Hype

Neural networks are often described with brain metaphors: neurons, synapses, learning, even "thinking." Strip those away and each artificial neuron does something a middle-school student could follow: multiply, add, and bend.

What makes neural networks powerful isn't a mysterious ingredient. It's that simple operation stacked into layers, repeated millions or billions of times, and tuned with calculus that dates back to the 1600s.

Without One Simple "Bend," Deep Networks Are Useless

Stack 100 layers of weighted sums, and you'd think you've built something very expressive. You haven't. A weighted sum of weighted sums is just another weighted sum:

W₂(W₁x) = (W₂W₁)x = Wx

Multiplying matrices together gives another matrix. Without something extra, a 100-layer network is mathematically identical to a single layer.

The extra ingredient is a nonlinear activation function, a small bend applied after each layer. That bend is what lets depth matter.

One Neuron

A single artificial neuron takes inputs x₁, x₂, …, xₙ and computes:

z = w₁x₁ + w₂x₂ + … + wₙxₙ + b
output = f(z)
  • w values are weights: how much each input matters
  • b is the bias: a baseline shift
  • f is the activation function

The most popular activation today is the ReLU (Rectified Linear Unit):

ReLU(z) = max(0, z)

It's zero for negative inputs and a straight line for positive ones. Simple, fast to compute, and just nonlinear enough.

A Worked Example

A neuron has inputs x = (2, 3), weights w = (0.5, −1) and bias b = 1:

z = 0.5×2 + (−1)×3 + 1 = 1 − 3 + 1 = −1
ReLU(−1) = 0

Change the second weight to −0.2:

z = 1 − 0.6 + 1 = 1.4
ReLU(1.4) = 1.4

A small change in one weight switched the neuron from "off" to "on." Training is the process of finding weights that switch the right neurons on for the right inputs.

Layers Are Matrix Multiplications

A layer of many neurons is computed all at once:

h = f(Wx + b)

W is a matrix with one row per neuron. A network that classifies handwritten digits might use:

  • 784 inputs (a 28 × 28 pixel image)
  • A hidden layer of 128 neurons
  • An output layer of 10 neurons (digits 0–9)

That's 784 × 128 + 128 weights and biases in the first layer, and 128 × 10 + 10 in the second: 101,770 parameters in total. Try a small layer with the matrix multiplication calculator.

Turning Outputs Into Probabilities: Softmax

The 10 output numbers (called logits) can be any values. The softmax function turns them into probabilities that sum to 1:

softmax(zᵢ) = e^(zᵢ) / Σ e^(zⱼ)

For logits (2.0, 1.0, 0.1), softmax gives about (0.66, 0.24, 0.10). The exponential function exaggerates differences, so the largest logit wins most of the probability.

Learning: Loss and Gradient Descent

To train, we compare predictions with correct answers using a loss function, typically cross-entropy: loss = −ln(probability assigned to the correct answer). Then we adjust every weight to reduce the loss:

w ← w − η × ∂L/∂w

The challenge is computing ∂L/∂w for every weight, possibly billions of them, efficiently.

Backpropagation Is the Chain Rule

A weight in an early layer affects the loss through a chain of later calculations. Calculus's chain rule handles exactly this:

∂L/∂w = ∂L/∂output × ∂output/∂h × ∂h/∂z × ∂z/∂w

Backpropagation computes these derivatives layer by layer, starting from the loss and moving backward, reusing results as it goes. It costs roughly the same as a couple of forward passes, no matter how many weights there are. The derivative rules are on the calculus formulas page.

An Insider Reference: Seventy Years of Ups and Downs

  • 1958: Frank Rosenblatt at Cornell introduced the perceptron, a single-layer neural network. The press predicted machines that would walk, talk and be conscious.
  • 1969: Marvin Minsky and Seymour Papert published Perceptrons, proving single-layer networks can't learn simple functions like XOR. Funding collapsed.
  • 1986: David Rumelhart, Geoffrey Hinton and Ronald Williams published their influential Nature paper on backpropagation, showing multi-layer networks could learn useful internal representations.
  • 2012: AlexNet, by Alex Krizhevsky, Ilya Sutskever and Hinton, won the ImageNet image recognition challenge with a top-5 error rate of about 15.3%, versus 26.2% for the runner-up. It had about 60 million parameters and was trained on two GPUs.

Hinton shared the 2018 Turing Award with Yoshua Bengio and Yann LeCun, and the 2024 Nobel Prize in Physics with John Hopfield.

Two Concepts Worth Knowing

Universal Approximation

In 1989, George Cybenko proved that a network with just one hidden layer and enough neurons can approximate any continuous function on a bounded region as closely as you like. It guarantees such a network exists, but says nothing about how to find its weights or how many neurons you'll need.

The Exponential Function

Softmax, cross-entropy and many activations rely on e^x and its inverse, ln x. Their neat derivatives (the derivative of e^x is e^x) make gradients cheap to compute. Explore them with the logarithm calculator.

Quick Answer: How Does a Neural Network Work Mathematically?

A neural network passes input numbers through layers. Each layer multiplies by a weight matrix, adds a bias, and applies a nonlinear activation such as ReLU. Training measures error with a loss function and uses backpropagation, which is the chain rule, to compute gradients, then adjusts the weights with gradient descent.

Try Them Yourself

Build a single neuron on paper with two inputs, then adjust its weights until it outputs 1 only when both inputs are 1. You'll have trained an AND gate, the first step Rosenblatt took in 1958.

Next → The Mathematics Behind Neural Network Training