The Mathematics Behind Machine Learning

The core method behind much of machine learning is more than 200 years old. In 1805, Adrien-Marie Legendre published the method of least squares to fit the orbits of comets. Carl Friedrich Gauss claimed he'd been using it since 1795.

Modern machine learning is vastly bigger, but the recipe is recognizably the same: choose a model with adjustable numbers, measure how wrong it is, and adjust the numbers to make it less wrong. That recipe rests on four pillars of mathematics: linear algebra, calculus, probability and optimization.

A Perfect Fit Is Usually a Bad Sign

It seems obvious that the best model is the one that fits your data perfectly. In machine learning, a model with zero error on its training data is often a warning sign.

Given 10 data points, a polynomial of degree 9 can pass through every single one exactly. But between and beyond those points it tends to swing wildly. It has memorized noise instead of learning the pattern. That's overfitting, and avoiding it is one of the central problems in the field.

Pillar 1: Linear Algebra, the Language of Data

Machine learning represents everything as vectors and matrices:

• A house with 3 bedrooms, 2 bathrooms and 1,800 square feet is the vector (3, 2, 1800)
• A dataset of 10,000 houses is a 10,000 × 3 matrix X
• A 28 × 28 grayscale image is a vector of 784 numbers

A linear model predicts with a dot product:

ŷ = w · x + b = w₁x₁ + w₂x₂ + … + wₙxₙ + b

For a whole dataset at once, it's a single matrix multiplication: ŷ = Xw + b. GPUs are fast at machine learning because they're fast at exactly this. Practice with the matrix multiplication calculator.

Pillar 2: Loss Functions, Measuring Wrongness

To improve a model, you need a number that says how bad it is. That's the loss function. For predicting numbers, the classic choice is mean squared error:

MSE = (1/n) Σ (yᵢ − ŷᵢ)²

For linear regression, minimizing MSE has an exact solution, the normal equation:

w = (XᵀX)⁻¹ Xᵀy

That's Legendre's least squares in matrix form. The transpose calculator shows what Xᵀ does.

For classification ("cat or dog?"), models output probabilities and use cross-entropy loss, which punishes confident wrong answers heavily:

Loss = −log(p_correct)

If the model gives the right answer 70% probability, the loss is −ln(0.7) ≈ 0.36. At 10%, it's −ln(0.1) ≈ 2.30.

Pillar 3: Calculus, Which Way Is Downhill?

Most models don't have a neat formula for the best weights. Instead we ask: if I nudge each weight slightly, does the loss go up or down?

That's a derivative. With many weights, the collection of partial derivatives is the gradient, ∇L, which points in the direction of steepest increase. So we step the opposite way:

w ← w − η × ∇L(w)

That's gradient descent, and η is the learning rate. Too small and training crawls. Too large and it overshoots and diverges. For neural networks with many layers, the gradient is computed with the chain rule, an algorithm known as backpropagation. See the rules on the calculus formulas page.

Pillar 4: Probability and Statistics

Data is noisy and the future is uncertain, so machine learning is fundamentally statistical:

• Probability distributions describe uncertainty in predictions
• Bayes' theorem updates beliefs as evidence arrives
• Maximum likelihood estimation picks parameters that make the observed data most probable

Here's a neat connection. If you assume prediction errors follow a normal distribution, maximizing likelihood turns out to be exactly the same as minimizing mean squared error. Least squares isn't arbitrary; it's what probability recommends for bell-curve noise. See the standard normal table.

Generalization: Train, Validate, Test

Because a perfect training fit can be misleading, practitioners split data:

The difference between training error and test error reveals how well the model generalizes. This is often framed as the bias–variance trade-off: simple models miss patterns (high bias), while overly flexible models chase noise (high variance).

An Insider Reference: Arthur Samuel's Checkers Program

The term "machine learning" is usually credited to Arthur Samuel, an IBM researcher. In a 1959 paper, "Some Studies in Machine Learning Using the Game of Checkers," he described a program that improved by playing against itself.

Samuel's program scored board positions with a weighted sum of features, such as piece advantage and mobility, and adjusted the weights based on results. Swap checkers for images or text and scale up the numbers, and that idea is still at the heart of modern AI.

Two Concepts Worth Knowing

Regularization

Regularization adds a penalty for large weights to the loss, such as λ × Σ w². It discourages wild, overfit models by preferring simpler explanations, a mathematical version of Occam's razor.

Feature Scaling

If one feature ranges from 0 to 1 and another from 0 to 100,000, gradient descent struggles. Standardizing each feature with a z-score, z = (x − μ) / σ, puts them on a common scale. Try it with the z-score calculator.

Quick Answer: What Math Do You Need for Machine Learning?

Machine learning relies on linear algebra (vectors and matrices to represent data and models), calculus (gradients to measure how to improve), probability and statistics (to model uncertainty and evaluate results), and optimization (gradient descent to minimize a loss function).

Try Them Yourself

• Matrix Multiplication Calculator: compute predictions for a batch
• Matrix Transpose Calculator: part of the normal equation
• Calculus Formulas: derivatives and the chain rule
• Statistics Formulas: mean, variance and standard deviation
• Z-Score Calculator: standardize features
• Logarithm Calculator: compute cross-entropy losses
• Unlocking Insights With Regression Analysis: the original machine learning model

Take five points from any chart, fit a straight line with least squares by hand, and compute its MSE. You've just trained your first machine learning model.