Why Matrices Are Everywhere in Machine Learning

In 1969, German mathematician Volker Strassen surprised the mathematical world by showing that two 2 × 2 matrices can be multiplied with 7 multiplications instead of the obvious 8. It sounds like a tiny improvement. Applied recursively to huge matrices, it beats the textbook method.

More than 50 years later, in 2022, DeepMind announced AlphaTensor, an AI system that discovered new matrix multiplication shortcuts, including multiplying 4 × 4 matrices (in arithmetic modulo 2) with 47 multiplications instead of the previous best of 49.

Why does anyone care so much about multiplying grids of numbers? Because modern machine learning is, at its computational core, an enormous amount of matrix multiplication.

"Intelligence" Runs on Multiply-and-Add

Machine learning models can translate languages, recognize faces and write code. It's natural to imagine something mysterious happening inside. Computationally, most of the work is simple: multiply matrices, add vectors, apply a basic function, repeat.

The sophistication is in the values of the matrices, learned from data, and in the scale: billions of numbers, multiplied trillions of times.

Data Is a Matrix

A dataset of houses with three features each:

        bedrooms  bathrooms  sq_ft
House 1 [   3        2       1800 ]
House 2 [   4        3       2400 ]
House 3 [   2        1        950 ]

That's a 3 × 3 matrix X. Real datasets have millions of rows. A grayscale image is a matrix of pixel brightness values. A color image is three matrices stacked together, a tensor. A sentence becomes a matrix where each row is a word's embedding vector.

Models Are Matrices

A linear model predicts with weights w:

ŷ = Xw

One matrix multiplication computes a prediction for every row at once.

A neural network layer does the same thing, then applies a simple nonlinear function f:

h = f(Wx + b)

The matrix W holds the layer's learned weights. A layer that maps 4,096 inputs to 4,096 outputs has a weight matrix with over 16.7 million entries. GPT-3, released by OpenAI in 2020, had 175 billion parameters, almost all of them entries in weight matrices. Try a small version with the matrix multiplication calculator.

Batches: Doing Everything at Once

Training processes examples in batches. Instead of running 64 examples through a layer one at a time, stack them into a matrix and do one multiplication:

H = f(XWᵀ + b)

This is why GPUs dominate machine learning. They were designed to perform thousands of simple arithmetic operations in parallel, which is exactly what large matrix multiplication needs. The matrix transpose calculator shows what Wᵀ does.

Attention Is Matrix Multiplication

The transformer architecture, introduced in the 2017 paper "Attention Is All You Need," powers today's large language models. Its core operation is:

Attention(Q, K, V) = softmax(QKᵀ / √d) V

• Q, K, V are matrices of queries, keys and values computed from the input
• QKᵀ is a matrix of dot products: how relevant each word is to each other word
• softmax turns each row into weights that sum to 1
• Multiplying by V blends information accordingly

Three matrix multiplications and a normalization. For a sequence of n tokens, QKᵀ is an n × n matrix, which is why long contexts are expensive.

Training Is Matrix Calculus

To train, we compute how the loss changes with every weight, the gradient. For a layer y = Wx, the gradient with respect to W is itself a matrix product:

∂L/∂W = (∂L/∂y) xᵀ

Backpropagation is a sequence of matrix multiplications running backward through the network, roughly doubling the multiplication workload of the forward pass. See derivative rules on the calculus formulas page.

An Insider Reference: The Race to Multiply Faster

Multiplying two n × n matrices the textbook way takes about n³ operations. Doubling the matrix size makes it 8 times slower.

• 1969: Strassen reduced the exponent to about 2.807
• 1990: Coppersmith and Winograd pushed it to about 2.376
• 2020s: theoretical improvements have brought it to roughly 2.37, though these algorithms are too slow in practice for real matrices
• 2022: DeepMind's AlphaTensor, described in Nature, treated finding multiplication shortcuts as a game and found algorithms that beat the best known methods for several small matrix sizes

In practice, the biggest speedups come from hardware and software engineering: GPUs, specialized AI chips, and libraries tuned to keep matrix data flowing efficiently through memory.

Low-Rank Tricks: Smaller Matrices, Same Power

A big matrix often contains much less information than its size suggests. Singular value decomposition (SVD) breaks a matrix into pieces ordered by importance, so it can be approximated by a product of two thin matrices.

In 2021, Microsoft researchers introduced LoRA (Low-Rank Adaptation). To fine-tune a large language model, LoRA freezes the original weight matrix W and learns a small update:

W′ = W + BA,   where B and A are tall-and-thin matrices

The researchers reported reducing the number of trainable parameters for GPT-3 fine-tuning by about 10,000 times, while matching the quality of full fine-tuning.

Two Concepts Worth Knowing

Tensor

A tensor generalizes matrices to more dimensions: a vector is 1D, a matrix is 2D, and a batch of color images is 4D (batch × height × width × channels). Libraries like PyTorch and TensorFlow are named after this idea.

Rank

The rank of a matrix is the number of independent directions it represents. Low-rank matrices can be stored and multiplied much more cheaply, which is the idea behind LoRA and model compression.

Quick Answer: Why Are Matrices Used in Machine Learning?

Machine learning uses matrices because data, model weights and intermediate results are all naturally grids of numbers. A neural network layer is a matrix multiplication plus a simple function, batches process many examples in one multiplication, and transformer attention is built from matrix products. GPUs accelerate exactly these operations.

Try Them Yourself

• Matrix Multiplication Calculator: run a tiny neural network layer
• Matrix Transpose Calculator: used in gradients and attention
• Matrix Addition Calculator: adding bias and updates
• Matrix Trace Calculator: a common matrix summary
• Calculus Formulas: gradients behind training
• The Math Behind Neural Networks: how layers learn

Multiply a 3 × 2 matrix of inputs by a 2 × 2 weight matrix using the calculator. You've just computed a neural network layer for a batch of three examples.