Math Tools Math Tools
Share:
← Back to Blog

Linear Algebra for Machine Learning: A Practical Introduction

Linear Algebra for Machine Learning: A Practical Introduction

By Math Tools ·

Linear Algebra for Machine Learning: A Practical Introduction

MIT professor Gilbert Strang taught linear algebra for more than 60 years. His course, 18.06, became one of the most-watched offerings on MIT OpenCourseWare, reaching millions of learners online. In 2019 he published Linear Algebra and Learning from Data, a book that connects the subject directly to machine learning.

Why so much interest? Because linear algebra is the language of machine learning. Data, models and training are all written in vectors and matrices. The good news: you don't need all of linear algebra. A handful of ideas cover most practical needs.

Most of ML Is Linear, Plus a Little Bit of Bend

Neural networks are famous for modeling complex, nonlinear patterns. Yet the vast majority of their computation is linear: matrix multiplications and additions. The nonlinear parts, activation functions like ReLU, are tiny and simple.

Master the linear parts, and you understand most of what a model actually computes.

1. Vectors and Norms

A vector is a list of numbers, like a data point (3 bedrooms, 2 baths, 1800 sq ft).

Its size is measured by a norm:

L2 norm:  ‖v‖₂ = √(v₁² + v₂² + … + vₙ²)
L1 norm:  ‖v‖₁ = |v₁| + |v₂| + … + |vₙ|

For v = (3, −4): ‖v‖₂ = 5, ‖v‖₁ = 7.

ML use: regularization. Adding λ‖w‖₂² to the loss (ridge) keeps weights small. Adding λ‖w‖₁ (lasso, introduced by Robert Tibshirani in 1996) pushes many weights to exactly zero, selecting features automatically. Check vector lengths with the right triangle calculator.

2. The Dot Product

a · b = a₁b₁ + a₂b₂ + … + aₙbₙ = ‖a‖ ‖b‖ cos θ

ML use: a linear model's prediction is a dot product, ŷ = w · x. Attention scores and cosine similarity in search are dot products too.

3. Matrix Multiplication

A matrix transforms vectors. Multiplying an (m × n) matrix by an (n × p) matrix gives an (m × p) matrix, where each entry is a dot product of a row and a column.

ML use: a neural network layer computes Wx + b. With a whole batch stacked into a matrix X, one multiplication computes predictions for every example. Practice with the matrix multiplication calculator.

4. Transpose, Identity and Inverse

  • Transpose Aᵀ flips rows and columns. See the matrix transpose calculator.
  • Identity I has 1s on the diagonal: AI = A.
  • Inverse A⁻¹ undoes A: AA⁻¹ = I. It exists only if A doesn't collapse any direction.

ML use: linear regression. The best-fit weights satisfy the normal equation:

w = (XᵀX)⁻¹ Xᵀy

For a line through the origin fitting points (1, 2), (2, 4.1), (3, 5.9), this reduces to w = Σxy / Σx² = 27.9 / 14 ≈ 1.993. In practice, libraries solve the system more stably without computing the inverse directly.

5. Eigenvectors and Eigenvalues

An eigenvector of a matrix A is a direction that A only stretches, without turning:

Av = λv

λ is the eigenvalue, the stretch factor. For:

A = [ 2  1 ]
    [ 1  2 ]

the eigenvectors are (1, 1) with eigenvalue 3, and (1, −1) with eigenvalue 1. Check: A·(1, 1) = (3, 3) = 3·(1, 1). ✓

ML use: PCA. Principal component analysis, introduced by Karl Pearson in 1901 and developed by Harold Hotelling in 1933, finds the eigenvectors of a dataset's covariance matrix. The eigenvector with the largest eigenvalue is the direction of greatest variation. Keeping the top few compresses data while preserving most of its structure. The matrix trace calculator is a nice side check: a matrix's trace equals the sum of its eigenvalues (2 + 2 = 3 + 1).

6. Singular Value Decomposition

SVD factors any matrix into three simpler ones:

A = U Σ Vᵀ

U and V are rotations; Σ is a diagonal matrix of singular values, sorted from largest to smallest. In 1936, Carl Eckart and Gale Young proved that keeping only the top k singular values gives the best possible rank-k approximation of A.

ML use: compressing images and models, recommendation systems, noise reduction, and low-rank fine-tuning methods like LoRA.

7. Rank

The rank of a matrix is the number of truly independent rows (or columns). A 1,000 × 1,000 matrix of rank 10 holds far less information than its million entries suggest.

ML use: if features are redundant (like height in inches and height in centimeters), XᵀX has low rank and can't be inverted. Regularization or dimensionality reduction fixes the problem.

An Insider Reference: Strang's Big Picture

Gilbert Strang famously organized linear algebra around what he called the four fundamental subspaces of a matrix: the column space, row space, null space and left null space. They answer practical questions: Which outputs can the model produce? Which input changes have no effect at all?

His teaching emphasized understanding matrices as transformations you can picture, not just grids of numbers to compute. That perspective is exactly what helps when debugging a model: asking what each matrix does to the data flowing through it.

Two Concepts Worth Knowing

Covariance Matrix

A covariance matrix records how each pair of features varies together. Its eigenvectors reveal the main directions of variation in a dataset. See the statistics formulas.

Linear Transformation

A linear transformation preserves addition and scaling and can always be represented by a matrix. Every layer of a neural network, before its activation function, is one.

Quick Answer: What Linear Algebra Do You Need for Machine Learning?

The most important linear algebra for machine learning covers vectors and norms, dot products, matrix multiplication, transposes and inverses (for linear regression), eigenvectors and eigenvalues (for PCA), singular value decomposition (for compression and recommendations), and matrix rank.

Try Them Yourself

Multiply the matrix [[2, 1], [1, 2]] by the vectors (1, 1), (1, −1) and (1, 0). Only two of the results point the same way as their input. Those two are the eigenvectors.

Next → Why Matrices Are Everywhere in Machine Learning