Vectors, Matrices, and Tensors Explained
A single small color photo, 224 pixels wide and 224 tall, becomes 150,528 numbers when an AI model reads it: 224 × 224 pixels, times 3 color channels. A batch of 32 photos is nearly 5 million numbers, arranged in a precise four-dimensional block.
That block is a tensor. The word sounds intimidating, and it's even in the names of major AI frameworks like TensorFlow. But the idea is simple: vectors and matrices, extended to more dimensions.
"Dimension" Means Two Different Things
In AI, you'll hear that an embedding has "768 dimensions" and also that an image batch is "a 4-dimensional tensor." These use the word in different senses:
- A vector with 768 dimensions is a single list of 768 numbers. It has one axis.
- A 4-dimensional tensor has four axes, like batch, height, width and color channel.
Mixing up number of entries with number of axes is one of the most common sources of confusion when learning machine learning.
The Ladder
| Object | Axes | Example | Shape |
|---|---|---|---|
| Scalar | 0 | a temperature: 21.5 | () |
| Vector | 1 | a word embedding | (768,) |
| Matrix | 2 | a grayscale image | (28, 28) |
| 3D tensor | 3 | a color image | (224, 224, 3) |
| 4D tensor | 4 | a batch of color images | (32, 224, 224, 3) |
Each step adds one more index needed to locate a single number. A matrix entry needs a row and a column. A pixel in a batch needs image number, row, column and channel.
Scalars
A scalar is just one number: a learning rate of 0.001, a loss of 2.3, a temperature. Multiplying a vector or matrix by a scalar scales every entry. Try it with the scalar multiplication calculator.
Vectors
A vector is an ordered list of numbers, like (0.2, −1.3, 4.0). In AI, vectors represent:
- One data example's features: (bedrooms, bathrooms, square feet)
- A word's embedding
- A single layer's bias values
Key operations are addition, scaling and the dot product, which measures how aligned two vectors are.
Matrices
A matrix is a grid of numbers with rows and columns. Its shape is written (rows, columns). In AI, matrices represent:
- A dataset: one row per example, one column per feature
- A layer's weights: a (4096, 4096) matrix has 16,777,216 learned numbers
- Attention scores: how much each token attends to each other token
Matrix multiplication is the workhorse. An (m × n) matrix times an (n × p) matrix gives an (m × p) matrix. The inner numbers must match. Practice with the matrix multiplication calculator, and swap rows with columns using the transpose calculator.
Tensors
In machine learning, a tensor is simply a multi-dimensional array: the general case of which scalars, vectors and matrices are special cases.
Common shapes:
- Text batch: (batch, sequence length, embedding size), like (8, 1024, 768)
- Image batch: (batch, height, width, channels)
- Video: (batch, frames, height, width, channels)
Broadcasting lets frameworks combine tensors of different shapes. Adding a bias vector of shape (768,) to a tensor of shape (8, 1024, 768) adds it to every token of every example, without copying the vector 8,192 times.
An Insider Reference: Tensors Came From Physics
The mathematical concept of tensors was developed by Italian mathematicians Gregorio Ricci-Curbastro and his student Tullio Levi-Civita, who published their "absolute differential calculus" in 1900.
Albert Einstein struggled to learn it while developing general relativity. His friend, mathematician Marcel Grossmann, introduced him to the subject, and Einstein corresponded with Levi-Civita. The field equations of 1915 are written in tensor form.
In physics, a tensor is defined by how its components transform when you change coordinates. Machine learning uses the word more loosely, for any multi-dimensional array of numbers. Both uses share the core idea of organizing numbers by several indices. Google released TensorFlow in 2015, and Facebook released PyTorch in 2016, cementing the term in AI.
Estimating Memory From Shapes
Shapes tell you exactly how much memory data needs. Multiply all the dimensions, then multiply by bytes per number:
| Format | Bytes per number |
|---|---|
| 32-bit float | 4 |
| 16-bit float | 2 |
| 8-bit integer | 1 |
Example 1: A batch of 32 color images at 224 × 224:
32 × 224 × 224 × 3 × 4 bytes ≈ 19.3 MB
Example 2: A 7-billion-parameter model:
7,000,000,000 × 2 bytes (16-bit) = 14 GB
7,000,000,000 × 4 bytes (32-bit) = 28 GB
That's why quantization, storing weights with fewer bits, matters so much for running models on laptops and phones. See the binary to decimal converter for how bits become numbers.
Two Concepts Worth Knowing
Shape
A tensor's shape lists the size of each axis. Most bugs in AI code are shape mismatches, like multiplying a (3, 4) matrix by a (3, 4) matrix, which doesn't work because the inner dimensions (4 and 3) don't match.
Rank (Order)
In ML frameworks, a tensor's rank or order often means its number of axes: a matrix has rank 2. (In linear algebra, a matrix's rank means something else: the number of independent rows or columns.)
Quick Answer: What Is the Difference Between a Vector, a Matrix and a Tensor?
A scalar is a single number, a vector is a one-dimensional list of numbers, and a matrix is a two-dimensional grid with rows and columns. A tensor generalizes these to any number of axes. For example, a batch of color images is a four-dimensional tensor with batch, height, width and channel axes.
Try Them Yourself
- Matrix Multiplication Calculator: check inner-dimension matching
- Matrix Transpose Calculator: swap a matrix's axes
- Scalar Multiplication Calculator: scale every entry
- Matrix Addition Calculator: combine same-shape matrices
- Binary to Decimal Converter: how numbers are stored in bits
- Why Matrices Are Everywhere in Machine Learning: where matrices do the work
Look up the parameter count of an open AI model, then estimate its memory size at 16 bits and at 4 bits. You'll see immediately why quantization makes large models fit on smaller devices.