What Does "Attention" Mean in AI? The Mathematics Explained

Read this sentence: "The animal didn't cross the street because it was too tired." What does "it" refer to? You know instantly: the animal. Change "tired" to "wide," and "it" now means the street.

Resolving that requires looking back at other words and deciding which ones matter. In AI, the mechanism that does this is called attention, and it's the core of the transformer architecture behind today's large language models.

Despite the name, there's nothing psychological about it. Attention is a weighted average, where the weights come from dot products. Let's work through it with real numbers.

Attention Is Mostly Averaging

"Attention" sounds like the model is focusing its mind. Mathematically, each word builds a new version of itself by taking a weighted average of information from other words. The model's learned skill is choosing good weights.

The whole mechanism fits in one line:

Attention(Q, K, V) = softmax(QKᵀ / √d) V

The Three Roles: Query, Key, Value

Every token's vector is transformed three ways, by three learned weight matrices:

• Query (Q): what this token is looking for
• Key (K): what this token offers, used for matching
• Value (V): the information this token passes along if selected

A helpful analogy is a library search. Your query is compared against each book's catalog entry (the key), and you take home a blend of the books' contents (the values), weighted by how well they matched.

Step 1: Score Every Pair With a Dot Product

The match between a query and a key is their dot product. A bigger dot product means the vectors point in more similar directions.

Let's use tiny 2-dimensional vectors. One token's query is q = (1, 0). Three tokens have keys:

k₁ = (0.9, 0.1)   →   q · k₁ = 0.9
k₂ = (0.2, 0.8)   →   q · k₂ = 0.2
k₃ = (−0.5, 0.5)  →   q · k₃ = −0.5

For all tokens at once, that's the matrix product QKᵀ. Check it with the matrix multiplication calculator.

Step 2: Scale by √d

Divide by the square root of the vector dimension d. Here d = 2, so √2 ≈ 1.414:

0.9 / 1.414 ≈ 0.636
0.2 / 1.414 ≈ 0.141
−0.5 / 1.414 ≈ −0.354

Why? If each component of the query and key vectors has variance 1, their dot product has variance d. Real models use d = 64 or more, so dot products can become large. Large inputs push softmax into giving nearly all weight to one token, where gradients become tiny and learning stalls. Dividing by √d keeps the scores at a stable scale. See square roots on the square roots list.

Step 3: Softmax Turns Scores Into Weights

weightᵢ = e^(scoreᵢ) / Σⱼ e^(scoreⱼ)

For our scores:

The weights are positive and sum to 1. Token 1 gets about half the attention because its key best matches the query.

Step 4: Blend the Values

Finally, the output is the weighted sum of the value vectors:

output = 0.505 × v₁ + 0.308 × v₂ + 0.188 × v₃

The token's new representation now contains information pulled mostly from token 1, with some from the others. In the "animal … it" sentence, a well-trained model gives "it" a high weight on "animal."

Multi-Head Attention

One set of weights captures one kind of relationship. The original transformer ran 8 attention heads in parallel, each with its own Q, K and V matrices. With a model dimension of 512, each head used 64-dimensional vectors.

Different heads can specialize: one might track which noun a pronoun refers to, another nearby words, another sentence structure. Their outputs are concatenated and combined with another matrix.

An Insider Reference: From Translation to Transformers

Attention for neural networks was introduced by Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio in a 2014 paper on machine translation. Their system learned which source words to look at while producing each translated word, alongside a recurrent neural network.

In 2017, Ashish Vaswani and colleagues at Google published "Attention Is All You Need." Their bold idea was to remove recurrence entirely and build the whole model from attention and simple feed-forward layers. Because every token can be processed in parallel, training became dramatically more efficient on GPUs. That architecture, the transformer, underlies today's large language models.

Where Word Order Comes From

Attention by itself ignores order: shuffling the tokens would just shuffle the outputs. The original transformer added positional encodings built from sine and cosine waves of different frequencies:

PE(pos, 2i)   = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Each position gets a unique pattern. Explore the waves with the sine calculator.

The Cost: Quadratic Growth

Every token compares itself with every other token, so attention builds an n × n score matrix. Doubling the length of the text quadruples the work. A 100,000-token context means about 10 billion pairwise scores per head per layer.

That's why long-context models rely on engineering advances. FlashAttention, introduced by Tri Dao and colleagues in 2022, reorganizes the calculation to use GPU memory far more efficiently without changing the result.

Two Concepts Worth Knowing

Weighted Average

A weighted average multiplies each value by a weight and adds them, with weights summing to 1. Attention outputs are weighted averages of value vectors.

Dot Product

The dot product of two vectors measures how aligned they are. In attention, it scores how relevant each key is to a query. See the cosine calculator for the angle view.

Quick Answer: What Is Attention in AI?

Attention lets each token in a sequence gather information from other tokens. Each token has query, key and value vectors. Dot products between queries and keys, scaled by √d and passed through softmax, give weights that sum to 1. The output is the weighted average of value vectors: softmax(QKᵀ/√d)V.

Try Them Yourself

• Matrix Multiplication Calculator: compute QKᵀ
• Matrix Transpose Calculator: build Kᵀ
• Square Roots List: scaling by √d
• Sine Calculator: positional encoding waves
• Logarithm Calculator: exponentials behind softmax
• Cosine Similarity: The Math Behind Semantic Search: dot products as similarity

Change the query in our example to (0, 1) and recompute the attention weights. Which token wins now, and why?