Math Tools Math Tools
Share:
← Back to Blog

The Mathematics Behind Large Language Models

The Mathematics Behind Large Language Models

By Math Tools ·

The Mathematics Behind Large Language Models

In 2022, researchers at DeepMind trained a language model called Chinchilla with 70 billion parameters. It outperformed their earlier model Gopher, which had 280 billion, four times as many.

Chinchilla wasn't built from a new kind of math. It was trained on about four times more text for a similar computing budget. The result reshaped how the industry builds AI, and it came from careful mathematical analysis of how model size, data and compute trade off.

Large language models (LLMs) can seem like magic. Underneath, they're a pipeline of well-understood mathematics. Here's how the pieces fit.

An LLM Never Chooses a Word, It Computes a Probability Distribution

When a chatbot writes a sentence, it looks like it's picking words. Mathematically, at each step it outputs a probability for every token in its vocabulary, often 50,000 to 200,000 options. Something else, a sampling rule, then picks one.

Everything the model "knows" is encoded in how it shapes those probability distributions.

Step 1: Text Becomes Tokens

Models don't read letters or whole words. They read tokens, chunks produced by algorithms like byte-pair encoding (BPE), which Rico Sennrich and colleagues adapted for neural translation in 2016. BPE starts from characters and repeatedly merges the most frequent adjacent pair into a new token.

Common words become single tokens; rare words split into pieces. In English, one token averages roughly three-quarters of a word. Each token gets an integer ID.

Step 2: Tokens Become Vectors

Each token ID looks up a row in an embedding matrix, turning it into a vector of hundreds or thousands of numbers. If the vocabulary has V tokens and each vector has d dimensions, the embedding matrix is V × d.

These vectors are learned so that tokens used in similar contexts end up close together in space. See What Is a Vector? The Math Behind AI Embeddings.

Step 3: Transformer Layers

The heart of an LLM is a stack of transformer layers, from the 2017 paper "Attention Is All You Need" by Ashish Vaswani and seven co-authors at Google. Each layer has two main parts:

Self-attention lets every token gather information from earlier tokens:

Attention(Q, K, V) = softmax(QKᵀ / √d) V

A feed-forward network transforms each token's vector independently:

FFN(x) = W₂ · f(W₁x + b₁) + b₂

Both are wrapped with residual connections (adding the input back to the output) and normalization. Stacking dozens of layers lets the model build increasingly abstract representations. Every step is built on matrix multiplication.

Step 4: Scores Become Probabilities

The final vector is multiplied by an output matrix to produce one score, a logit, per vocabulary token. The softmax function turns logits into probabilities:

P(token i) = e^(zᵢ) / Σⱼ e^(zⱼ)

All probabilities are positive and sum to 1. Explore the exponential and log functions with the logarithm calculator.

Step 5: Training Minimizes Surprise

During training, the model predicts every next token in huge amounts of text. The loss is cross-entropy, the negative log of the probability given to the correct token:

Loss = −(1/N) Σ log P(correct token)

If a model assigns the correct tokens probabilities 0.5, 0.25, 0.8 and 0.1, the average loss is about 1.15. A related number, perplexity, is e raised to that loss, here about 3.16. It's as if the model were choosing among about 3 equally likely options at each step. Lower is better.

Gradient descent and backpropagation then adjust billions of weights to lower the loss. See Gradient Descent Explained With Simple Mathematics.

Step 6: Generating Text

To write, the model repeatedly predicts a distribution, samples a token, appends it, and runs again. Settings like temperature reshape the distribution:

P(token i) = e^(zᵢ / T) / Σⱼ e^(zⱼ / T)

Low T makes output focused and predictable; high T makes it more varied.

An Insider Reference: Scaling Laws

In 2020, OpenAI researchers led by Jared Kaplan published "Scaling Laws for Neural Language Models." They found that loss falls as a power law as you increase parameters, data or compute:

L(N) ≈ (N_c / N)^α

A widely used rule of thumb from this line of work estimates training compute as:

Compute ≈ 6 × parameters × training tokens

For GPT-3 (175 billion parameters, about 300 billion tokens), that's about 3 × 10²³ floating-point operations.

In 2022, Jordan Hoffmann and colleagues at DeepMind revisited the question in the Chinchilla paper. They concluded that for a fixed compute budget, many models were too big and undertrained, and that data should grow in proportion to parameters, roughly 20 training tokens per parameter. Power laws appear as straight lines on a log-log plot; try the logs with the logarithm calculator.

Two Concepts Worth Knowing

Softmax

Softmax converts a list of real numbers into a probability distribution by exponentiating and normalizing. It exaggerates differences, so the highest score gets most of the probability.

Power Law

A power law has the form y = axᵏ. Doubling x always multiplies y by the same factor. LLM scaling laws are power laws, which is why progress needs exponentially more resources for steady gains.

Quick Answer: How Do Large Language Models Work Mathematically?

LLMs split text into tokens, convert each token into a vector, and pass the vectors through transformer layers of attention and feed-forward matrix operations. The output is turned into a probability for every possible next token with softmax. Training minimizes cross-entropy loss using gradient descent across billions of parameters.

Try Them Yourself

Take a four-word sentence and assign your own probability to each word given the previous ones. Compute the average −ln(p). You've just measured your own "language model loss."

Next → Probability: The Mathematics Behind Language Models