Probability: The Mathematics Behind Language Models

In 1951, Claude Shannon ran an experiment. He showed people a passage of English text one letter at a time and asked them to guess the next letter. From how often they guessed right, he estimated that English carries only about 0.6 to 1.3 bits of information per letter, far below the 4.7 bits you'd need if all 26 letters were equally likely.

In other words, English is highly predictable. That predictability is exactly what a language model captures. Strip away the neural networks, and a language model is a precise mathematical object: a probability distribution over text.

Always Choosing the Most Likely Word Produces Bad Writing

If a model knows the probability of every next word, you'd think the best output comes from always picking the most probable one.

It doesn't. In 2019, Ari Holtzman and colleagues published "The Curious Case of Neural Text Degeneration." They showed that always taking the top choice, or searching for the single most probable sentence, tends to produce repetitive, bland loops. Human writing is less predictable than that. Good generation requires sampling, deliberately sometimes choosing less likely words.

Step 1: The Chain Rule of Probability

What's the probability of the sentence "the cat sat"? Break it into steps:

P(the, cat, sat) = P(the) × P(cat | the) × P(sat | the, cat)

This is the chain rule of probability, and it's exact. Every language model uses it. The hard part is estimating each conditional probability, P(next word | everything before it).

Step 2: The Simplest Model, Counting Pairs

A bigram model approximates by looking only at the previous word:

P(word | previous) ≈ count(previous, word) / count(previous)

Take this tiny training text:

the cat sat on the mat . the cat ate . the dog sat on the rug .

"the" appears 5 times. It's followed by "cat" twice, and by "dog," "mat" and "rug" once each. So:

This is how early n-gram language models worked, using counts of 2, 3 or more words from huge text collections.

Step 3: The Zero Problem

What's P(sat | dog)? "Dog" is followed only by "sat" in our text, so that's fine. But what about P(ran | dog)? We never saw it, so its count is zero, and any sentence containing "dog ran" gets probability 0, even though it's perfectly reasonable.

The classic fix is smoothing. Add-one (Laplace) smoothing adds 1 to every count:

P(word | previous) = (count + 1) / (count(previous) + V)

Where V is the vocabulary size. With V = 9 in our example, an unseen pair after "the" gets (0 + 1)/(5 + 9) ≈ 0.071 instead of 0. More refined smoothing methods followed, but the deeper solution came from neural networks, which generalize from similar words instead of just counting exact matches.

Step 4: Neural Language Models

Modern models replace counts with a neural network that reads the whole context and outputs a score for every token. Softmax converts scores into probabilities:

P(token i | context) = e^(zᵢ) / Σⱼ e^(zⱼ)

Because words become vectors, the model learns that "dog ran" is plausible from seeing "cat ran," "horse ran" and "dog walked." See The Mathematics Behind Large Language Models.

Step 5: Measuring Quality With Perplexity

How good is a language model? Measure how surprised it is by real text it hasn't seen.

Cross-entropy averages the negative log probability of each actual token:

H = −(1/N) Σ log P(actual token)

Perplexity converts that into a more intuitive number:

Perplexity = e^H

A perplexity of 20 means the model is, on average, as uncertain as if it were choosing uniformly among 20 options. Lower is better. Early n-gram models on standard benchmarks often had perplexities in the hundreds; modern neural models reach far lower values. Try natural logs with the logarithm calculator.

Step 6: Sampling Strategies

Once a model produces a distribution, how do you pick a token?

• Greedy: always pick the most probable. Deterministic, but prone to repetition.
• Temperature: divide scores by T before softmax. T < 1 sharpens, T > 1 flattens.
• Top-k sampling: keep only the k most likely tokens and sample among them. Angela Fan and colleagues used it for story generation in 2018.
• Top-p (nucleus) sampling: Holtzman's team proposed keeping the smallest set of tokens whose probabilities add up to at least p (like 0.9), then sampling from that set. It adapts automatically: when the model is confident, the set is small; when it's uncertain, it's larger.

An Insider Reference: Shannon's Guessing Game

Shannon's 1951 paper, "Prediction and Entropy of Printed English," didn't just estimate information per letter. In his 1948 paper, he'd also generated text by sampling from letter and word statistics. His word-pair approximation produced lines like:

THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER THAT THE CHARACTER OF THIS POINT IS THEREFORE ANOTHER METHOD

It's nonsense, but it sounds eerily like English. Today's models follow the same principle, estimating the probability of the next piece of text, with vastly better estimates.

Two Concepts Worth Knowing

Conditional Probability

Conditional probability P(A | B) is the probability of A given that B has happened. A language model is a machine for estimating P(next token | previous tokens). See the statistics formulas.

Entropy

Entropy measures average uncertainty in bits: H = −Σ p log₂ p. A fair coin has 1 bit. Predictable text has low entropy per token, which is what makes compression and prediction possible.

Quick Answer: How Do Language Models Use Probability?

A language model estimates the probability of each possible next token given the previous text, and multiplies those conditional probabilities to score whole sentences (the chain rule). Early models counted word sequences; modern neural models compute probabilities with softmax. Text is generated by sampling, often with temperature, top-k or top-p.

Try Them Yourself

• Logarithm Calculator: compute cross-entropy and perplexity
• Statistics Formulas: conditional probability rules
• Random Number Generator: sample from your own bigram model
• Division Tables: turn counts into probabilities
• Base-2 Logarithm Table: entropy in bits
• How Probability Powers Artificial Intelligence: probability across AI

Build a bigram table from a paragraph you like, then generate a "sentence" by rolling dice weighted by the probabilities. You'll have recreated Shannon's 1948 experiment.