How AI picks the next word: softmax and logarithms

AI chat assistants are now part of everyday life, and many people wonder how they “know” what to write. There is no magic: behind every answer sits a huge amount of arithmetic, and the core ideas are the same ones you meet in high school and early college. Conditional probability, exponential functions, logarithms and scientific notation. This article uses the topic as an excuse to review exactly those four tools, with small numbers made up for the example.

The core idea: predicting the next word

A language model takes a piece of text and answers one precise question: how likely is each word in the vocabulary to come next? After “the sky is”, words like “blue” or “cloudy” should get high probability, and “screwdriver” should get almost none.

Mathematically this is a conditional probability: the probability that the next word is $w$ given the preceding context, written $P(w \mid \text{context})$ . For it to be a valid distribution, every value must lie between 0 and 1 and the values over all possible words must sum to exactly 1.

Softmax: from arbitrary scores to probabilities

Internally the model produces a score for each word, any real number, possibly negative or larger than 1. To turn scores into probabilities it applies softmax:

P(w_i)=\frac{e^{z_i}}{\sum_{j} e^{z_j}}

Softmax: the exponential makes everything positive and dividing by the sum makes the total 1.

The exponential function does the heavy lifting. Since $e^{x}$ is always positive, no probability comes out negative. Since it is increasing, a higher score still means a higher probability. It also amplifies gaps: a score just 1 higher becomes about 2.718 times larger, because $e^{z+1}/e^{z}=e$ . Dividing by the total is the same normalization you use to turn counts into percentages.

The exponential is always positive and increasing: each extra unit of score multiplies the value by about 2.718. Order is kept and differences are amplified.

Worked example: three candidate words

Suppose a toy vocabulary with three candidates after “the sky is”: “blue” with score 2, “cloudy” with score 1 and “screwdriver” with score 0.

Computing softmax step by step

$\displaystyle e^{2}\approx 7.389,\quad e^{1}\approx 2.718,\quad e^{0}=1$ Exponentiate each score, using e ≈ 2.718.
$\displaystyle 7.389+2.718+1=11.107$ Add them up to get the denominator.
$\displaystyle \frac{e^{2}}{e^{2}+e^{1}+e^{0}}\approx\frac{7.389}{11.107}\approx 0.665$ Divide each exponential by the sum. For "blue":
$\displaystyle \frac{2.718}{11.107}\approx 0.245,\qquad \frac{1}{11.107}\approx 0.090$ Repeat for the other two words.
$\displaystyle 0.665+0.245+0.090=1.000$ Check: the three probabilities must add to 1 (any tiny gap is rounding).

So roughly 66.5% for “blue”, 24.5% for “cloudy” and 9% for “screwdriver”. A score of 0 did not become probability 0, because $e^{0}=1$ . That is why real models give absurd words a small but never exactly zero probability.

Softmax of scores 2, 1 and 0: about 66.5% for blue, 24.5% for cloudy and 9% for screwdriver. No bar reaches zero.

The probability of a whole sentence

Generating a sentence repeats the process word by word. By the chain rule of probability, the chance of the full sequence is the product of each step’s chance, conditioned on everything before it:

P(w_1,w_2,\dots,w_n)=P(w_1)\cdot P(w_2\mid w_1)\cdots P(w_n\mid w_1,\dots,w_{n-1})

Chain rule: each factor depends on all previous words.

Suppose a four-word sentence where the model gave 0.5, 0.4, 0.2 and 0.1 at each step. The sentence probability is $0.5 \cdot 0.4 \cdot 0.2 \cdot 0.1 = 0.004$ , or 0.4%. Now picture a 50-word answer where every step has probability 0.1: the product is $0.1^{50}=10^{-50}$ . Computers store numbers with limited precision, and values that small can round to zero, a problem called underflow.

Running probability of the sentence: 1, then 0.5, 0.2, 0.04 and 0.004 after multiplying by 0.5, 0.4, 0.2 and 0.1. Just four words and the product collapses.

Why logarithms come in

The fix is to work with log-probabilities, using $\log(a\cdot b)=\log a+\log b$ . Taking logs of the chain rule turns the product into a sum:

\log P(w_1,\dots,w_n)=\sum_{k=1}^{n}\log P(w_k\mid w_1,\dots,w_{k-1})

Sentence log-probability: the sum of each step's log.

Checking the four-word example (base-10 logs)

$\displaystyle \log 0.5\approx -0.301,\ \log 0.4\approx -0.398,\ \log 0.2\approx -0.699,\ \log 0.1=-1$ Take the log of each factor.
$\displaystyle -0.301-0.398-0.699-1=-2.398$ Add them.
$\displaystyle 10^{-2.398}\approx 0.004$ Undo the log by raising 10 to the result and compare with the direct product.

Both routes give 0.004. For the 50-word case, instead of storing $10^{-50}$ we store just $-50$ . Because log is increasing, the sentence with the highest log-probability is also the most probable one. Real models usually use the natural log, but the logic is the same in any base.

Orders of magnitude and scientific notation

Model sizes are described with phrases like “billions of parameters”. Read them with scientific notation: a billion is $10^{9}$ , a trillion is $10^{12}$ . The order of magnitude is roughly the integer part of the base-10 log. If a hypothetical model A has $10^{9}$ parameters and model B has $10^{11}$ , B is two orders of magnitude larger, meaning 100 times larger, not “twice as large”.

Common mistakes

Thinking softmax is just dividing by the sum. Without the exponential, negative scores would give negative “probabilities”.
Adding probabilities of consecutive steps. Sequential events combine by multiplication.
Forgetting that log-probabilities are negative. Probabilities are at most 1, so their logs are at most 0; closer to 0 means more likely.
Mixing up orders of magnitude. Going from $10^{9}$ to $10^{12}$ is 1,000 times more, not 3 times.
Treating high probability as certainty. 66.5% still leaves 33.5% for other words.

Frequently asked questions

Does softmax always return values between 0 and 1?

Yes. Each term is positive and divided by a sum that contains it, so with more than one word it lies strictly between 0 and 1, and all of them add up to 1.

Why base e for the exponential?

Any base above 1 would work. Base e is preferred because its derivative is itself, which keeps training calculations simple.

Does taking logs change which sentence is most likely?

No. The logarithm is increasing, so it preserves order.

Do I need calculus to follow this?

Not for what is shown here. Conditional probability and the rules of exponents and logarithms are enough.