How Large Language Models Predict the Next Word

July 17, 2026

A friend once asked me whether the chatbot was actually “thinking” about her question. The honest answer: not the way she meant. What’s happening underneath is narrower and stranger than that — the model is repeatedly predicting the single most likely next chunk of text, called a token, based on everything written so far.

It Doesn’t See Words, It Sees Tokens

Text gets broken into tokens before a model ever sees it — sometimes whole words, sometimes word fragments. The model doesn’t reason in English; it reasons in these fragments, converting them into long lists of numbers it can do math on.

One Tiny Prediction at a Time, Thousands of Times Over

At each step, the model looks at every token so far and calculates a probability for what comes next, across its entire vocabulary. It picks one, adds it to the text, and repeats. A whole paragraph is really thousands of these tiny predictions chained together.

Nothing Is Recalled — It’s Regenerated Fresh

Those probabilities come from patterns learned during training, when the model was shown enormous amounts of text and adjusted itself to get better at guessing the next token. Nothing is “recalled” the way a search engine recalls a webpage; it’s regenerated fresh each time.

Fluency and Accuracy Aren’t the Same Thing

This next-token mechanism is why these models write fluently about almost anything, and why they can state a wrong fact with total confidence — fluency and accuracy are separate properties. A sentence can be grammatically perfect and statistically likely while being factually wrong.

My Takeaway

Treat fluent-sounding output as a draft to verify, not a verdict, especially for facts, numbers, or anything with real consequences. I remind myself of this every time a model sounds a little too sure of itself.