Tokenization: How LLMs Actually Read Text

Before a language model reads a single word of your prompt, the text is broken into tokens — chunks that are usually pieces of words, not whole words and not individual letters. The model only ever sees these tokens, as a sequence of integer IDs. It is an invisible step most people never think about, and it quietly explains a surprising amount of how LLMs behave.

Your bill is counted in tokens. Your context window is measured in tokens. Several of the model's most famous failures come from tokens. And the reason a sentence in Hindi or Japanese can cost two or three times more than the same sentence in English is tokens. Once you see how text becomes tokens, a lot of mysterious LLM behavior stops being mysterious.

Words in, tokens out

Tokenizers split text using an algorithm — usually byte-pair encoding — that starts from characters and merges the most frequent pairs into larger tokens. Common words become a single token; rare words, names, and code get split into several pieces. The result is a vocabulary of sub-word units the model was trained on.

• A token is roughly four characters of English — about three-quarters of a word on average, but it varies wildly by content.
• Common words are one token; rare ones fragment. The splits into a single token; a long technical term or an unusual name splits into many.
• Whitespace and punctuation count too. A leading space is often part of the token, which is why the and the can tokenize differently.
• The tokenizer is fixed per model. It is chosen and frozen at training time, so you cannot change how a given model chunks your text.

Why it explains so much

Tokenization is behind a cluster of behaviors that otherwise look random. Models struggle to count the letters in a word or reverse it because they never see the letters — they see a few opaque chunks. They fumble certain arithmetic because numbers get split into inconsistent pieces. Character-level tasks are hard for the same reason: the unit the model reasons over is the token, not the character.

It also has a direct cost. Code and structured formats tokenize inefficiently — lots of punctuation and rare symbols means more tokens per line. And languages other than English, especially ones with non-Latin scripts, were under-represented in most tokenizers, so the same meaning takes far more tokens — a real tax on both your bill and your context budget for multilingual apps.

Seeing the tokens, in code

The fastest way to build intuition is to tokenize something yourself and look at the pieces:

# Tokens, not words: see how the model actually chops up your text.
import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Tokenization is sneaky.")     # -> a list of integer token IDs
print(len(tokens), tokens)                          # the count is what you pay for
print([enc.decode([t]) for t in tokens])            # see the pieces: words, sub-words, spaces

# "strawberry" can split into "str", "aw", "berry" — which is why a model miscounts its r's.

Run that on your own prompts — especially code, JSON, and non-English text — and the abstract token count turns concrete. You will see exactly where your tokens (and your money) are going.

The model does not read characters or words; it reads tokens. Almost every time an LLM does something strangely dumb with letters, numbers, or spelling, tokenization is the reason.

What to do with this

• Budget in tokens, not words. Count tokens to estimate cost and fit context; the word-to-token ratio shifts with code, numbers, and language.
• Mind the multilingual tax. Non-English and non-Latin text can cost several times more tokens — factor it into pricing and context limits.
• Do not ask the model to count letters. Character-level tasks fight the tokenizer; use code or tools for spelling, counting, and exact string work.
• Watch code and structured data. Punctuation-heavy formats burn tokens fast, which matters for long files and big JSON payloads.
• Inspect when costs surprise you. If a prompt is pricier than expected, tokenize it — you will usually find a few rare strings exploding into many tokens.

The bottom line

Tokenization is the hidden layer between your text and the model, and it shapes cost, context, and capability. Models read sub-word tokens, not words or letters — which is why you pay per token, why context limits bite, why non-English text costs more, and why LLMs are oddly bad at spelling and counting. None of that is random once you see the tokens.

You cannot change a model's tokenizer, but understanding it makes you a better builder: you budget accurately, you stop asking the model to do character-level work it cannot, and you design prompts and pick models with the token tax in mind. Look at the tokens once, and you will never read an LLM the same way again.