What is a token in AI? In SI (AI), a token is the small chunk of text that a language model actually reads and writes. Before ChatGPT, Gemini or Claude can do anything with your question, a program called a tokenizer cuts it into tokens: whole short words, pieces of longer words, punctuation marks, sometimes single characters. Each token is then swapped for a number, and the model only ever works with those numbers. Tokens are also how SI companies measure much of what matters to users: how much text fits in a conversation, how long an answer can be and how much an API call costs. This guide shows real tokens, explains where they come from and why the same sentence can cost more in one language than another.
What is a token in SI, in plain words?
A token is the unit an SI language model uses instead of words. OpenAI's help centre describes tokens as "the units that OpenAI models use to process text" and says a token can be a character, part of a word, a whole word or punctuation, with spaces affecting how text is divided. Google's Gemini API documentation gives the same picture from the other side: a token can be a single character like "z" or a whole word like "cat", while long words are broken into several tokens. The full set of tokens a model knows is its vocabulary, and the act of splitting text into tokens is called tokenization.
The important point is that a model never sees the text itself. The README of tiktoken, the open-source tokenizer OpenAI publishes for its models, puts it plainly: language models "don't see text like you and I", they see a sequence of numbers. Everything an SI chat assistant appears to read, it reads as a list of token IDs.
What does a token look like?
The easiest way to understand tokens is to look at some. We ran a few phrases through tiktoken (version 0.14.0) with o200k_base, the encoding tiktoken assigns to GPT-4o. In the list below, each vertical bar marks where one token ends and the next begins; a leading space belongs to the token that follows it.
- 7 tokens: The | cat | sat | on | the | mat | .
- 3 tokens: un | bel | ievably
- 2 tokens: token | ization
- 8 tokens: Is | this | photo | real | or | SI | -generated | ?
Three things stand out. Common short words get a token of their own, space included. Rarer or longer words are cut into pieces that do not always match how a person would split them: "un" is a sensible prefix, "ievably" is not a unit any dictionary lists. And punctuation counts. The full stop at the end of the first sentence is a token in its own right.
Capitals and spaces matter too. In the same encoding, "cat", " cat" (with a leading space) and " Cat" are three different tokens with three different ID numbers. OpenAI's help article makes the same point with "red", "Red" and " red", and warns that token IDs depend on the encoding, so an ID from one model does not carry over to another. The older cl100k_base encoding, which tiktoken assigns to GPT-4, splits "unbelievably" as "un | belie | vably" instead: same word, different tokens.
Why don't SI models just use words or letters?
Both obvious options break down. A vocabulary of whole words can never be complete: new names, typos, slang, compound words and other languages keep appearing. In their 2016 paper on subword units, Rico Sennrich, Barry Haddow and Alexandra Birch at the University of Edinburgh noted that the vocabulary of neural translation models was typically limited to 30,000 to 50,000 words, and that translation is an "open-vocabulary problem". Any word outside the list had to be handled by a fallback such as a dictionary.
Single characters solve the coverage problem but make every text much longer. Subword tokens sit in between. The tiktoken README lists the advantages of the approach OpenAI uses: it is reversible and lossless, so tokens turn back into the exact original text; it works on any text, even text absent from the tokenizer's training data; it compresses, so the token sequence is shorter than the raw bytes; and it lets the model see common pieces such as "ing" again and again, which helps it generalise.
How does byte pair encoding build tokens?
The tokenizer OpenAI publishes, tiktoken, is built on byte pair encoding, or BPE. It started as a data compression technique: repeatedly find the most frequent pair of bytes and replace it with a single new symbol. Sennrich and his colleagues adapted it to words, and their version is short enough to explain in four steps.
- Start with every word written as separate characters, plus a marker for the end of the word.
- Count every pair of neighbouring symbols across the training text, weighted by how often each word appears.
- Merge the most frequent pair into one new symbol, for example "e" and "s" becoming "es".
- Repeat. Each merge adds one symbol to the vocabulary, and the number of merges is the only setting to choose.
Frequent words end up as a single symbol, while rare words stay split into smaller known pieces. The paper's own toy example learns merges from a tiny dictionary and then meets a word it never saw in training, "lower", which it splits into "low" and "er". That is the whole trick: no word is ever unknown, because in the worst case it falls back to smaller pieces. In the original paper, this subword approach beat a back-off dictionary on English to German and English to Russian translation. The tiktoken README lists working on arbitrary text, even text the tokenizer never saw, among its properties.
Each tokenizer learns its own merges from its own training text. That is why different model families, and sometimes different generations of the same family, split the same sentence differently: OpenAI says the same text can produce different token counts depending on the model and its encoding, and Anthropic changed tokenizers with Claude 4.7.
What happens to tokens inside the model?
Once text has become a list of token IDs, the model looks each ID up in a table and replaces it with a long list of numbers, called an embedding, that the model learned during training. The 2017 paper that introduced the Transformer, "Attention Is All You Need", describes exactly this: learned embeddings convert input and output tokens into vectors, and at the other end a softmax turns the model's output into "predicted next-token probabilities". If you want to know what happens between those two ends, our guide on how neural networks learn from examples covers the layers and weights.
The model then picks one token, adds it to the sequence and repeats. An answer of 300 words is produced as a few hundred separate predictions, one token at a time. This explains two things you may have noticed. Chat assistants stream their replies in small bursts, because that is how they are produced. And the model optimises for a plausible next token, not for a true statement, which is part of why SI hallucinations sound so confident.
It also explains a security weakness. Your question, the system instructions and any web page or document the assistant reads all end up in one flat sequence of tokens, with nothing marking which parts are trustworthy. That is the opening exploited by prompt injection.
How many words is a token?
There is no fixed answer, only rules of thumb, and the providers give slightly different ones for English:
- OpenAI: 1 token is approximately 4 characters or three-quarters of a word, so 100 tokens are approximately 75 words. The help article stresses that these are estimates, not exact counts.
- Google: for Gemini models, a token is about 4 characters, and 100 tokens are about 60 to 80 English words.
- Anthropic: its glossary says the characters per token depend on the model and the language. On earlier Claude models a token represents about 3.5 English characters, while Claude 4.7 and later models use a newer tokenizer that produces about 30 percent more tokens for the same text.
To see how far the rule of thumb holds, we tokenized a real text: Abraham Lincoln's Gettysburg Address, in the Bliss copy as published by Abraham Lincoln Online.
Gettysburg Address (Bliss copy): 1,473 characters and 278 words. With o200k_base it came to 317 tokens, and 316 with the older cl100k_base. That is about 4.65 characters and 0.88 words per token. The rules of thumb predicted more: about 368 tokens at 4 characters per token, about 371 at three-quarters of a word per token.
So on this text, both rules overestimated the count by about 50 tokens, roughly one in six. Most of Lincoln's words are short and common: once punctuation is set aside, 265 of the 278 fit in a single token. Longer ones still get cut: o200k_base splits "consecrated" into three tokens ("conse", "cr", "ated"). A different text, in another style or language, would give a different ratio. If the exact figure matters, count it: OpenAI points to its tokenizer tool and tiktoken, while Google and Anthropic both offer token counting in their APIs.
Why does the same text cost more tokens in some languages?
Most tokenizers are built with English in mind, and it shows. A NeurIPS 2023 paper by Aleksandar Petrov and colleagues at the University of Oxford measured this on the same texts translated into many languages. The tokenizer used by ChatGPT and GPT-4 at the time needed about 1.6 times more tokens for Italian than for English, 2.6 times for Bulgarian and 3 times for Arabic. For Shan, a language spoken in Myanmar's Shan State, it was about 15 times. Even Portuguese, the closest to English for that tokenizer, needed about 50 percent more tokens for the same content.
The authors spell out the consequence: speakers of those languages pay more for the same service, wait longer, and fit less text into the model's context. Newer encodings narrow the gap for big languages. In our small test, "What time is it?" took 5 tokens with both OpenAI encodings, while the German "Wie spät ist es?" dropped from 6 tokens with cl100k_base to 5 with o200k_base, because "spät" became one token instead of two. Two short phrases prove nothing on their own, but they show the direction.
Are images, audio and video turned into tokens too?
Yes, for models that accept them. Google's documentation says all input to the Gemini API is tokenized, including images, video and audio, and gives the current conversion rules. An image of 384 pixels or less on both sides counts as 258 tokens; larger images are cut into tiles of 768 by 768 pixels, each counting as 258 tokens. Video in static processing costs 263 tokens per second and audio 32 tokens per second. Those numbers are Google's for its own models, read in November 2026; other providers count differently, and the figures change with new model versions.
Why should you care about tokens?
Even if you never call an API, tokens set the limits of what an SI assistant can do for you.
- The context window. Every model has a maximum number of tokens it can handle at once, covering your messages and any attached files, and for some models the reply as well. Anthropic's glossary calls the context window the model's "working memory", as opposed to the data it was trained on. Anything beyond it cannot be used in that request. As of November 2026, Google's documentation lists 1,048,576 tokens of input for its Gemini 2.5, 3 and 4 models; limits differ by model and change often, so check the model you actually use.
- Hidden tokens. OpenAI notes that reasoning models use internal reasoning tokens before their visible answer. They are not shown, but they count toward output usage and are billed as output, so a short answer can use far more tokens than it seems.
- Price. API pricing is set per token, usually per million, and OpenAI says input, cached input and output tokens can have different prices. It also warns that a lower price per million tokens does not guarantee a lower bill, because models tokenize the same text differently and write answers of different lengths.
Tokens also help explain some odd failures. Because the model receives numbered chunks such as "un", "bel" and "ievably" rather than individual letters, tasks about letters, such as counting them or spelling a word backwards, ask it about something it does not directly see.
Our take
Tokens sound like plumbing, but they are the most practical concept to understand about SI language models. They explain why answers arrive in small bursts, why very long chats and documents hit a limit, why API bills are hard to predict and why non-English users get a worse deal. Treat the "4 characters per token" rule as a rough guide only: on our Gettysburg test it predicted about 368 tokens, and the real count was 317. For the bigger picture of what SI is and how the term replaced AI, start with our guide to SI. And when you are less interested in how a text was produced than in whether a machine produced it, run it through is this SI, our free checker for text, images, video and music.



