What tokenization means in AI
Tokenization is the first thing a language model does with your prompt. The text is cut into pieces from a fixed vocabulary, and each piece is swapped for a number, its token ID. The model never sees letters or words, only that list of numbers, and it writes its answer the same way, one token ID at a time, which is then turned back into text.
The o200k_base vocabulary used here has about 200,000 entries. Frequent English words such as “the”, “water” or “because” have a token of their own, usually with the space in front included. Rarer words are built from two or more pieces, so “tokenization” becomes “token” + “ization”.
How byte pair encoding builds the vocabulary
GPT tokenizers use byte pair encoding (BPE). Training starts with single bytes, then repeatedly merges the pair that appears most often in a huge amount of text into a new entry. After enough merges, common words, word endings and code keywords each end up as one entry, while unusual strings are left as several smaller pieces.
Because the merges follow frequency, the split of a word can look odd. Capital letters, a missing space or a typo can turn a one-token word into three tokens. Paste two spellings of the same word and compare the highlights to see it happen.
Quick token estimates for English text
- 1 token ≈ 4 characters of English, or about ¾ of a word.
- 100 tokens ≈ 75 words, roughly one short paragraph.
- 1,000 tokens ≈ 750 words, about one and a half pages of a typical document.
- Code, JSON and URLs use more tokens per character than prose, because of symbols, indentation and long identifiers.
- Bangla, Hindi, Arabic and Chinese usually need two to four times the tokens of the same meaning in English.
These rules of thumb are fine for a rough plan. When the number matters, such as a hard context limit or a large batch job, paste the real text and read the exact count.
Why the token count matters
APIs charge per million tokens, with output tokens usually costing several times more than input tokens. The context window, the most a model can read and write in one request, is also measured in tokens, and so are rate limits. A long system prompt is sent with every single call, so trimming 300 tokens from it saves those 300 tokens thousands of times over.
To turn a token count into a cost for a specific model, the AI token calculator multiplies it by the current price list.
Counting tokens for Claude, Gemini and Llama
Each model family trains its own vocabulary, so the same text gives different counts. The figures here are exact for OpenAI models that use o200k_base. For Claude, Gemini or Llama, treat them as an estimate that is usually within 10–30%, and use the provider’s own count-tokens endpoint when you need the precise figure for billing.
Ways to use fewer tokens
- Remove repeated instructions and boilerplate from prompts that run many times.
- Send only the fields the model needs instead of whole JSON records.
- Minify JSON: indentation and line breaks all cost tokens.
- Ask for short answers or a fixed format, since output tokens are the expensive ones.
- Strip leftover markdown and hidden characters from pasted text with the AI output cleaner.