Quick access

Saved in this browser. Sign in to keep it on all your devices.

AI Tokenizer

See how GPT models split your text into tokens. Each token is highlighted, with its ID one click away. Uses the o200k_base encoding behind GPT-5, GPT-4.1 and GPT-4o, and runs entirely in your browser.

Tokens
0
Characters
0
Words
0
Characters per token
–

Loading the tokenizer… the count is an estimate for now. The tokenizer could not load. Check your connection and refresh the page; counts are estimates until then. Exact for GPT-5, GPT-4.1, GPT-4o and o-series models. Claude and Gemini use their own tokenizers, so expect different counts.

Tokens appear here as you type.

Showing the first tokens. The counts above cover all of your text; Copy IDs copies every token.

Spaces show as · and line breaks as ↵. A highlighted block with several IDs is one character spread over several tokens, such as an emoji.

Want the price? Compare what your prompt costs on GPT, Claude and Gemini →

Version 1.1 Updated What's new

Release notes

  1. v1.1 Latest
    • Added an in-depth guide below the tool, written from what the tool actually does.

About the AI Tokenizer

Language models read text as tokens: common words are one token, rarer words, numbers, emoji and non-English scripts split into several. The tokenizer shows exactly where those splits fall, with each token highlighted and its ID, using the o200k_base encoding behind GPT-4o, GPT-4.1 and GPT-5.

How to use it

  1. 1 Paste or type your text, or try the example.
  2. 2 Read the token, character and word counts, and see each token highlighted.
  3. 3 Switch to Token IDs to see the numbers the model reads, and copy them as JSON.

Frequently asked questions

What is a token?

A token is a chunk of text a language model reads as one unit. In English, one token is about 4 characters or three quarters of a word, but the exact split depends on the tokenizer.

Which models does this tokenizer match?

It uses o200k_base, the encoding for GPT-4o, GPT-4.1, GPT-5 and the o-series models, so counts for those are exact. Claude, Gemini and Llama use their own tokenizers, so their counts differ, usually by 10–30%.

Why do Bangla, Hindi or emoji use more tokens?

Tokenizers are trained mostly on English, so other scripts and emoji are split into smaller pieces, sometimes single bytes. The same sentence can cost two to four times as many tokens in Bangla as in English.

Why does a word with a leading space count differently?

Most tokens include the space before a word, so " hello" and "hello" are different tokens. Spaces are shown as dots so you can see this.

Is my text sent anywhere?

No. The tokenizer runs entirely in your browser, so your text never leaves your device.

What tokenization means in AI

Tokenization is the first thing a language model does with your prompt. The text is cut into pieces from a fixed vocabulary, and each piece is swapped for a number, its token ID. The model never sees letters or words, only that list of numbers, and it writes its answer the same way, one token ID at a time, which is then turned back into text.

The o200k_base vocabulary used here has about 200,000 entries. Frequent English words such as “the”, “water” or “because” have a token of their own, usually with the space in front included. Rarer words are built from two or more pieces, so “tokenization” becomes “token” + “ization”.

How byte pair encoding builds the vocabulary

GPT tokenizers use byte pair encoding (BPE). Training starts with single bytes, then repeatedly merges the pair that appears most often in a huge amount of text into a new entry. After enough merges, common words, word endings and code keywords each end up as one entry, while unusual strings are left as several smaller pieces.

Because the merges follow frequency, the split of a word can look odd. Capital letters, a missing space or a typo can turn a one-token word into three tokens. Paste two spellings of the same word and compare the highlights to see it happen.

Quick token estimates for English text

  • 1 token ≈ 4 characters of English, or about ¾ of a word.
  • 100 tokens ≈ 75 words, roughly one short paragraph.
  • 1,000 tokens ≈ 750 words, about one and a half pages of a typical document.
  • Code, JSON and URLs use more tokens per character than prose, because of symbols, indentation and long identifiers.
  • Bangla, Hindi, Arabic and Chinese usually need two to four times the tokens of the same meaning in English.

These rules of thumb are fine for a rough plan. When the number matters, such as a hard context limit or a large batch job, paste the real text and read the exact count.

Why the token count matters

APIs charge per million tokens, with output tokens usually costing several times more than input tokens. The context window, the most a model can read and write in one request, is also measured in tokens, and so are rate limits. A long system prompt is sent with every single call, so trimming 300 tokens from it saves those 300 tokens thousands of times over.

To turn a token count into a cost for a specific model, the AI token calculator multiplies it by the current price list.

Counting tokens for Claude, Gemini and Llama

Each model family trains its own vocabulary, so the same text gives different counts. The figures here are exact for OpenAI models that use o200k_base. For Claude, Gemini or Llama, treat them as an estimate that is usually within 10–30%, and use the provider’s own count-tokens endpoint when you need the precise figure for billing.

Ways to use fewer tokens

  • Remove repeated instructions and boilerplate from prompts that run many times.
  • Send only the fields the model needs instead of whole JSON records.
  • Minify JSON: indentation and line breaks all cost tokens.
  • Ask for short answers or a fixed format, since output tokens are the expensive ones.
  • Strip leftover markdown and hidden characters from pasted text with the AI output cleaner.