What Is a Token, and How to Count It Exactly: Tokenization, Counting Methods, and Cost Optimization

Once you've been building on LLMs for a while, the same questions keep coming back:

• How many tokens did that one call actually cost? Why is the bill higher than expected?
• Why does the same sentence cost so many more tokens in Chinese than in English?
• The context window says 128K — how much text can I actually fit, and what happens when I exceed it?
• I want to estimate cost before a call — what tool should I use to count tokens?
• Are tokens from GPT, Claude, and Llama the same thing? Can the numbers be reused across them?
• Is "1 token ≈ 4 characters" actually accurate?

These questions keep recurring not because tokens are complicated, but because most explainers stop at "a token is roughly 3/4 of an English word" — without covering how tokenization works, why every vendor's tokenizer differs, or how to actually count tokens precisely and save money in engineering.

This article won't stay at the encyclopedia level of "a token is a piece of text." It explains tokens from a real engineering perspective:

• What a token is and how a model slices text into tokens
• Why OpenAI, Anthropic, and Meta tokens are not interchangeable
• How to count tokens exactly with tiktoken, vendor APIs, and open-source tools
• Why Chinese and English have very different token efficiency, and what that means for cost
• How tokens relate to billing, the context window, and caching
• How to manage a token budget in a real AI application without stepping on landmines

If you build LLM apps, RAG, AI support, or an AI platform, this should give you a complete framework from "understand tokens" to "count them precisely" to "optimize cost."

Set the boundary first: a token is neither a character nor a word

This is the part most worth clarifying early. A lot of cost estimates go wrong not because the math is wrong, but because token gets confused with "character" or "word" from the start.

The boundary between three "text units"

In one line:

• Characters are a length unit for humans.
• Words are a semantic unit for traditional NLP.
• Tokens are the "hard currency" LLMs, billing, and context windows actually accept.

These three counts are close, but never equal. The classic, dangerous mistake is estimating tokens with len(text) (character count) and then budgeting and quoting prices from that.

Why you can't just use character count

For English, the rule of thumb is:

• 1 token ≈ 4 characters ≈ 0.75 English words

But this is an average, not a guarantee. Common pieces (the, ing) often compress to a single token, while rare words, proper nouns, URLs, and code get split into several. Chinese is more extreme — we'll cover that separately.

Get the boundary right, and token math stops drifting.

An overview table to build intuition first

What a token is: the smallest unit from text to model input

A token is the smallest unit an LLM uses to process text — often translated as a "subword piece."

It is neither a full word nor a single character, but a subword fragment in between. A model doesn't see text; it sees a sequence of token ids (integers). Before text enters the model, a tokenizer slices it into tokens and looks them up into ids.

Typical implementations include:

• OpenAI's tiktoken (GPT family)
• Anthropic Claude's proprietary tokenizer
• HuggingFace tokenizers / Google SentencePiece (Llama, Mistral, and other open models)

Its core is not just "chopping text up," but:

• Subword splitting so common words save tokens and rare ones lose no information
• A fixed vocabulary (tens to hundreds of thousands) that can encode any text
• A splitting strategy learned from training corpus statistics, not hand-written rules
• The same word may map to a different number of tokens in different models

The root problem it solves

In language models, some requirements are hard:

• The vocabulary must be finite yet represent infinitely many texts
• Common words shouldn't waste tokens (the shouldn't split into three letters)
• Rare words, typos, and obscure glyphs must still be encodable — never "unrecognized"
• Multilingual text, code, and emoji all need coverage
• Splitting must be reproducible and reversible (tokens can decode back to text)

These share a few traits:

• Balance frequency (short for common, long for rare)
• Balance coverage (no text should be unrepresentable)
• Determinism (the same text always tokenizes the same way for a given model)

Tokens and tokenizers exist for exactly these problems.

BPE: the core algorithm that decides how tokens are cut

Most modern tokenizers (GPT, Claude, Llama included) are built on Byte Pair Encoding (BPE).

BPE's idea is refreshingly simple:

1. Start by splitting text into the smallest units (bytes or characters).
2. Find the most frequent adjacent pair in the corpus and merge it into a new token.
3. Repeat step 2 until the vocabulary reaches the target size.

The result: frequent combinations (ing, tion, http) merge into single tokens, while rare or unusual content falls back to finer pieces or even single bytes. This is why "1 token ≈ 4 characters" is an average — common pieces are far below 4 chars/token, rare ones far above.

Let's see what a real text gets cut into, using OpenAI's tiktoken:

import tiktoken

# gpt-4o uses o200k_base; gpt-4 / gpt-3.5 use cl100k_base
enc = tiktoken.encoding_for_model("gpt-4o")

text = "Hello, world! 什么"

token_ids = enc.encode(text)
print("token count:", len(token_ids))
print("token id sequence:", token_ids)
print("each token decoded back to text:")
for tid in token_ids:
    print(repr(enc.decode([tid])))

You'll see something like:

token count: 7
token id sequence: [13225, 11, 5905, 0, 24402, 104421, 104421]
each token decoded back to text:
'Hello'
','
 'world'
'!'
'什'
'么'
'么'   # (exact splits vary by model/version — run it yourself)

What this shows:

• Hello, world, ! are common pieces that fit in one token
• The Chinese "什么" gets split into multiple tokens — the root of "Chinese costs more" below
• Token ids are integers; that id sequence is what the model actually processes
• encode/decode are reversible: enc.decode(enc.encode(text)) == text

Why every model's tokens differ: tokenizers aren't unified

This is the easiest trap: tokens from different models are not interchangeable.

The same text may produce noticeably different token counts on GPT-4o, Claude, and Llama. The reason isn't that one is "more advanced" — each vendor trained their model on their own corpus, with their own target vocabulary size, and learned their own BPE vocabulary.

The three mainstream tokenizers compared

In one line:

• OpenAI tokens can be counted exactly and offline with tiktoken.
• Claude tokens can only be counted exactly via the official count_tokens endpoint.
• Open-model tokens need their own tokenizer file loaded to be accurate.

The same text, three different splits

import tiktoken

text = "人工智能（AI）正在改变软件开发"

# Just two OpenAI vocabularies, to feel how even one vendor changes between generations
enc_new = tiktoken.encoding_for_model("gpt-4o")      # o200k_base
enc_old = tiktoken.encoding_for_model("gpt-4")       # cl100k_base

print("gpt-4o :", len(enc_new.encode(text)), "tokens")
print("gpt-4  :", len(enc_old.encode(text)), "tokens")

Even two generations from the same vendor differ (GPT-4o's o200k_base optimizes for multilingual, so Chinese is cheaper than on cl100k_base); cross-vendor differences are only larger.

Blunt rule of thumb: talking about token counts without naming the model is like talking about prices without naming the currency. Count against the model you'll actually call.

How to count tokens exactly: four methods

This is the most practical section. "How to count" isn't a one-line answer — pick the method by scenario.

Method 1: tiktoken (OpenAI models — offline, exact, fastest)

If you're calling OpenAI models, tiktoken is the first choice: fully local, no network, millisecond-fast, and matches official billing.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

# But real billing isn't "concatenate all text and count" — it counts by the messages structure.
# Below is OpenAI's recommended chat token estimate (incl. per-message overhead)
def count_chat_tokens(messages: list[dict], model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    # per-message template overhead differs per model; this approximates gpt-4o
    num = 0
    for msg in messages:
        num += 4  # fixed structural overhead per message (role/separators, etc.)
        for key in ("role", "content", "name"):
            if key in msg and msg[key]:
                num += len(enc.encode(msg[key]))
    num += 2  # priming
    return num

messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "What is a token?"},
]
print("chat input_tokens ≈", count_chat_tokens(messages))

Why this matters:

• encoding_for_model picks the right vocab automatically — don't hand-guess cl100k vs o200k
• Real billing counts the messages structure, not a concatenated string — each message has fixed template overhead
• This estimate is off by only a few tokens from official, which is solid for budgeting and truncation

Method 2: Vendor count_tokens endpoint (Claude — exact)

You can't use tiktoken for Claude because its vocabulary isn't public. Anthropic offers a dedicated counting endpoint that matches billing:

curl https://api.anthropic.com/v1/messages/count_tokens \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "messages": [
      {"role": "user", "content": "What is a token, and how do I count it?"}
    ]
  }'

Returns:

{
  "input_tokens": 12
}

Notes:

• This endpoint is free — it exists purely for estimation
• It accounts for the messages structure, system prompt, and tools definitions, so it's more accurate than any hand calculation
• The cost is one network request, so don't call it per-item on a very hot path

Method 3: HuggingFace tokenizers / SentencePiece (open models)

For Llama, Mistral, Qwen and other open models, you load each model's own tokenizer files:

from transformers import AutoTokenizer

# Load the tokenizer shipped with the model
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

text = "What is a token, and how do you count it?"
print("token count:", len(tokenizer.encode(text)))
print("token ids:", tokenizer.encode(text))

Key points:

• You must use the tokenizer that matches the target model — wrong model, wrong count
• AutoTokenizer pulls files from the Hub automatically; for offline use, download them ahead of time
• This is the authoritative way to count when self-hosting open models

Method 4: Estimation rules (fast approximation, model-independent)

Often you don't need exact — just a magnitude (for quota, product pricing). A rule of thumb is fastest:

# A conservative, dependency-free upper-bound estimate for Chinese-mixed text
def estimate_tokens_naive(text: str) -> int:
    # Chinese: round up at 1.5x as an upper bound; English: 4 chars per token
    chinese = sum(1 for c in text if "一" <= c <= "鿿")
    other = len(text) - chinese
    return int(chinese * 1.5 + other / 4) + 1

The value here is zero dependencies, zero latency — but remember it's an upper bound, not something to put on a customer invoice.

Comparing the four methods

Don't make "estimation" do the job of "billing," and don't let "API counting" throttle every request's performance. Matching the method to the scenario is the real point of this section.

Chinese vs. English token efficiency

This is where Chinese-speaking teams most often overspend on LLMs.

Same meaning, but Chinese simply costs more tokens

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

def count(text: str) -> int:
    return len(enc.encode(text))

en = "The quick brown fox jumps over the lazy dog."
zh = "敏捷的棕色狐狸跳过了那只懒狗。"

print("English:", count(en), "tokens /", len(en), "chars")
print("Chinese:", count(zh), "tokens /", len(zh), "chars")

Running this makes it obvious:

• English: 9 words, averaging "under 1 token/word"
• Chinese: a dozen glyphs, almost 1 token or more per glyph

Why this matters

It hits engineering and product both:

• Cost: the same product copy costs more in the Chinese version over time
• Context window: a 128K window fits less Chinese "page length" than English
• GEO / AI visibility: LLMs consume your web content by the token, so a Chinese page delivers less effective content per window — clean, information-dense content wins
• Truncation strategy: the gap between "truncate by chars" and "truncate by tokens" is much larger for Chinese — you must truncate by tokens

In short: budget Chinese LLM applications as "more expensive per token" and leave real headroom.

Tokens and billing: input, output, caching

Understanding tokens is how you read the bill. Most LLMs bill by the token, with different prices for input and output.

Three basic billing rules

• Input tokens: everything you send (system + history + user input + tools definitions).
• Output tokens: what the model generates. Output is usually 3–5× more expensive than input.
• Total cost = input tokens × input price + output tokens × output price.

Indicative prices (for structure only — always check each vendor's official pricing):

The point isn't the numbers, but three things:

• Output costs far more than input — making the model talk less is the most direct cost win
• Cache hits are dirt cheap — repeated long prompts via caching save an order of magnitude
• Model tiers differ enormously in price — use the smallest model that works

max_tokens: a hard cap on output

Setting max_tokens on the call is the first gate against cost blowups:

from anthropic import Anthropic

client = Anthropic()

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,          # at most 512 output tokens, hard cap
    messages=[{"role": "user", "content": "Explain a token in one sentence."}],
)
print(resp.usage)            # usage has input_tokens / output_tokens

• max_tokens caps output; it doesn't reduce input billing
• usage.input_tokens and usage.output_tokens are the authoritative numbers for reconciliation
• In production, always set max_tokens — one chatty reply can spike a single call's cost

Prompt caching: the single biggest saver

If your request has large repeated content (long system prompt, fixed knowledge chunks, few-shot examples), prompt caching lets that part hit at a very low price:

# Anthropic: mark a cacheable block
curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 256,
    "system": [
      {"type": "text", "text": "You are a support agent. ...(very long fixed persona and knowledge)...", "cache_control": {"type": "ephemeral"}}
    ],
    "messages": [{"role": "user", "content": "How do I get a refund?"}]
  }'

Rules of thumb:

• Only content that is fixed, repeated, and long enough is worth caching (must meet a minimum cache-token threshold)
• Writing to cache has a cost (slightly above normal input), but it pays off if reused multiple times
• High-frequency support and batch jobs with a fixed prompt are where caching pays the most

Tokens and the context window

The context window is the maximum number of tokens a model can "see" in one call. The key insights:

• Window = input tokens + output tokens — they share the same budget
• A big window doesn't mean "stuff whatever you want" — fuller means more latency, more cost, and sometimes "middle content gets ignored"
• Exceeding the window either errors or is silently truncated, depending on SDK/config

What to do with over-long history: truncate by tokens

The most common multi-turn problem is "history grew past the limit." The right fix is rolling truncation by token budget, not by message count:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

def token_of(text: str) -> int:
    return len(enc.encode(text))

def trim_history(messages: list[dict], budget: int, keep_system: bool = True) -> list[dict]:
    """Drop the oldest non-system messages by token budget until total <= budget."""
    system = [m for m in messages if m["role"] == "system"] if keep_system else []
    rest = [m for m in messages if m["role"] != "system"]

    used = sum(token_of(m["content"]) for m in system + rest)
    # Drop oldest history first (earlier turns in rest)
    while used > budget and rest:
        dropped = rest.pop(0)
        used -= token_of(dropped["content"])
    return system + rest

messages = [
    {"role": "system", "content": "You are an assistant."},
    {"role": "user", "content": "Turn 1 long text..."},
    {"role": "assistant", "content": "Reply 1..."},
    {"role": "user", "content": "Turn 2..."},
]
print(trim_history(messages, budget=200))

Three things this captures:

• Always keep system (persona/rules must not be cut)
• Truncate by tokens, not by message count — truncating Chinese by count is wildly off
• Drop the oldest, keep recent context — the standard strategy for chat

More mature systems replace simple truncation with summarization or vector retrieval (RAG), but only once "simple truncation can't keep up" — don't over-engineer from day one.

Managing tokens in a real AI application

Stitching the above together gives you the token-management pipeline of a real app.

A division-of-labor table

When to count exactly vs. estimate

• Count exactly: high rate, customer-facing quotes, hard quota enforcement, truncation tipping points.
• Estimate: internal coarse rate limiting, early-stage magnitude checks, non-billing paths.

The test is simple: will this number directly become money or directly shape user experience? If yes, count exactly; if no, estimate.

Reference architecture: a chat app's token-budget pipeline

Division of labor

• Entry service: receives the request, assembles messages
• Counting module: tiktoken (OpenAI) or count_tokens (Claude) for input tokens
• Budget controller: if over budget, truncate history or trigger summarization
• Cache layer: fixed prompts go through prompt caching
• Call layer: calls the model with max_tokens
• Reconciliation module: takes real tokens from usage into the billing ledger

Architecture diagram

flowchart LR
    A[User input] --> B[Assemble messages]
    B --> C[Count input_tokens]
    C --> D{Over budget?}
    D -- Yes --> E[Truncate/summarize history]
    E --> B
    D -- No --> F[Fixed prompt via cache]
    F --> G[Call model with max_tokens]
    G --> H[Output output_tokens]
    H --> I[Reconcile usage into billing]

Pipeline notes

1. The user sends a message; the entry service assembles system, history, and current input into messages.
2. The counting module computes input_tokens for the target model.
3. The budget controller checks against the context budget.
4. If over budget, it truncates or summarizes the oldest history and loops back to step 2.
5. Long fixed prompts go through prompt caching, billed at the cache rate on hit.
6. The model is called with max_tokens set.
7. The model returns; usage.output_tokens is the output count.
8. The reconciliation module writes the real numbers from usage to the billing ledger and compares against the estimate.

The keys here:

• Counting happens before the call; truncation happens before the call
• Output has a cap; cost has caching
• Reconciliation uses real usage, not estimates

With these responsibilities clear, token-related incidents drop sharply.

How tokens add up for a single request

Take "user asks one question, model answers" and break down where the tokens actually come from.

1. System prompt: fixed persona and rules

You are a concise technical assistant. Answer in at most 3 sentences.

Usually repeated and fixed — the best candidate for caching.

2. Tools / function definitions: overhead that's easy to miss

[
  {"name": "search", "description": "Search the knowledge base", "input_schema": {"type": "object", "properties": {"q": {"type": "string"}}}}
]

Tools count toward input_tokens, and JSON costs more tokens than prose. More tools = more fixed overhead.

3. Conversation history: grows linearly with turns

user: What is a token?
assistant: A token is the smallest unit an LLM uses to process text...
user: Then how do I count them?

Multi-turn input keeps growing — the root reason you need truncation/retrieval.

4. Current user input: usually the smallest share

Do Chinese or English cost more tokens?

5. Model output: billed at the output price

Chinese costs more tokens, because the tokenizer covers Chinese glyphs less densely than English...

Once you see these five parts, the "why is my bill so high" mystery usually disappears — it's typically system + tools + history that dominate, not the user's single input.

Common misconceptions, busted

Misconception 1: 1 token is always 4 characters

No. That's an English average. Chinese, code, and JSON all differ, and different models differ too.

Misconception 2: Token counts are reusable across models

No. GPT, Claude, and Llama each have their own vocabulary; the same text yields different counts. Always count against the target model.

Misconception 3: len(text) is accurate enough

Roughly OK for English, wildly off for Chinese. Anything touching billing or truncation can't use character count.

Misconception 4: A 128K context window means 128K characters fit

No. 128K is tokens, and input + output share it. Chinese fits far less text than 128K characters.

Misconception 5: Output tokens cost the same as input

No. Output is usually 3–5× more expensive. Making the model talk less beats compressing input.

Misconception 6: Caching only saves a little

Wrong. Cached input can drop to around 10%. For high-frequency scenarios with a long fixed prompt, caching is the single biggest win.

Misconception 7: The more token optimization, the better

Usually the opposite. Over-optimization (per-item exact counting, aggressive compression, caching everywhere) adds complexity and latency for diminishing returns. Get max_tokens, token-based truncation, and fixed-prompt caching right first; only then pursue fine-grained tricks.

Anti-over-engineering: don't obsess over token optimization

This part is necessary. Many teams don't get token management wrong — they over-optimize.

A practical anti-over-engineering checklist

A few rules of thumb

• If you only need a magnitude, start with estimation rules — don't wire tiktoken across the whole chain first.
• If a prompt isn't long or reused, skip caching — write costs may eat the gains.
• If you only fear exceeding the window, do plain token-based truncation first — don't jump to summarization + vector retrieval.
• If it's a high-frequency call, switch to a smaller model and enable caching — that beats any "extreme prompt compression."

The hardest part of token optimization isn't knowing the tricks — it's knowing when not to use them.

Quick selection guide

When to prefer tiktoken

• Calling OpenAI models
• Need offline, millisecond counting
• Doing input budgeting, history truncation

When to prefer the vendor count_tokens endpoint

• Calling Claude (vocab not public)
• Need an estimate that matches official billing exactly
• Counting frequency is low enough to accept one network request

When to prefer tokenizers / SentencePiece

• Self-hosting open models (Llama, Mistral, Qwen, etc.)
• Need offline counting matched to that model's vocab

When to prefer estimation rules

• Internal rate limiting, quota magnitude checks
• Early-stage product quotes, non-billing paths
• Zero-dependency, zero-latency quick judgments

Summary

Compress the whole article into a few lines:

• A token is the smallest unit an LLM uses to process text — split by subword, not characters or words.
• Tokenizers differ per vendor; OpenAI, Claude, and Llama tokens aren't interchangeable.
• tiktoken is the first choice for exact offline counting on OpenAI models.
• The count_tokens endpoint is the only authoritative way to count Claude tokens exactly.
• Chinese costs more tokens than English — budget both cost and window as "more expensive."
• Output costs more than input; max_tokens and prompt caching are the two most practical tools.
• The context window is shared by input + output; when over, truncate by tokens, not by message count.

Mature token management isn't counting every request down to the single token, nor compressing prompts to the limit.

Mature practice is:

• Knowing when to count exactly vs. estimate
• Keeping cost gates in place (max_tokens + caching)
• Truncating by tokens, reconciling with usage
• Not over-engineering for marginal gains

Get these right, and token-related cost and stability usually stay under control.