How to Compare Token Usage Across OpenAI-Compatible Models

An OpenAI-compatible endpoint can accept the same JSON request and still count a different number of tokens.

That is the first rule of a fair model comparison: API compatibility is not tokenizer compatibility.

Two endpoints may both support POST /v1/chat/completions, the same messages array, and familiar usage fields such as prompt_tokens and completion_tokens. Behind that shared interface, however, they may use different tokenizers, chat templates, tool formatting, caching rules, or reasoning-token accounting.

If you compare only the final total_tokens number, you can easily draw the wrong conclusion about prompt efficiency or cost.

This guide shows how to run an apples-to-apples token benchmark across OpenAI-compatible models, normalize the results, and use TokenTest to check whether each endpoint's usage evidence is present and plausible.

The short answer

To compare token usage fairly:

1. Send the same semantic task and the same request structure to every endpoint.
2. Lock temperature, output limit, tools, response format, streaming mode, and cache state.
3. Record input, output, total, cached, and reasoning tokens separately.
4. Compare both raw tokens and tokens per unit of useful work.
5. Repeat each test and flag usage fields that are missing or internally inconsistent.

Do not expect equal counts. The goal is not to force every model to return the same number. The goal is to understand why the numbers differ and whether the reporting is trustworthy.

Why OpenAI-compatible models count tokens differently

“OpenAI-compatible” usually describes an API surface. It means a provider, gateway, or self-hosted server accepts requests shaped like an OpenAI API request.

It does not guarantee that the backend uses OpenAI's tokenizer.

1. The tokenizer vocabulary can differ

A tokenizer breaks text into model-readable units. Different model families use different vocabularies and segmentation rules, so the same sentence, JSON object, code block, or Chinese paragraph can produce different token counts.

This is especially visible with:

• non-English text;
• whitespace-sensitive code;
• long identifiers and URLs;
• JSON keys and punctuation;
• emoji or uncommon Unicode characters.

OpenAI's own documentation recommends using the tokenizer or token-counting method associated with the model you actually call. A local estimate from another model family may be useful for planning, but it is not authoritative for a third-party endpoint.

2. The server may apply a different chat template

Your request contains roles and message content. The model usually receives a serialized sequence with special control tokens that mark the system message, user message, assistant turn, or tool call.

Hugging Face's chat-template documentation explains that models can use different control tokens even when the conversation looks identical to the developer. vLLM's OpenAI-compatible server likewise depends on a model chat template for chat requests.

That means this request:

{
  "messages": [
    {"role": "system", "content": "Answer concisely."},
    {"role": "user", "content": "Explain prompt caching."}
  ]
}

can become a different internal token sequence on different backends.

3. Tools and structured output add hidden request structure

Tool definitions, JSON Schema, response-format instructions, and provider-added wrappers can increase input usage even when the visible user prompt stays unchanged.

OpenAI's token-counting guide notes that request structure matters, including tools and schemas. For a fair benchmark, you must either use the exact same tool schema everywhere or run a separate text-only test.

4. Output accounting may include more than visible text

Some models expose reasoning or thinking tokens separately. Some include non-visible model work inside output usage. Some OpenAI-compatible layers flatten detailed usage into a simpler completion_tokens number, while others omit the detail entirely.

Therefore, “the answer is 80 visible words” does not imply that every endpoint should report the same output-token count.

5. Caching changes the meaning of repeated runs

An endpoint may report cached input tokens, cache reads, or cache creation tokens. Another endpoint may support caching but omit those details from its OpenAI-compatible response.

If one run is warm and another is cold, total token counts may look similar while billable or processed-token behavior differs. Keep the first-run and repeated-run results separate.

The benchmark setup: lock these variables first

Use a small test matrix rather than one giant prompt. Each test should isolate one source of token variation.

Also record the exact model ID returned by the endpoint when available. A friendly alias can silently point to a different backend later.

A five-test token comparison suite

The following prompts are deliberately compact. You can paste them into TokenTest and run them against multiple configured endpoints.

Test 1: short English instruction

Summarize the following support issue in exactly three bullets. Each bullet must be under 12 words.

The customer upgraded yesterday. The dashboard still shows the old plan, but the invoice shows the new charge. They already signed out and back in.

What it tests: baseline English tokenization, instruction overhead, and short-output discipline.

Test 2: English and Chinese semantic pair

English:

Explain why prompt caching can reduce repeated-input processing. Use two sentences and no jargon.

Chinese:

请用两句话解释为什么提示词缓存可以减少重复输入的处理量，不要使用专业术语。

What it tests: language-dependent tokenizer efficiency. Compare each language within the same model first, then compare models.

Test 3: JSON extraction

Extract customer_id, renewal_date, plan, and risk_level. Return valid JSON only.

Account AC-1049 is on the Pro annual plan. Renewal is 2026-08-15. The customer has opened three billing tickets and said they may cancel.

What it tests: punctuation, field names, structured-output discipline, and visible output per reported output token.

Test 4: long repeated prefix

Policy reference:
- Refunds require an account identifier.
- Annual plans may be refunded within 14 days of renewal.
- Fraud claims must be escalated.
- Do not promise a refund before review.

Using only the policy above, draft a two-sentence reply to a customer requesting a refund 10 days after annual renewal.

Run it twice without changing anything.

What it tests: repeated-request behavior and cache-token evidence. Do not assume that a second run is cached unless the endpoint reports evidence.

Test 5: tool-schema overhead

Run Test 3 once as plain JSON output and once with a function/tool definition containing the same four fields.

What it tests: how much request-side overhead the endpoint adds for tools and schemas.

Record a normalized result, not just total tokens

For each run, collect this worksheet:

Then calculate metrics that answer a useful question:

input efficiency = input tokens / input characters
output density = visible output characters / output tokens
task efficiency = total tokens / valid completed task
cache share = cached input tokens / input tokens

These ratios are not universal quality scores. They are diagnostic signals.

For example, a low total-token result is not efficient if the model fails the JSON schema. A longer answer is not automatically wasteful if it completes a task that shorter answers miss. Mark task validity first, then compare token efficiency among valid outputs.

How to read the usage object safely

OpenAI-style chat responses commonly expose:

{
  "usage": {
    "prompt_tokens": 142,
    "completion_tokens": 38,
    "total_tokens": 180
  }
}

Newer or provider-specific responses may also expose nested details for cached input, reasoning, accepted prediction tokens, audio tokens, or other modalities.

Normalize those fields into your own internal schema, but preserve the raw response. Use null or not reported when a field is absent. Do not convert a missing cache or reasoning field to zero, because zero means the endpoint explicitly reported none; missing means you do not know.

Also run an arithmetic check when the fields make that possible:

reported input + reported output ≈ reported total

The exact relationship can depend on the provider's schema, but an unexplained mismatch deserves investigation.

Common comparison mistakes

Comparing aliases instead of pinned models

An alias may change its backend without changing your request. Save the requested model, returned model, test date, and endpoint.

Using one local tokenizer for every endpoint

A local tokenizer is useful for a rough estimate only when it matches the deployed model and chat template. For authoritative planning, use the provider's native counter when available and verify the live usage response.

Mixing text-only and tool-enabled requests

Tool definitions can materially change input usage. Treat tool calling as a separate benchmark track.

Treating missing details as zero

If reasoning or cache fields are absent, label them not reported. Otherwise, you may incorrectly reward an endpoint for hiding usage detail.

Ranking models by total tokens without checking the answer

Token efficiency only matters after the output passes the task. Validate JSON, instruction compliance, language, and factual constraints before calculating a winner.

Running each prompt only once

Output length varies. Run at least three repetitions and report the median, plus the minimum and maximum when variance is meaningful.

Where TokenTest fits

TokenTest is designed for live endpoint verification rather than offline tokenization alone.

Its current product interface includes checks for:

• usage presence and plausibility;
• input, output, and cache token evidence;
• total-token consistency;
• input-token monotonicity as prompts grow;
• output-token reasonableness;
• stop-limit linkage;
• stream-usage consistency;
• cache behavior across repeated calls;
• thinking or reasoning-token evidence.

That makes it useful when you are comparing a direct provider endpoint with a gateway, reseller, router, or self-hosted OpenAI-compatible server.

Start with the five prompts above, export the results, and keep the raw JSON alongside your normalized worksheet. TokenTest says API keys are not stored, but you should still follow your organization's credential and data-handling policies.

A practical decision rule

Use three layers:

1. Local estimate for fast editing feedback.
2. Provider-native preflight counter for request sizing when available.
3. Live TokenTest run for runtime usage integrity and cross-endpoint comparison.

This avoids two bad shortcuts: assuming one tokenizer represents every compatible model, or assuming every usage object tells the whole story.

FAQ

Should two OpenAI-compatible models report the same input-token count?

No. They can accept identical request JSON while using different tokenizers, chat templates, special tokens, and server-side wrappers.

Is the model with fewer tokens always cheaper or better?

No. Pricing can differ, and a low-token output has no value if it fails the task. Compare current provider pricing separately and calculate token efficiency only for valid outputs.

Can I use tiktoken for every OpenAI-compatible endpoint?

Only as an estimate unless the endpoint confirms that it uses the matching tokenizer and request template. Use native counting or live usage for authoritative results.

What if an endpoint reports only total tokens?

Keep the result, but mark the missing input/output details as unavailable. That endpoint provides less evidence for diagnosis and cost attribution.

How often should I rerun the benchmark?

Rerun after model-version changes, gateway changes, chat-template changes, or usage-schema changes. Also rerun important production routes on a schedule so aliases do not drift unnoticed.

Final checklist

Before you name a token-efficiency winner, confirm that you:

• used identical semantic inputs;
• locked generation and tool settings;
• separated cold and repeated runs;
• recorded raw and normalized usage fields;
• distinguished zero from not reported;
• validated the output before scoring efficiency;
• repeated each case;
• and saved the model version and test date.

Then run the comparison in TokenTest and use the evidence—not API compatibility alone—to decide which endpoint fits your production workload.

Sources

• TokenTest homepage and live product behavior: https://tokentest.io/
• TokenTest manual: https://tokentest.io/manual
• OpenAI token-counting guide: https://developers.openai.com/api/docs/guides/token-counting
• OpenAI API reference: https://developers.openai.com/api/reference
• Hugging Face chat templates: https://huggingface.co/docs/transformers/chat_templating
• vLLM OpenAI-compatible server documentation: https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html