A library for benchmarking different vocabulary mining methods on real inference workloads

A vocabulary miner can find repeated strings and produce an impressive token count without demonstrating that the patch is useful in production. Three separate problems have to be measured:

1. A tokenizer can compress text while making the model worse.
2. A token removed from a cached prompt prefix may remove essentially no GPU prefill work.
3. Candidate savings are not additive because new tokens compete for overlapping spans.

VocabPerf makes those stages explicit. It audits the completed tokenizer on traffic reserved for evaluation rather than adding independent candidate estimates; applies the same adaptation recipe to competing vocabularies; evaluates quality at equal text boundaries; and then runs an A/B test on a serving engine. The goal is not to declare one algorithm universally best. It is to make vocabulary mining claims comparable and reproducible.

Coding agent prompts are highly repetitive. They contain system prompts, tool definitions, repository context, previous turns, and fresh tool output. Serving engines can cache blocks from repeated prefixes and reuse the corresponding KV state.

Consider two vocabulary candidates that each replace five old tokens with one new token. On paper, both save four tokens. But suppose the first candidate occurs primarily in a warm system prompt while the second occurs in newly appended compiler output. The first makes the tokenizer count smaller, but the serving engine was not going to recompute those cached tokens anyway. The second removes fresh prefill work on every occurrence.

Prompt length matters too. On the L4 profile used in this study, median prefill time increased from approximately 52 ms at 2,724 tokens to 200 ms at 8,168 tokens. Removing a few fresh tokens from a long prompt can therefore have a different marginal value than removing the same number from a shorter one.

VocabPerf's experimental selector, PrefillValue, scores an occurrence roughly that way. For every chronological request, it tokenizes the full prompt, simulates the prefix cache one block at a time, identifies the uncached suffix, and attributes measured L4 prefill work to candidate occurrences in that suffix.

It also handles overlap. If both raise_for_status and .raise_for_status() are candidates, selecting one changes the remaining value of the other. PrefillValue greedily selects the candidate with the highest current value, applies it to the workload, and recomputes the marginal value of the remaining candidates.

Comparing tokenizer papers from their headline numbers is nearly meaningless if they use different corpora, models, vocabulary sizes, training recipes, quality tests, and serving stacks. VocabPerf instead treats a miner as one component in a controlled pipeline.

For the main study, five selectors received the same SWE Hero coding agent traffic and the same vocabulary budget of 256 tokens:

• Frequency, a deliberately simple control.
• AdaptiVocab, independently implemented from its published frequency × tokens eliminated objective.
• Continued BPE, using the official tokenizer extension implementation.
• AgentVocab, using its official structural/content mining and gradient ranking components.
• PrefillValue, VocabPerf's selector that accounts for cache state and overlap.

This is a comparison of miner components, not a reproduction of every paper's full system. In particular, AgentVocab also uses a training procedure with two LoRA stages; the study deliberately replaces that with a common recipe that updates only the added rows, so vocabulary selection is the variable being tested.

Every patch appended 256 literal tokens while preserving existing token IDs. Each new row was initialized from its decomposition into old tokens. The base Qwen2.5 Coder 0.5B model was frozen, and only 229,376 new row parameters were trained for 1,000 steps. All patches used the same replay mixture: 25% target domain traffic and 75% external coding agent traffic.

AdaptiVocab removed slightly more raw prompt tokens, but PrefillValue removed more tokens from regions the GPU would actually have to prefill. Against the best published miner component in this setup, PrefillValue improved predicted reduction in prefill time by 1.74 percentage points, or 7.56% relative.

There is an important negative result: an internal selector that used uniform token value but the same greedy overlap accounting predicted a 25.38% reduction, slightly better than PrefillValue's 24.69%. The robust conclusion is therefore that cache and overlap awareness matter. This experiment does not establish that the current equation weighted by hardware is universally optimal.

Tokenizer compression alone says nothing about model quality. VocabPerf uses two types of gates.

First, each patch was evaluated using paired bits per byte on untouched examples from both the target domain and external coding agent traces. Bits per byte makes models using different tokenizations comparable at the same text boundary. All five patches passed the predeclared 5% quality margin, although PrefillValue's external gate passed narrowly.

Second, the two efficiency finalists ran deterministic pass@1 on all 164 HumanEval+ and 378 MBPP+ tasks:

Both patches retained 100% of the base model's aggregate score and passed the 5% quality tests for each suite. That does not mean they behaved identically. PrefillValue produced six MBPP+ regressions and four improvements. Equal totals can conceal different failures, and 542 tasks on a small model are nowhere near a statistical guarantee of behavioral parity.

The new token IDs also appeared only rarely in generated output. The strongest evidence here is for compressing prompts and reducing prefill work. It is not evidence for accelerating long autoregressive generation.

The two finalist patches were materialized as normal Hugging Face checkpoints and served with vLLM 0.23.0 in BF16 on the same NVIDIA L4.

The serving workload contained 128 prompts reserved for evaluation and 699,660 tokens from the base tokenizer. Each request generated one token to isolate prompt processing. The first chronological pass began with an empty workload cache and then accumulated prefix hits naturally:

The engine counters are the cleanest result: PrefillValue removed 4.32 percentage points more freshly computed prompt tokens while removing slightly fewer raw tokens overall.

The timing result is exploratory because the first chronological pass contains one aggregated trial per patch. Throughput is also the reciprocal of duration, so its percentage increase should not be confused with an equal reduction in dollars.

When the identical requests were replayed into an already populated, nearly fully warm cache, the advantage changed substantially and AdaptiVocab performed better. That is expected: once almost every prompt block is cached, PrefillValue's target has mostly disappeared. Real value depends on traffic distribution, routing, cache policy, batching, hardware, and model size.

VocabPerf is not a universal vocabulary and it is not a proxy for a hosted API discount. A provider must control the tokenizer and model weights to use a patch.

The current result uses one small dense model, one primary public trace distribution, one GPU type, and one serving engine for the final matched test. The useful next experiments test whether the central result survives different systems:

• larger dense and MoE coding models;
• workloads with long outputs and reasoning, where output head overhead and decode behavior matter more;
• traces from multiple tenants with realistic cache eviction and routing;
• A100, H100, and costs across different accelerators;
• repeated serving trials with controlled cache state and batching;
• stronger adaptation recipes when row only training does not meet a team's chosen quality margin.

VocabPerf's larger goal is not “PrefillValue wins every table.” It is making vocabulary claims testable under an equal budget, a common quality protocol, and a real serving workload.

A vocabulary miner leaderboard based only on tokens saved ends too early. A useful vocabulary for a specific workload has to answer three independent questions:

1. Does the completed tokenizer actually shorten traffic reserved for evaluation?
2. Does it remove computation the serving engine would otherwise perform?
3. Can the model learn the new symbols without an unacceptable quality loss?

On this small coding agent study, selection that accounts for cache state and overlap removed more uncached work on the first pass than the published miner components at the same budget of 256 tokens. The two finalists retained the base model's aggregate EvalPlus score. It did not win every cache regime, and the hardware weighting did not beat every internal ablation.

That is a narrower result than a miraculous cost reduction. It is also a more useful one.