Hugging Face proposes Model Integration Contracts after a UMT5 training bug went unnoticed in Transformers for two months

• pip install "transformers-mic[runtime]"; the tmic run org/model --local-only command drafts a contract, builds the small model and records baselines locally. Nothing is uploaded (HF blog; CLI docs).
• For new models, an LLM drafts the usage code through Inference Providers, billed to the user's own account. Humans approve any changed expectations.
• Published contracts cover models including SmolLM, T5, Whisper, BERT, CLIP, LLaMA and Mistral3 (huggingface-mic). Gaps are tracked in the coverage space.

• Each contract has three parts: the small test model (pinned), a runnable Markdown usage example, and reviewed baseline outputs. It covers an architecture plus a usage path, not one checkpoint (HF blog).
• How models are shrunk: reduce width and depth but keep every layer type (sliding vs. full attention, dense vs. MoE). Tokenizer and processor files are copied byte for byte, and the original weight file format is kept so conversion code still runs. Weights are checked for finiteness and responsiveness. Llama 4 Scout goes from 108.6B to 44.7M parameters (shrink report).
• Reproducibility: a fleet dataset pins contract revisions and the package versions used to record baselines. The default tolerance is about 1e-4 (evidence).
• The test must follow the real code path: only the default SDPA path caught UMT5. Testing the eager attention path alone missed it.

All of the figures below come from Hugging Face's own reports. None has been replicated independently.

• Replay over past commits: Hugging Face replayed 1,468 commits × 8 test models, May–October 2026. Five regressions were in range. One was caught at its introducing commit (UMT5) and four were missed because the contracts didn't call the affected APIs. There was one false alarm, from a Wav2Vec2 random-number shift. After the missing calls were added, all five were caught (evidence).
• CI demo: with the UMT5 fix reverted, the check failed with label log-probabilities off by 1.22 and loss off by 0.204. The T5 runner step took 13.4 s on CPU (HF blog). The demo used a proof branch, not production pins (evidence).
• Stability: 770 checks across dependency upgrades produced zero flags. Maximum drift was 1.3e-5 on the test models.

Transformers already publishes tiny random checkpoints (tiny-random-*, hf-internal-testing). Those mostly check that a model loads and runs a forward pass. MICs add a standard recipe, real preprocessing, and training checks: loss, finite gradients, and whether an optimizer step changes weights. Hugging Face says the suite already runs 88,000+ CPU tests per run on main and 200,000+ GPU tests daily. Full GPU integration tests on every PR are impractical at that scale (HF blog). Similar SDPA masking bugs have occurred elsewhere, for example in Doge.

• Coverage depends on which APIs a contract calls: generic forward-plus-generate checks missed most of the replayed historical bugs (evidence).
• Random vs. overfit weights: overfit Whisper test models caught 24/24 injected faults; random ones caught 18/24. Across six model families the totals were 126/126 vs. 120/126. These were synthetic faults, and no real regression has been caught only by overfit weights (fixtures doc).
• Some bugs need real weights: both Whisper variants missed an assisted-decoding bug that the real checkpoint exposed.
• Precision: some random bfloat16 seeds produced discrete output changes.

• Teams fine-tuning on Transformers v5.15–v5.17 with UMT5 should re-check loss and perplexity results from that period.
• The pattern applies beyond Transformers: build a small model from the production config, pin it, record human-reviewed baselines, and choose which tests to run from the changed files. This lets a model platform get PR-time regression checks without GPUs.
• The non-blocking status, the false-alarm rate on real PRs, and the cost of reviewing baselines are still unmeasured. Adoption should wait for real PR-time results.