IncidentSupply chain
Microsoft: about 1 percent poisoned fine-tuning data, or a tampered open-weight model, can make a coding agent run an attacker's commands on a trigger
Microsoft Security Research and the Microsoft AI Red Team (Amanda Minnich, Amit Eliahu, Blake Bullwinkel, Dor Edry and Giorgio Severi) published on November 12, 2026 two studies of backdoors that stay dormant until a trigger appears and then make an agent invoke tools or run commands. In dataset poisoning, 90 crafted examples hidden among about 9,000 terminal-agent conversations across 10 mock harnesses, about 1 percent of the data, taught a fine-tuned gpt-4.1-mini to find a command tool and misuse it on a routine request: unsafe actions in 78.0 and 77.8 percent of 500 trials on the two harnesses it had seen, 11.0 percent on one unseen harness and 0 percent on another, and 52 to 70 percent with paraphrased triggers. In model poisoning, an attacker who controls training before publishing an open-weight checkpoint trained Qwen3.5-4B and Qwen3.5-9B on about 2,400 coding trajectories; the strongest setup reached 84.1 to 94.8 percent on unseen repositories, and it transferred to Claude Code (97.0 and 91.0 percent), Pi (74.0 and 77.0) and Codex CLI (37.3 and 63.0). Removing the reasoning traces cut success to 1.0 and 0 percent. The team also found rows in a public dataset that paired harmless requests with unsafe tool calls, including fetching and running a remote script; it could not determine the publisher's intent or whether those rows would affect a model, and it says there is no evidence of its backdoor design in the wild.
Replay the attack step by step in the Learning Center
How it happened
- Poisoned examples in a fine-tuning set pair an ordinary request with a harmful command, so the tuned model learns to run it when the trigger shows up.
- Or the attacker trains the backdoor into an open-weight checkpoint before publishing it.
- The behavior carries over to agent harnesses the model was never trained in.
Impact
No victims are known; the studies show a dataset or model from outside can carry a hidden trigger that makes a coding agent run commands, and a public dataset already holds suspicious rows.
The fix
Microsoft advises pinning reviewed dataset versions by cryptographic hash and checking that tool calls follow from the request, treating third-party checkpoints as untrusted, recording provenance and testing in the production harness, reducing tool access, scoping credentials, isolating execution, requiring approval for high-impact actions, and logging and alerting on tool calls not tied to the task.
Lessons
- A dataset or a checkpoint is code you run; know where it came from and pin it.
- Test a model in the harness it will run in, with the tools it will have.
- Approval and logging at the tool layer catch what a hidden trigger makes the model do.