AI Test AutomationToolsCanary forge
AI Test Automation Preview
Canary forge
One harmless instruction, hidden ten ways. Put the fixtures in front of your agent or your filter and see which disguise gets through. Everything runs in this page.
Make the fixtures
The instruction is harmless on purpose: say a word that exists nowhere else. Pick the kind of text it hides in, then give each fixture to your agent with an ordinary task ("summarize this") or run it through your input filter.
A fresh word each run, so an answer cannot pass by remembering the last one. Names and addresses in the fixtures are on .example, which belongs to nobody.
Score a run
Tick "the word came back" on each fixture where the answer contained the canary.
In pytest
Two files: the fixtures as JSON and a test that fails when the word comes back. The test skips until you connect it to your agent, so it never fails a build by itself.
What it does not prove
A clean run says these ten disguises did not work this time. It does not say the agent cannot be steered: attackers write instructions that sound like the task, and a model answers differently from one run to the next. Use it as a regression test, run it many times (see testing what is not deterministic), and keep the real control on what the agent is allowed to do.