Skip to the content
AI Test AutomationPreview AI Security Learning Center

AI Test AutomationLearnAI and agents

AI and agents Preview

AI agents for testing: faster, and still trustworthy

AI agents now plan, write, run and repair tests, and AI coding assistants write much of the code being tested. How teams use them, the tools established and new, the practices Anthropic documents for testing with Claude Code, and how to keep AI speed from costing quality.

How teams use them

AI agents in the testing workflow today

Each pattern with the tools that do it and the risk the sources name. None of them removes the need for a person who owns what the tests mean.

89%

89% of organizations in the World Quality Report 2025-26 (more than 2,000 senior executives in 22 countries and 10 sectors) are piloting or deploying Gen AI in quality engineering: 37% in production and 52% in pilots. The report says test case design and requirements refinement now lead adoption.

Capgemini, Sogeti and OpenText (World Quality Report 2025-26), 2025

Generating and improving unit tests

An assistant or agent writes unit tests for existing code, runs them, and keeps only the ones that build, pass reliably and add coverage; Meta's ACH goes further by planting simulated bugs (mutants) of a kind an engineer describes and keeping only tests that catch them. Meta reports engineers accepted 73% of TestGen-LLM's recommendations, and Uber says its AutoCover agent now writes about 11% of all new tests reviewed and added to its codebase.

Tools
GitHub Copilot (Chat and agent mode), Diffblue Cover (Java), Uber AutoCover (internal), Meta TestGen-LLM and ACH (internal)
Watch out for
A generated test that passes only shows the code does what it does today. Meta writes that without a test oracle its tool "cannot automatically determine whether a failing test has found a bug, or whether it merely contains an incorrect test assertion", and Diffblue says its suites "capture the current behavior of the code".

Sources: Meta: Automated Unit Test Improvement using Large Language Models at Meta (TestGen-LLM), arXiv, Feb 2024 · Meta TestGen-LLM paper, full text (test oracle limit, filters) · Meta: Mutation-Guided LLM-based Test Generation at Meta (ACH), arXiv, Jan 2025 · Uber and University of Washington: AutoCover, ICSE-SEIP 2026 · Diffblue Cover docs: FAQs

Planning and writing end-to-end tests from a plan

A planner agent explores the running app and writes a test plan in Markdown, then a generator agent turns each scenario into Playwright test code, checking selectors and assertions live as it performs the steps. Playwright shipped these agents in version 1.56 (October 6, 2025), and GitHub Copilot Chat and Cypress cy.prompt offer similar plain-language-to-test generation.

Tools
Playwright Test Agents (planner, generator), GitHub Copilot Chat with Playwright, Cypress cy.prompt
Watch out for
Output varies from run to run and is often wrong at first. Playwright says "Generated tests may include initial errors that can be healed automatically by the healer agent", GitHub notes "Copilot Chat responses are non-deterministic", and Cypress lists gaps such as "'not.exist' assertions from natural language are not supported".

Sources: Playwright docs: Test Agents · Playwright v1.56.0 release notes (Oct 6, 2025) · GitHub Docs: Creating end-to-end tests for a webpage (Copilot Chat cookbook)

Self-healing locators and tests

When a test fails because a button or field changed, the tool finds the closest matching element or has an agent patch the locator, wait or data, then reruns the test. Playwright's healer agent, mabl auto-heal, Applitools Execution Cloud and Cypress cy.prompt all do this, and Applitools keeps using the healed selector in future runs.

Tools
Playwright healer agent, mabl auto-heal, Applitools Execution Cloud self-healing, Cypress cy.prompt self-healing
Watch out for
A heal can turn a real regression green. Playwright's healer can return "a skipped test if the healer believes that functionality is broken", Applitools picks "the element that most closely resembles the missing element", and mabl writes about cases where "bugs reach production that tests should have caught but didn't due to auto-heal modifications".

Sources: Playwright docs: Test Agents (healer) · mabl blog: The Future of Auto Heal Testing with Adaptive AI and Continuous Learning (Nov 19, 2025) · Applitools docs: Execution Cloud self-healing

Goal-driven exploratory testing by browser and mobile agents

Instead of following a fixed script, an agent is given a goal (at Uber, requesting and completing a trip) and decides what to tap or click from what it sees on the screen, or explores a web app through Playwright MCP. Uber says its DragonCrawl agent ran with "99%+ stability" in November and December 2023 and "blocked ten high-priority bugs from impacting customers" in its first three months.

Tools
Uber DragonCrawl (internal), Playwright MCP, Playwright planner agent, Claude computer use
Watch out for
The agent can wander, loop or make things up: Uber quotes the view that LLMs "kind of spew nonsense sometimes", hit loops "(i.e., scrolling up and down in a feed)", and in some cities its agent requested scheduled trips instead of regular ones. Agents that read web pages can be steered by them and should not hold real logins: Anthropic warns that "In some circumstances, Claude will follow commands found in content even when they conflict with your instructions" and advises "Avoiding giving the model access to sensitive data, such as account login information", and Microsoft says Playwright MCP "is not a security boundary".

Sources: Uber Engineering: DragonCrawl, generative AI for high-quality mobile testing (Apr 23, 2024) · Microsoft: Playwright MCP README (security notes) · Anthropic docs: Computer use tool, security considerations

Generating test data and mocks

Testers describe the records they need in plain language, and an agent writes scripts that fill databases, files or mock objects with realistic fake data. Tonic's Fabricate Data Agent does this for whole databases, and GitHub documents Copilot Chat "creating mock objects that you can use for unit tests".

Tools
Tonic Fabricate Data Agent, GitHub Copilot Chat (mock objects), Cypress cy.prompt placeholders (keep secrets out of prompts)
Watch out for
Fake data modeled on real data can still expose real people. The UK ICO says "Some synthetic data generation methods have been shown to be vulnerable to model inversion attacks, membership inference attacks and attribute disclosure risk", and whether the output is still personal data "depends on whether the personal information on which you model the synthetic data can be inferred from the synthetic data itself." Tonic notes an LLM trained on sensitive data "may inadvertently memorize specific details and regurgitate them".

Sources: Tonic docs: About the Fabricate Data Agent · GitHub Docs: Testing code (Copilot Chat cookbook, mock objects) · UK ICO: Synthetic data (privacy-enhancing technologies guidance) · Tonic guide: LLM data privacy

Triaging and fixing flaky tests and failures

An agent reads a failing test, its run history and logs, works out a likely cause such as timing, environment or test order, and opens a draft pull request with a fix or explains the root cause. Uber reports its FlakyGuard system repaired 47.6% of reproducible flaky tests with 51.8% of fixes accepted by developers, and Bitbucket Pipelines has a beta agent that does the same and raises a draft pull request.

Tools
Uber FlakyGuard (internal), Bitbucket Pipelines flaky test fixer agent (beta), Atlassian Flakinator (statistical detection and quarantine)
Watch out for
Making a test pass is not the same as fixing it. A Cypress post on DEV Community puts it plainly: "Increase the Cypress timeout and the test goes green, but you have not removed the flake. You have hidden a real bug and taught your suite to lie." Atlassian's agent is "currently in beta" and raises a draft pull request for a person to review, and at Uber only about half of the proposed fixes were accepted.

Sources: Uber and UT Austin: FlakyGuard, Automatically Fixing Flaky Tests at Industry Scale (arXiv, Nov 2025) · Atlassian Support: AI-driven flaky test remediation (Bitbucket Pipelines) · Atlassian Engineering: Taming test flakiness, Flakinator (Dec 8, 2025) · Cypress on DEV Community: Teaching an agent to triage flaky tests

Reviewing pull requests for missing tests

An AI reviewer comments on each pull request, and teams can write their "test expectations" into a repository instructions file that Copilot follows, while GitHub Code Quality adds "Test-coverage metrics on pull requests". Some reviewers also write the missing tests: in CodeRabbit you comment "@coderabbitai generate unit tests" and receive them as a separate pull request or a new commit.

Tools
GitHub Copilot code review, GitHub Code Quality (coverage on pull requests), CodeRabbit
Watch out for
The reviewer can miss gaps and invent problems. GitHub says "Copilot may not identify all of the problems that are present in code, especially where changes are large or complex", that it "has a risk of hallucination", and that it "should be supplemented with careful human code review." A coverage number only shows which lines ran: "Coverage shows which code executed, not whether the assertions are correct" (VS Code docs).

Sources: GitHub Docs: About GitHub Copilot code review · GitHub Docs: Responsible use of GitHub Copilot code review · CodeRabbit docs: Unit test generation · VS Code docs: Test existing code with AI

Security testing by AI agents (pentest, fuzzing, bug hunting)

Agents attack running web apps, or read code and write fuzz targets, to find vulnerabilities and then try to prove each one; XBOW uses "validators, automated peer reviewers that confirm each vulnerability XBOW uncovers". XBOW reached the top of HackerOne's US ranking in 2025, and Google's AI-written fuzz targets found 26 new vulnerabilities, including CVE-2024-9143 in OpenSSL.

Tools
XBOW, Google Big Sleep, OSS-Fuzz with AI-generated fuzz targets, Claude Code security review GitHub Action
Watch out for
Noise and trust are the problems. Of nearly 1,060 XBOW submissions, 208 were duplicates, 209 informative and 36 not applicable, and its team reviewed findings before submitting to follow HackerOne's policy on automated tools; curl's Daniel Stenberg wrote in July 2025 that AI slop made up "about 20% of all submissions" so far that year, with each report engaging "3-4 persons".

Sources: XBOW blog: how XBOW reached the top of the HackerOne US leaderboard (Jun 24, 2025) · Google Security Blog: Leveling Up Fuzzing, finding more vulnerabilities with AI (Nov 20, 2024) · Google Project Zero: From Naptime to Big Sleep (Nov 2024) · Daniel Stenberg: Death by a thousand slops (July 14, 2025)

The tools

Established and up and coming

Described from their own pages, never ranked by opinion; open-source projects carry their GitHub stars, refreshed every morning. No tool here paid to be listed.

Choose tools that leave you code you own

Prefer tools that leave you real test code you own and can run without them: QA Wolf says its tests are Playwright and Appium code, "open source, exportable, and yours to keep", and Checksum says its tests are delivered as Playwright code that lives in your repository. A hosted tool can change its terms or close.

Sources: QA Wolf · Checksum

Agents that plan, write and fix tests

Playwright Test Agents

Microsoft

Three agent definitions that ship with Playwright: the planner "explores the app and produces a Markdown test plan", the generator turns that plan into Playwright Test files, and the healer "executes the test suite and automatically repairs failing tests".

  • Introduced in Playwright v1.56, published on GitHub on 2025-10-06.
  • Set up with npx playwright init-agents --loop=vscode (or claude, codex, opencode); the docs say the definitions "should be regenerated whenever Playwright is updated".
  • When the healer believes the feature itself is broken, it can leave "a skipped test" rather than force a pass.
Open source · Apache-2.0★ 100.5k

Sources: Playwright docs: Test agents · Playwright release notes (v1.56) · GitHub release v1.56.0

Checksum

Checksum

Generates, runs and maintains end-to-end and API tests with AI. Tests "are delivered as real code: Playwright for end-to-end tests that lives in your repository."

  • When a selector or flow changes, Checksum "detects the failure, fixes the test, and opens a PR for your review."
  • Monitors production errors so that "every bug becomes a test".
  • Pricing: "No per-seat fees. No per-run charges." It is tied to how many tests Checksum maintains.
Commercial

Sources: Homepage

Cypress cy.prompt

Cypress.io

An AI-powered Cypress command that "turns natural language test steps into executable Cypress commands". The generated code is cached, and you can replace the cy.prompt call with the generated commands for version-controlled tests.

  • Introduced in Cypress 15.4.0 and moved to beta in 15.13.0; it "Requires Cypress Cloud account and/or valid record key", although the Cypress app itself is MIT licensed (cypress-io/cypress).
  • Limits: end-to-end tests only, "Chromium-based browsers only (Chrome, Edge)", and no API testing.
  • If cached selectors fail, "Cypress regenerates code for that step automatically on the next run"; password, credit card and hidden field values are excluded before data is sent to the AI model.
Commercial

Sources: Cypress docs: cy.prompt · Cypress docs: AI test generation guide · Cypress app repository (MIT)

GitHub Copilot (tests and cloud agent)

GitHub

GitHub's AI coding assistant. In chat, /tests will "Generate unit tests for the selected code", and the Copilot cloud agent can "edit files and run tests and linters in an ephemeral cloud development environment".

  • /tests is listed for VS Code, Visual Studio, JetBrains and Xcode; /fixTestFailure ("Find and fix a failing test") is listed for VS Code only.
  • "Each Copilot cloud agent session has a maximum execution time of 59 minutes", and it "can open one pull request per task".
  • GitHub's own guidance: "Review and test the output before using it in production."
Commercial

Sources: GitHub Docs: Copilot Chat cheat sheet · GitHub Docs: Writing tests with GitHub Copilot · GitHub Docs: About Copilot cloud agent

Momentic

Momentic

An AI testing platform for web, Android and iOS apps. Its agent, Mo, tests a running app from a URL or mobile build plus a short brief, and regression tests are steps and assertions written "in plain-English YAML".

  • "Momentic finds controls again when the UI changes during a run"; a lasting fix is proposed as a test-file repair for you to review.
  • Setup is npx @momentic/wizard@latest; repository tests can run as GitHub Actions checks.
  • Its homepage marks "iOS testing in beta".
Commercial

Sources: Homepage · Docs

QA Wolf

QA Wolf

An agentic QA platform ("Agentic QA that just works") where a Mapping AI "autonomously explores your app and documents its workflows" and an Automation AI writes test code for web, iOS and Android, which then runs in parallel on deploy.

  • Tests are Playwright and Appium code: "open source, exportable, and yours to keep."
  • A fully managed option, Coverage-as-a-Service, "embeds full-time QA engineers with your team".
  • Covers web apps, Electron apps, Android phones and tablets, and iPhones and iPads.
Commercial

Sources: Homepage

Open-source browser agents

Browser Use

Browser Use

An open-source Python library that lets an AI agent operate a real web browser to finish a task you describe in plain language. Its GitHub description reads "Agents that use the browser."

  • Needs "Python >= 3.11" and installs with uv add browser-use.
  • Works with its own BU2 model, with OpenAI, Anthropic and Google models through the ChatOpenAI, ChatAnthropic and ChatGoogle wrappers, or with a local model through Ollama.
  • MIT licensed and runs on your own machine or in the paid Browser Use Cloud; its site lists "Testing & QA" as a showcase category.
Open source · MIT★ 121.4k

Sources: GitHub repository · README · Homepage

Stagehand

Browserbase

An SDK for browser agents that lets code mix normal browser steps with natural-language actions (act, observe, extract). Its site puts it this way: "Playwright was built for testing, Stagehand is built for agents."

  • Self-healing actions: "When sites change, we detect it and refresh how the actions happen on the page automatically."
  • extract() "returns schema-validated data".
  • "One complete browser driver across TypeScript, Python, and Go"; runs locally with Chrome or on Browserbase's hosted browsers.
Open source · MIT★ 26.1k

Sources: GitHub repository · README · Homepage

Skyvern

Skyvern

"Skyvern automates browser-based workflows using LLMs and computer vision." Instead of relying only on XPath selectors, it uses vision LLMs to read and act on pages, and its site lists Testing and QA as a use case ("Browser testing that adapts to UI changes").

  • README: "Skyvern is a Playwright extension that adds AI-powered browser automation."
  • page.validate(prompt) checks page state and returns true or false, for example whether a user is logged in.
  • Licensed AGPL-3.0, with a separate managed Skyvern Cloud.
Open source · AGPL-3.0★ 23.4k

Sources: GitHub repository · README · Homepage

Midscene.js

ByteDance (web-infra-dev)

A vision-based GUI agent for end-to-end testing and automation in natural language across web, mobile and desktop apps (GitHub: "GUI Agent for E2E Testing"). Like a person, it "locates elements by their appearance and position" instead of by selectors.

  • Runs on the web through Playwright or Puppeteer, on Android, iOS and HarmonyOS devices and simulators, and on desktop apps.
  • Flows can be written in YAML; aiQuery extracts data, aiAssert checks results, and HTML reports show screenshots and the AI's decisions.
  • MIT licensed, copyright ByteDance; model choice is open ("Supports DeepSeek, Qwen, GPT, Gemini, Kimi, and more").
Open source · MIT★ 15.6k

Sources: GitHub repository · README · Homepage

Unit and API test generation

Diffblue Cover

Diffblue

Described in its docs as "a reinforcement learning AI platform that automatically writes comprehensive, human-like Java unit tests", aimed at saving developer time, raising coverage and reducing regression risk.

  • Comes "as an IntelliJ IDE plugin, a CLI application, and a CI integration".
  • Diffblue Agents, a separate product, works through your existing AI coding agent platform; its Testing Agent does "autonomous regression unit test generation for Java and Python projects".
  • Diffblue Agents says "only verified output is committed to your current branch" and "Anything that fails validation is rolled back."
Commercial

Sources: Diffblue Cover docs · Diffblue Agents docs · Homepage

Qodo

Qodo (formerly CodiumAI)

Qodo, which started as CodiumAI, now describes itself as "an AI code review and governance platform for engineering teams shipping at the speed AI writes code." Its testing page lists generating and updating tests "based on code diffs or commits".

  • Renamed in a blog post dated September 30, 2024, "Introducing Qodo (formerly Codium)"; the IDE plugin Codiumate became Qodo Gen for "context-aware code and test generation".
  • The testing page also lists finding "gaps in test coverage, from single pull requests to full repositories" and the ability to "Run tests and auto-fix failures directly in the IDE".
  • Its open-source test generator, Qodo Cover (formerly Cover-Agent), says on GitHub: "This repository is no longer maintained."
Commercial

Sources: Homepage · Qodo blog: rename announcement · Qodo: automated testing solution page · Qodo Cover repository

API tests from real traffic

Keploy

Keploy

An open-source "API and integration testing tool" that "records API calls, database queries, and streaming events" and replays them as tests with data mocks, so tests come from real traffic instead of being written by hand.

  • Keploy "uses eBPF to capture traffic at the network layer", so it needs no application code changes and works across languages.
  • "Keploy's core local record-and-replay platform is open source under the Apache 2.0 license"; enterprise features add recording from Kubernetes, staging and production.
  • Its site says AI generates API tests "from traffic, specifications, and application context".
Open source · Apache-2.0★ 19.9k

Sources: GitHub repository · README · Homepage

Postman Agent Mode

Postman

"Postman's AI-powered automation, built into the Postman app." You describe a task in natural language and it can write and update API tests, generating post-response test scripts for your requests.

  • Docs: "Tell Agent Mode what to do in natural language, and it generates post-response scripts for you."
  • Product page: "Postman AI can fix broken requests, write test scripts, generate docs, organize collections, and more."
  • "This feature is available to users who consent to using AI."
Commercial

Sources: Postman docs: Agent Mode overview · Postman docs: Write test scripts · Product page

Visual testing

Applitools

Applitools

Visual AI testing. Applitools Eyes adds visual and functional checks to web and mobile tests, and Applitools Autonomous lets teams "Build, run, and maintain tests without writing code".

  • Autonomous "crawls your sitemap to build a starter test suite with Visual AI checkpoints already in place", and an LLM turns plain-English steps into test steps.
  • The Applitools MCP server helps "create, update, review, and resolve visual tests" from Cursor, GitHub Copilot, Cline and Claude Code; its setup tools support only the Playwright Fixtures SDK (JavaScript and TypeScript).
Commercial

Sources: Homepage · Applitools Autonomous · Docs: Applitools MCP Server

Meticulous

Meticulous

Records real sessions through a script tag in local, staging and preview environments and turns them into "visual end-to-end tests". On each pull request you "see the impact it will have across user workflows before you hit merge".

  • By default it "mocks out the responses from your backend" by replaying recorded responses, so tests are side-effect free.
  • Built "from the Chromium level up with a deterministic scheduling engine" to avoid flaky runs.
  • Supports Angular, Next.js, Vue.js, React, Svelte Kit and Vite, with CI on GitHub, GitLab, Netlify and CircleCI.
Commercial

Sources: Homepage

Test platforms with AI built in

BrowserStack AI agents

BrowserStack

A set of AI agents in BrowserStack's testing cloud, billed as "AI agents at every step of testing lifecycle", from turning requirements into test cases to healing broken locators at run time.

  • Agents listed: Test Case Generator, Low-Code Authoring, Test Failure Analysis, A11y Issue Detection, Self-healing, Visual Review, A11y Issue Remediation and Test Deduplication.
  • Five of them, Test Case Generator, Low-Code Authoring, Self-Healing, A11y Issue Detection and Visual Review, were announced in a press release dated June 30, 2025.
  • Several agents can be called through the BrowserStack MCP server from "GitHub Copilot, Claude, or Cursor".
Commercial

Sources: BrowserStack AI page · Press release (June 30, 2025) · Docs: AI agents via the MCP server

Functionize

Functionize

"The Agentic Quality Platform": you describe what your app should do and its Studio "writes the tests, runs them, and keeps them current as the app changes."

  • Design principle stated on its site: "Generative intent, deterministic core".
  • On a failed run, "Studio investigates the run and names the failing step and the cause", and "Every change is a restore point."
  • QuickConnect reaches private apps through "a secure per-session tunnel with no inbound ports".
Commercial

Sources: Homepage · Studio overview · AI agents page

KaneAI

TestMu AI (formerly LambdaTest)

An AI testing agent to "Plan, author, and evolve end-to-end tests for web and mobile apps in natural language".

  • "KaneAI exports generated tests to Selenium, Playwright, Cypress, and Appium in multiple languages".
  • LambdaTest now goes by "TestMu AI (Formerly LambdaTest)"; lambdatest.com/kane-ai redirects (301) to testmuai.com/kane-ai/.
Commercial

Sources: KaneAI product page · Old LambdaTest URL (redirects)

Katalon

Katalon

"The AI platform for software quality", covering web, mobile, API and desktop apps. "From a requirement, a Jira ticket or a prompt, AI writes test cases for you to review and approve."

  • Self-healing and failure analysis, with a human check: "Katalon AI fixes the test and you review it."
  • Can be driven "directly from your own AI coding agent like Copilot, Claude, Codex, Cursor and others."
  • "You can start free with no credit card".
Commercial

Sources: Homepage

mabl

mabl

An agentic testing platform for web, mobile and API testing. You "Describe a flow, paste a Jira ticket, or connect via Atlassian Rovo" and "mabl authors the test", then runs it and deals with failures mid-run.

  • Each failure is classified as a "Real regression, app change, or noise".
  • The mabl MCP server (blog post dated August 20, 2025) lets AI assistants create and run mabl tests from the IDE, for example from a Jira ticket.
  • Homepage: "Your agents know the code. mabl independently verifies your application".
Commercial

Sources: Homepage · Features page · mabl blog: the mabl MCP server

Testim

Tricentis

Tricentis Testim offers "AI-powered testing for custom web, mobile, and Salesforce apps", using AI-based smart locators so UI tests keep working as the app changes.

  • "Smart locators understand your app, identify elements, and automatically self-heal".
  • "Agentic Test Automation makes it possible to build all of your Salesforce tests without code and using natural language."
  • Part of the Tricentis portfolio, alongside Tosca.
Commercial

Sources: Homepage

testRigor

testRigor

A generative AI test automation tool where "you can use free-flowing plain English to build test automation", which it then executes as written.

  • "Because testRigor tests are not dependent on XPath, they are ultra-stable."
  • Lists web, mobile, desktop, API, email, SMS and phone calls, 2FA and mainframe testing, plus AI-native apps and features.
Commercial

Sources: Homepage

Tricentis Tosca

Tricentis

An enterprise test automation platform. Its Agentic Test Automation is "an AI assistant for Tosca Cloud" that "helps you create test cases with natural language through a chat window", for web, SAP Fiori, SAP GUI and Salesforce apps.

  • Two modes: Co-create (the default, step by step) and Autonomous, which "runs all steps in sequence until the test is complete, without your input."
  • Drives apps through TBox (technical properties) or Vision AI, which "uses visual elements to recognize controls"; it "can create new test cases but can't update existing ones."
  • The Tosca MCP Server lets AI assistants "create, organize, update, use, and run test assets"; tested with Claude Code, Claude Desktop, Cursor and VS Code.
Commercial

Sources: Tosca Cloud docs: Agentic Test Automation · Tosca Cloud docs: Generate test cases · Tosca 2026.1 docs: MCP integration

With Claude Code

Testing with Claude Code: the practices Anthropic documents

From Anthropic's own Claude Code documentation. The thread through all of them: give Claude a check it can run, keep the writer and the grader apart, and enforce the limits in settings rather than asking for them.

Give Claude a check it can run, and ask for proof

Claude stops when the work looks done. Without a check it can run, the docs say "you become the verification loop: every mistake waits for you to notice it." A check is anything that returns a pass or fail Claude can read: a test suite, a build exit code, a linter, a script that diffs output against a fixture, or a browser screenshot compared against a design.

How: Put the check in the prompt, as the docs show: "write a validateEmail function. example test cases: user@example.com is true, invalid is false, user@.com is false. run the tests after implementing". For UI work, paste the design and ask Claude to "take a screenshot of the result and compare it to the original. list differences and fix them".

Claude Code docs: Best practices for Claude Code

Reproduce the bug with a failing test, then fix the root cause

Give Claude the symptom, the likely location and what 'fixed' looks like, and have it write a failing test before it changes the code. Be specific about the scenario and your testing preferences, and ask for root-cause fixes instead of suppressed errors.

How: Docs prompts to copy: "users report that login fails after session timeout. check the auth flow in src/auth/, especially token refresh. write a failing test that reproduces the issue, then fix it". For a new test: "write a test for foo.py covering the edge case where the user is logged out. avoid mocks." For a broken build: "fix it and verify the build succeeds. address the root cause, don't suppress the error".

Claude Code docs: Best practices for Claude Code

Separate the writer from the grader

The docs say "A fresh context improves code review since Claude won't be biased toward code it just wrote," and apply the same idea to tests: "have one Claude write tests, then another write code to pass them." Before you count unattended work as done, a reviewer subagent "sees only the diff and the criteria you give it, not the reasoning that produced the change", so the agent doing the work is not the one grading it.

How: Run a Writer session and a Reviewer session in parallel (worktrees keep their edits apart). Run the bundled /code-review skill, which "reviews the current diff for bugs in a fresh subagent".

Claude Code docs: Best practices for Claude Code

Write exact test commands into CLAUDE.md

A project CLAUDE.md is where "build and test commands" and common workflows go, shared with the team through version control. Claude treats CLAUDE.md "as context, not enforced configuration", so instructions must be concrete enough to verify.

How: Run /init to generate a starter; it creates a file "with build commands, test instructions, and project conventions it discovers." Write "Run `npm test` before committing" instead of "Test your changes". Keep each file under 200 lines ("Longer files consume more context and reduce adherence").

Claude Code docs: How Claude remembers your project (CLAUDE.md)

Make 'do not edit the tests' a rule Claude Code enforces

"Permission rules are enforced by Claude Code, not by the model. Instructions in your prompt or CLAUDE.md shape what Claude tries to do, but they don't change what Claude Code allows." Rules are evaluated "deny, then ask, then allow", and "An allow rule can't carve an exception out of a deny rule."

How: Once the tests are reviewed and committed, add an Edit deny rule for the test folder to the settings the implementation session uses, for example Edit(/tests/**). Put the rule in the project's .claude/settings.json or .claude/settings.local.json, where a leading slash resolves to the primary working directory, as in the docs' Edit(/src/**/*.ts) example; in user settings it resolves to ~/.claude. The docs note that an Edit deny rule covers Claude's file tools, the Bash file commands Claude Code recognizes and redirections, but not a script that opens files itself; to stop every process from writing there, enable the sandbox.

Claude Code docs: Configure permissions

Use hooks for checks that must run every time

Hooks are user-defined shell commands that Claude Code runs at set points in its lifecycle, which "gives you deterministic control: certain actions always happen rather than relying on the LLM to choose to run them." A PreToolUse deny blocks the tool "even in bypassPermissions mode or with --dangerously-skip-permissions".

How: In .claude/settings.json: a PostToolUse hook with matcher Edit|Write runs your formatter or linter on each edited file (the docs' example pipes the file path to npx prettier --write). A PreToolUse hook script that exits with code 2 blocks edits to protected paths and tells Claude why (the docs' .claude/hooks/protect-files.sh lists .env, package-lock.json and .git/; add your test paths to that list).

Claude Code docs: Automate actions with hooks

Set a /goal with a measurable end state and a 'tests unchanged' constraint

/goal sets a completion condition and Claude keeps working toward it. After each turn a separate model checks the condition, "so completion is decided by a fresh model rather than the one doing the work." The evaluator "doesn't run commands or read files independently", so the condition must be something Claude's own output in the conversation can show.

How: Docs example: /goal all tests in test/auth pass and the lint step is clean. Give the condition "One measurable end state", "A stated check" such as "`npm test` exits 0", and constraints such as "no other test file is modified".

Claude Code docs: Keep Claude working toward a goal

Run the test suite and code review in subagents

"Running tests, fetching documentation, or processing log files can consume significant context. By delegating these to a subagent, the verbose output stays in the subagent's context while only the relevant summary returns to your main conversation."

How: Prompt: "Use a subagent to run the test suite and report only the failing tests with their error messages". Define project subagents in .claude/agents/ and check them into version control.

Claude Code docs: Subagents

Tell Claude that tests check the solution, they do not define it

Anthropic's prompting guide warns that "Claude can sometimes focus too heavily on making tests pass at the expense of more general solutions". For long tasks across several context windows it suggests tracking tests in a structured file such as tests.json and reminding Claude: "It is unacceptable to remove or edit tests because this could lead to missing or buggy functionality."

How: Add the guide's wording to the prompt or CLAUDE.md: "Implement a solution that works correctly for all valid inputs, not just the test cases. Do not hard-code values or create solutions that only work for specific test inputs." "Tests are there to verify correctness, not to define the solution." "If the task is unreasonable or infeasible, or if any of the tests are incorrect, please inform me rather than working around them." The same guide's safety prompt adds: "don't bypass safety checks (e.g. --no-verify)".

Claude Platform docs: Prompting best practices

Run Claude headless in CI with pinned-down permissions

claude -p runs Claude non-interactively for CI, pre-commit hooks and scripts. It "exits with code 0 on success and a non-zero code when the run fails, so your scripts can branch on the exit status."

How: Docs example: claude -p "Run the test suite and fix any failures" --allowedTools "Bash,Read,Edit". In CI add --bare, which skips auto-discovery of hooks, skills, subagents, plugins, MCP servers, auto memory and CLAUDE.md and is "useful for CI and scripts where you need the same result on every machine".

Claude Code docs: Run Claude Code programmatically (non-interactive mode)

Add Claude review to pull requests, but keep CI and a human merge

anthropics/claude-code-action@v1 runs Claude Code inside GitHub workflows, when someone mentions @claude or on events such as pull_request. The docs' advice: "Grant the workflow only the permissions it needs, and review Claude's changes before merging."

How: Run /install-github-app in Claude Code (or copy examples/claude.yml). For automatic reviews, use the docs' code-review workflow, which installs the code-review plugin and runs /code-review:code-review --comment when a pull request is opened, updated, reopened or marked ready for review; Claude skips drafts, closed pull requests and some others.

Claude Code docs: Claude Code GitHub Actions

Stay the approver, and isolate risky runs

The security guide is explicit: "You're responsible for reviewing proposed code and commands for safety before approval." For sensitive code it lists "Review all suggested changes before approval", "Use project-specific permission settings for sensitive repositories" and "Regularly audit your permission settings with /permissions".

How: With untrusted content: "Review suggested commands before approval", "Avoid piping untrusted content directly to Claude" and "Verify proposed changes to critical files". For extra isolation, run the whole Claude Code process inside the sandbox runtime or a dev container, and use virtual machines to run scripts and make tool calls, especially when they interact with external web services.

Claude Code docs: Security

Speed without losing quality

Make AI-assisted testing better, not just faster

AI makes writing tests cheap, which makes it easy to have many tests that prove little. These practices, from GitHub, martinfowler.com, the Thoughtworks Technology Radar, Meta and Google, keep them honest.

  1. Review AI-written tests against the requirements, not just the green bar

    GitHub's guide to raising test coverage with Copilot lists what developers should not do. First on the list: "Accept generated tests without reviewing logic. Make sure the tests reflect actual requirements and handle realistic inputs and outputs." Also: "Rely on Copilot to guess undocumented business rules. Always provide context through prompts or comments."

    GitHub also warns against skipping edge assertions: "If you only test 'happy paths,' you risk missing regressions." And it keeps the judgment with people, listing as a mistake: "Treat Copilot as a substitute for human code reviews. Copilot accelerates the process, but you still need to apply engineering judgment."

    GitHub Docs: Increasing test coverage in your company with GitHub Copilot

  2. Catch tests that only repeat what the code already does

    In an informal evaluation on martinfowler.com (Birgitta Böckeler, listed 10 August 2026), some agent-written tests "checked the implementation's output against itself, re-running the same code to produce the 'expected' answer", and one run had an "active TOTAL-row bug (headcount summed into dollars) enshrined by a test". Agents also "frequently overshot and implemented more than the current test demanded".

    A test that takes its expected value from the code under test passes even when the code is wrong, as the enshrined TOTAL-row bug shows. Prompting for TDD did not fix this on its own: "Mutation scores across the solutions didn't show any signals that TDD runs produced meaningfully better mutation scores than non-TDD runs." The author now writes: "I monitor and improve regression quality with the help of mutation testing, instead of giving elaborate TDD instructions and hoping for the best."

    martinfowler.com: TDD inside the agent loop - theater or actual value? (Birgitta Böckeler)

  3. Use mutation testing to catch 'perpetually green' tests

    The Thoughtworks Technology Radar puts mutation testing in Trial (April 2026) and calls it "the most honest signal for evaluating the real fault-detection capability of a test suite". It introduces deliberate bugs into the code to verify that tests fail when behavior breaks, using tools "such as Stryker, Pitest or cargo-mutants", "particularly in core domain logic".

    "With AI-generated test cases now commonplace", the Radar says mutation testing catches "perpetually green" tests: those "that pass regardless of logic changes due to missing assertions or decoupled mocks". Coverage can hide them, because "high coverage percentages can mask logically hollow tests or generated code that has never been meaningfully asserted."

    Thoughtworks Technology Radar: Mutation testing (Trial, Apr 2026)

  4. Read the mutation score, not only the coverage number

    Stryker "introduces changes to your code, then runs your unit tests against the changed code." "If your tests fail then the mutant is killed. If your tests passed, the mutant survived." It comes as StrykerJS (JavaScript and TypeScript), Stryker.NET (C#) and Stryker4s (Scala).

    "Code coverage doesn't tell you everything about the effectiveness of your tests." Stryker's measure is the kill rate: "The higher the percentage of mutants killed, the more effective your tests are." Each surviving mutant is a planted bug that an AI-written suite let through.

    Stryker Mutator docs

  5. Treat an agent's 'tests pass' as a claim, and check for skipped or deleted tests

    In Birgitta Böckeler's experiment on agent autonomy (martinfowler.com, 05 August 2025), "AI frequently claimed the build and tests were successful and moved on to the next step, even though they were not, and even though our instructions explicitly stated that the task is not done if build or tests are failing." Under brute-force fixes she lists "I can't get the test to work right now, let's skip it for now and move on to the next task".

    Deterministic checkpoints that block until tests are green help, but she expects agents "could still delete or skip tests to get beyond that checkpoint", and asks: "And who reviews the generated tests, especially as the application gets larger?" Her conclusion: "AI is not ready to create and maintain a maintainable business software codebase without human oversight."

    martinfowler.com: How far can we push AI autonomy in code generation? (Birgitta Böckeler)

  6. Wire deterministic checks into the agent's loop, before the commit

    The Thoughtworks Technology Radar (Trial, April 2026) recommends feedback sensors for coding agents: "deterministic quality gates such as compilers, linters, structural tests and test suites" wired into agentic workflows "so that failures trigger timely self-correction." Teams can add "a reviewer agent responsible for running checks and triggering corrections" or a companion process that agents can query.

    The sensors increase trust in generated results and "reduce routine steering work for the human in the loop." Timing matters: "Whenever possible, these sensors should run during the coding session and report clean results before a commit is made, rather than relying on post-commit checks."

    Thoughtworks Technology Radar: Feedback sensors for coding agents (Trial, Apr 2026)

  7. Send agent pull requests through the same CI gates and an independent approver

    GitHub's Copilot cloud agent can push only to a single branch and is "subject to any branch protections and required checks." "By default, workflows are not triggered until Copilot cloud agent's code is reviewed and a user with write access to the repository clicks the 'Approve and run workflows' button." Its draft pull requests "must be reviewed and merged by a human".

    GitHub also "Prevents the user who asked Copilot cloud agent to create a pull request from approving it. This maintains the expected controls in the 'Required approvals' rule and branch protection." The person who prompted the agent is not counted as the independent reviewer, and the agent's code meets the same required checks as anyone else's.

    GitHub Docs: Risks and mitigations for GitHub Copilot cloud agent

  8. Filter AI-generated tests with objective gates before people review them

    Meta's TestGen-LLM "verifies that its generated test classes successfully clear a set of filters that assure measurable improvement over the original test suite, thereby eliminating problems due to LLM hallucination." On Instagram Reels and Stories, "75% of TestGen-LLM's test cases built correctly, 57% passed reliably, and 25% increased coverage."

    Those numbers show why raw output needs gates: only a quarter of the generated tests added coverage. With the filters in place, "it improved 11.5% of all classes to which it was applied, with 73% of its recommendations being accepted for production deployment by Meta software engineers." Engineers still decided what shipped.

    Meta, arXiv 2402.09171: Automated Unit Test Improvement using Large Language Models at Meta (Feb 2024)

  9. Aim test generation at the faults you care about

    Meta's ACH uses mutation testing to steer an LLM. It "focuses on generating currently undetected faults that are specific to an issue of concern", then generates tests that catch those faults.

    Each generated test has a concrete job: catch a fault the current suite misses. At Messenger and WhatsApp test-a-thons "engineers accepted 73% of its tests, judging 36% to privacy relevant." Its LLM-based equivalent mutant detector reached "a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing)."

    Meta, arXiv 2501.12862: Mutation-Guided LLM-based Test Generation at Meta (Jan 2025)

  10. Strengthen automated testing before AI raises your change volume

    Google's 2025 DORA report found "a positive relationship between AI adoption on both software delivery throughput and product performance", but "AI adoption does continue to have a negative relationship with software delivery stability."

    DORA's explanation: "Without robust control systems, like strong automated testing, mature version control practices, and fast feedback loops, an increase in change volume leads to instability." And: "AI doesn't fix a team; it amplifies what's already there."

    Google Cloud blog: Announcing the 2025 DORA Report: State of AI-Assisted Software Development (23 Sep 2025)

Testing AI systems

The toolbox: open tools that test models and agents

Each runs from a command line or a few lines of Python, so it fits a CI job. More projects, ranked, in Open-source AI security repos.

promptfoo

promptfoo, now part of OpenAI · MIT license

Tests prompts, agents and RAG systems with assertions, and red teams them with plugins for indirect prompt injection, data exfiltration, excessive agency, SSRF and MCP.

  • OpenAI announced the acquisition on March 9, 2026; the project says it remains open source and MIT licensed and keeps supporting other providers and models.
  • Runs from YAML (promptfooconfig.yaml); eval exits with code 100 when a test fails, so a CI job fails with it.
  • --repeat runs each test several times, for agents that answer differently each run.
npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json
npx promptfoo@latest redteam run

Sources: GitHub: promptfoo · promptfoo: joining OpenAI · promptfoo docs: CI/CD · promptfoo docs: red team plugins

garak

NVIDIA · Apache 2.0

The Generative AI Red-teaming and Assessment Kit: probes a model for prompt injection, jailbreaks, data leakage and hallucination, against Hugging Face, OpenAI, Bedrock, LiteLLM, REST endpoints and local models.

  • Command line; release v0.17.0 (September 9, 2026) added an EU AI Act mapping.
  • --spec selects what to run (the older --probes is deprecated).
python -m pip install -U garak
garak --target_type openai --target_name <model> --spec probes.promptinject

Sources: GitHub: NVIDIA/garak · garak CLI reference

PyRIT

Microsoft · MIT license

The Python Risk Identification Tool for generative AI: a framework of targets, converters and scorers for red teaming, with multi-turn attacks such as Crescendo, TAP and Skeleton Key.

  • Use it as a scanner (pyrit_scan), a shell, a web interface or a Python library; v1.1.0 came out on September 4, 2026.
  • The project moved from Azure/PyRIT to microsoft/PyRIT, and the old repository was archived on March 27, 2026.
pip install pyrit
pyrit_scan

Sources: GitHub: microsoft/PyRIT · PyRIT documentation

Inspect

UK AI Security Institute and Meridian Labs · MIT license

A framework for evaluations of large language models, with over 200 ready-made evaluations, a log viewer and VS Code integration.

  • Epochs run every sample several times (--epochs 5), the way to measure an agent that is not the same twice.
  • Version 0.3.277 was released on October 6, 2026.
pip install inspect-ai
inspect eval my_task.py --model openai/<model> --epochs 5

Sources: GitHub: UKGovernmentBEIS/inspect_ai · Inspect documentation: tasks and epochs

DeepTeam

Confident AI · Apache 2.0

An open red teaming framework for LLM systems, "penetration testing, but for LLMs": pick the vulnerabilities and attacks, point it at your model through a callback or a YAML file.

  • Python (red_team with a model_callback) or a CLI run from a YAML config.
pip install -U deepteam
deepteam run config.yaml

Sources: GitHub: confident-ai/deepteam · DeepTeam: getting started

AgentDojo

ETH Zurich SPY Lab · MIT license

A benchmark environment for prompt injection attacks and defenses on LLM agents, with realistic tools and tasks; published at NeurIPS 2024.

  • Compare defenses (for example a tool filter) against named attacks on the same tasks.
pip install agentdojo
python -m agentdojo.scripts.benchmark --model <model> --defense tool_filter --attack tool_knowledge

Sources: GitHub: ethz-spylab/agentdojo · arXiv: AgentDojo

pass^k, the number to gate on

τ-bench (Sierra), 2024 · Anthropic's guide to agent evals, 2026

pass@k asks whether at least one of k runs succeeds; pass^k asks whether all k do. For a security test, only pass^k means anything: 75 percent per run is about 42 percent over three runs.

  • τ-bench found gpt-4o below 25 percent pass^8 on its retail tasks.

Sources: arXiv: τ-bench · Anthropic: Demystifying evals for AI agents

Secure the tester

When the tester is an agent, secure the tester

Agents that test software log in, read every page they test and keep screenshots, so the same incidents apply to them. What their own docs warn about, and seven rules, each from a case in the Learning Center.

Playwright Test Agents

Microsoft Playwright · since 1.56 (October 2025)

Three agents: the planner explores the app and writes a Markdown test plan, the generator turns it into Playwright tests, and the healer runs the suite and repairs failing tests until they pass or guardrails stop it.

  • The healer can also leave a test skipped if it believes the functionality is broken: review what it changes.
  • Playwright 1.60 (May 2026) added test.abort(), which its docs show stopping a test from publishing to a shared page: a guardrail you can put in a fixture.
npx playwright init-agents --loop=claude

Sources: Playwright docs: test agents · Playwright docs: test.abort()

Playwright MCP

Microsoft · Apache 2.0

An MCP server that lets an agent drive a browser through Playwright's accessibility tree, with no vision model.

  • Its README says plainly that it is not a security boundary, and that a persistent profile stores all the logged-in information.
  • Isolated mode drops the session's storage when the browser closes, and secrets can be masked in what the model sees.
npx @playwright/mcp@latest --isolated

Sources: GitHub: microsoft/playwright-mcp

Saved logins in tests

Playwright authentication guide

A saved browser state holds cookies and headers that can impersonate the test account. Playwright strongly discourages committing it and recommends ignoring playwright/.auth.

  • Anthropic's computer use docs add: a dedicated VM or container, no login details for the model, a domain allowlist, and a person confirming consequential actions.

Sources: Playwright docs: authentication · Claude docs: computer use tool

  1. Give it test accounts, never a person's

    An agent that logs in drives a real session. Give it accounts made for testing, with the least access the tests need, in an environment with no production data.

    From: practice: least privilege · Replit Agent deletes a production database during a code freeze

  2. Keep secrets out of the folder it works in

    Coding and test agents read their working folder. A .env file there is one prompt injection away from leaving, as the Copilot CLI case showed.

    From: GitHub Copilot CLI · practice: secrets hygiene

  3. Let it reach only the hosts the tests need

    An allowlist for network egress turns an injected "send this to" into a failed request. Test the allowlist itself (the egress recipe above).

    From: OpenAI agents reached staging servers among 55 websites, Asymmetric Security finds · practice: break trifecta

  4. Give it a private place for artifacts

    Screenshots, traces and videos hold whatever was on screen. Agents with nowhere approved to put them have created public repositories instead.

    From: PixelLeak · practice: output channels

  5. Treat every page it tests as untrusted input

    The application under test, its test data and the web pages it loads can all carry instructions. A test agent obeys them as readily as any other agent.

    From: PleaseFix · practice: untrusted content

  6. Review what it heals

    A self-healing test that quietly picks a different button can pass on exactly the change it should have caught. Make healed locators a reviewed change, not a silent one.

    From: practice: human approval

  7. Run security tests many times, and require every run to pass

    A model that resists an attack 19 times in 20 fails one user in 20. Agent tests repeat (the recipes above run 20 times) and pass only when every run passes.

    From: practice: monitoring