The same test, twice: the way teams wrote it, and the way AI writes, runs or heals it now. Unit tests, property tests, API contracts, Playwright, Cypress and Selenium, load, stress and soak, the evals that test AI itself, and the MCP servers that let an AI drive all of it. Each pair says what AI adds, what it costs, and where it fails.
Pick a layer
The same test, before and after AI
Copy the classic test to a project without AI, copy the AI version where a coding agent or an MCP server is already in the loop, and keep the person: every AI version below still needs someone who decides what the test means and reads what the AI changed.
No install and no server: both labs run in your browser on the examples above. The first grades tests the way mutation testing does; the second shows what a self-healing test agent does on a UI change, and when it must not.
Mutation lab: do these tests prove anything?
The pricing function from the unit example, with 0 bugs planted one at a time. A test suite that catches a bug kills the mutant; one that stays green has proved nothing about that line. Run the human suite, the AI suite, then add your own assertion.
pricing.py · the function under test
def apply_discount(price, percent):
if percent < 0:
raise ValueError("percent must be 0 or more")
percent = min(percent, 100)
discounted = price * (100 - percent) / 100
return max(0.0, round(discounted, 2))
Watch it instead (18 seconds)
0%human suite
0%AI suite
0%both, plus yours
Stryker calls a score of 80 and above a success and 60 to 80 a warning; its break threshold fails the build below the number you set.
Healer lab: a UI change, a failing test, and a repair that waits for you
A checkout page lives inside this box. Play the first scenario to watch the classic test run, a design change rename the button, the test fail, and the healer read the accessibility snapshot and propose a patch. Play the second to see the case where it must not heal.
shop.example.test/cart
Your cart
Trail runners, size 9$119.00
Thank you
Watch it instead (15 seconds)
tests/checkout.spec.ts · the test as written
The healer's patch is a code change. Approve it and the test reruns; reject it and the failure stands for a person to look at.
Unit and property
Unit and property tests
Unit tests: a human-written pytest, then the tests an assistant writes and mutation testing grades
pytestGitHub CopilotmutmutStrykerJS
The classic test
tests/test_pricing.py · python
# The classic unit test: one function, known inputs, exact expectations, no network.
from pricing import apply_discount
def test_ten_percent_off_rounds_to_cents():
assert apply_discount(19.99, percent=10) == 17.99
def test_no_negative_prices():
assert apply_discount(5.00, percent=150) == 0.00
With AI
tests/test_pricing_generated.py · python
# The AI version: an assistant wrote these from the function and its docstring. Keep them only
# if they pass against the real code, fail against a broken copy (mutation testing below) and
# add a case the human tests did not have. Delete the rest: a passing test that proves
# nothing is maintenance with no value.
import pytest
from pricing import apply_discount
@pytest.mark.parametrize(
("price", "percent", "expected"),
[
(100.00, 0, 100.00), # no discount
(100.00, 100, 0.00), # everything off
(0.00, 50, 0.00), # nothing to discount
(0.01, 50, 0.01), # never rounds below a cent
(19.99, 33, 13.39), # odd percentage, rounding checked by hand
],
)
def test_generated_edges(price, percent, expected):
assert apply_discount(price, percent=percent) == expected
def test_rejects_a_negative_percent():
with pytest.raises(ValueError):
apply_discount(10.00, percent=-5)
Watch it done (1:00)
Grade AI-written tests with mutation testing 1:00
Chapters and transcriptRun it
pytest -q
# grade the generated tests: a mutant that survives is a test that proves nothing
pip install mutmut && mutmut run && mutmut results
# JavaScript projects: npx stryker init, then npx stryker run (stryker.config.json)
What AI adds
Volume and edge cases nobody had time to type: Meta reports engineers accepted 73% of the tests its TestGen-LLM recommended, and Uber says its AutoCover agent writes about 11% of all new tests added to its codebase (both on the AI and agents page). Mutation testing turns "the tests pass" into "the tests catch planted bugs": Stryker's thresholds (high 80, low 60) grade the mutation score, and its break threshold fails the build below the score you set (the default, 0, never fails it).
The risk
AI tests love to assert what the code does today, bugs included, and to mock the thing under test. Read the expected values yourself (the 13.39 above was checked by hand), and run mutation testing before trusting a coverage number.
Property tests: let Hypothesis invent the inputs, and let an assistant propose the properties
Hypothesispytest
The classic test
tests/test_tokens.py · python
# Example-based: three inputs a person thought of.
from tokens import count_tokens
def test_examples():
assert count_tokens("") == 0
assert count_tokens("hello") == 1
assert count_tokens("hello world") == 2
With AI
tests/test_tokens_properties.py · python
# Property-based: Hypothesis generates the inputs and reports any exception it raises. The
# properties themselves are the part an assistant is good at proposing from a docstring
# ("never negative", "concatenation never loses tokens", "whitespace adds nothing"), and the
# part a person must still approve, because a wrong property passes just as quietly.
from hypothesis import given, strategies as st
from tokens import count_tokens
@given(st.text())
def test_never_negative(s):
assert count_tokens(s) >= 0
@given(st.text(), st.text())
def test_joining_never_loses_tokens(a, b):
assert count_tokens(a + " " + b) >= max(count_tokens(a), count_tokens(b))
@given(st.text(alphabet=" \t\n", min_size=1))
def test_whitespace_alone_is_zero(ws):
assert count_tokens(ws) == 0
Watch it done (0:46)
Let Hypothesis find the bug 0:46
Chapters and transcriptRun it
pip install hypothesis && pytest -q
What AI adds
Inputs no one would type. Hypothesis is "a powerful addition to unit testing", not a replacement, and the documented example is exactly the shape above: @given(st.integers()) on a plain test function. An assistant proposes properties cheaply; the generator checks thousands of cases against each one.
The risk
A property that is too weak passes forever and a property that is wrong makes you "fix" good code. Review each property as a specification, because that is what it is.
Integration and contract: a hand-written client test, then an OpenAPI schema fuzzed by Schemathesis
pytestSchemathesisPact
The classic test
tests/test_orders_client.py · python
# The integration test: the real HTTP client against a running test instance of the service.
import httpx
BASE = "http://127.0.0.1:8080"
def test_create_then_read_an_order():
created = httpx.post(f"{BASE}/orders", json={"sku": "A1", "qty": 2}, timeout=5)
assert created.status_code == 201
order_id = created.json()["id"]
read = httpx.get(f"{BASE}/orders/{order_id}", timeout=5)
assert read.status_code == 200
assert read.json()["qty"] == 2
With AI
ci/api-contract.sh · bash
#!/usr/bin/env bash
# The generated version: Schemathesis reads the OpenAPI schema and generates the requests, so
# every operation gets cases, including the ones nobody wrote a test for. It reports server
# errors (ST101) and undocumented status codes (ST103), which is what an AI-written handler
# most often gets wrong: it answers, but not the way the contract says.
set -euo pipefail
uvx schemathesis run http://127.0.0.1:8080/openapi.json \
--header "Authorization: Bearer $TEST_TOKEN" \
--max-examples 200 \
--continue-on-failure \
--report junit
# Consumer-driven contracts (Pact) stay the right tool when two teams own the two sides:
# the consumer records what it expects, and the provider is verified against the real
# implementation, not mocks.
Watch it done (0:49)
Fuzz the contract: Schemathesis against an OpenAPI schema 0:49
Chapters and transcriptRun it
bash ci/api-contract.sh
# JUnit XML lands in schemathesis-report/ for the CI dashboard
What AI adds
Coverage of the whole contract instead of the paths someone remembered. Schemathesis generates cases from the schema, caps them per operation with --max-examples, keeps going after the first failure with --continue-on-failure, and writes a JUnit report with --report junit; its failures name the rule broken, such as ST101 Server error and ST103 Undocumented status code. Pact's model is the complement: "Given, Upon receiving, With, Will respond with", verified "against the actual provider implementation, not mocks".
The risk
A schema that lies produces tests that lie. Generated requests also hit real endpoints, so point them at a test instance with test credentials, and sanitize the output (Schemathesis masks the token by default; --output-sanitize false turns that off).
End to end with Playwright: the test a person wrote, then the planner, generator and healer agents
PlaywrightPlaywright Test AgentsPlaywright MCP
The classic test
tests/checkout.spec.ts · typescript
// The classic Playwright test: role-based locators and web-first assertions that wait.
import { test, expect } from '@playwright/test';
test('a signed-in user can check out', async ({ page }) => {
await page.goto('/cart');
await page.getByRole('button', { name: 'Checkout' }).click();
await page.getByLabel('Card number').fill('4242 4242 4242 4242');
await page.getByRole('button', { name: 'Pay' }).click();
await expect(page.getByRole('heading', { name: 'Thank you' })).toBeVisible();
});
With AI
specs/checkout.md and tests/checkout.spec.ts · bash
# The AI version, Playwright 1.56 and later: three agent definitions for your coding agent.
npx playwright init-agents --loop=claude # or --loop=vscode, --loop=opencode
# 1. planner: explores the running app and writes specs/checkout.md, a Markdown test plan
# 2. generator: turns the plan into tests/checkout.spec.ts, verifying selectors and
# assertions live as it performs the scenarios
# 3. healer: when a test fails, replays the steps, inspects the UI for an equivalent element,
# patches the locator, wait or data, and reruns until it passes or its guardrails stop it;
# a test it believes is really broken is skipped, not forced green
npx playwright test
# Regenerate the definitions whenever Playwright is updated: they are version-specific.
Watch it done (1:08)
Playwright: the failure an agent can read, and the healer 1:08
Chapters and transcriptRun it
npx playwright init-agents --loop=claude && npx playwright test
What AI adds
A plan a person can read before a line of test code exists, generated tests checked against the live app as they are written, and repairs for the breakage that eats most UI-test time. Playwright's own description: the planner "explores the app and produces a Markdown test plan", the generator "turns the Markdown plan into Playwright Test files", the healer "runs the test suite and tries to repair failing tests". Under the hood the agents drive the browser through Playwright's MCP server (@playwright/mcp), which gives any coding agent tools such as browser_navigate, browser_snapshot and browser_click and works from accessibility snapshots rather than pixels.
The risk
A healer that "fixes" a test around a real bug hides the bug. Playwright's healer skips a test it believes is broken instead of forcing it green; read every healed diff as a code change, and keep an eye on what the browser agent was allowed to reach (--isolated runs a fresh profile with no saved logins).
End to end with Cypress: the recorded test, then cy.prompt() steps in plain English
Cypresscy.prompt()Cypress Cloud
The classic test
cypress/e2e/login.cy.js · javascript
// The classic Cypress test: selectors a person chose and kept up to date.
describe('login', () => {
it('signs in with a valid account', () => {
cy.visit('/login');
cy.get('[data-test=email]').type('ada@example.test');
cy.get('[data-test=password]').type(Cypress.env('TEST_PASSWORD'), { log: false });
cy.get('[data-test=submit]').click();
cy.contains('h1', 'Welcome back').should('be.visible');
});
});
With AI
cypress/e2e/login.prompt.cy.js · javascript
// The AI version (Cypress 15.13 and later, beta): plain-English steps, each turned into
// Cypress commands; when a selector breaks between runs, the step is regenerated.
// Needs a Cypress Cloud account or record key, Chromium-based browsers, end-to-end tests
// only; canvas and iframes are not supported.
describe('login', () => {
it('signs in with a valid account', () => {
cy.prompt([
'visit /login',
'type ada@example.test into the email field',
'type the test password into the password field',
'click the sign in button',
'the page shows a heading that says Welcome back',
]);
});
});
Run it
npx cypress run --record --key "$CYPRESS_RECORD_KEY"
What AI adds
Tests a product owner can read, and self-healing for the small UI changes that break selectors: Cypress says "When a selector fails between runs, Cypress regenerates code for that step automatically" and that it "works best when it catches occasional, unexpected changes". cy.prompt() was introduced in 15.4.0 as experimental and moved to beta in 15.13.0, which removed the experimentalPromptCommand flag.
The risk
The steps go to Cypress Cloud ("Without a Cloud account, there is no secure channel through which to route those requests"), so the test text and page content leave your machine; keep secrets out of the prompt (the example says "the test password", never the value). Generated code "may look like" the examples, and one documented example did not match its own prompt word for word, so export and read the generated code before you depend on it.
End to end with Selenium: the brittle locator, then a self-healing driver
SeleniumHealeniumJava
The classic test
src/test/java/CheckoutTest.java · java
// The classic Selenium test: a locator that breaks the day the markup changes.
WebDriver driver = new ChromeDriver();
driver.get("https://shop.example.test/cart");
driver.findElement(By.cssSelector("#cart > div.actions > button.btn-primary")).click();
driver.findElement(By.id("card-number")).sendKeys("4242424242424242");
driver.findElement(By.xpath("//button[text()='Pay']")).click();
assert driver.findElement(By.tagName("h1")).getText().equals("Thank you");
With AI
src/test/java/CheckoutTest.java and healenium.properties · java
// The self-healing version: wrap the driver. When a locator stops matching, Healenium looks
// for the control that most resembles the one it saw before (an LCS-based algorithm with
// machine-learned priorities), uses it if the match is above score-cap, and reports the
// healed locator with a screenshot. Maven: com.epam.healenium:healenium-web:3.5.8.
WebDriver delegate = new ChromeDriver();
SelfHealingDriver driver = SelfHealingDriver.create(delegate);
driver.get("https://shop.example.test/cart");
driver.findElement(By.cssSelector("#cart > div.actions > button.btn-primary")).click();
driver.findElement(By.id("card-number")).sendKeys("4242424242424242");
driver.findElement(By.xpath("//button[text()='Pay']")).click();
assert driver.findElement(By.tagName("h1")).getText().equals("Thank you");
// healenium.properties, in test/resources
// recovery-tries = 1
// score-cap = 0.5
// heal-enabled = true
// hlm.server.url = http://localhost:7878
Run it
# start hlm-backend first (healenium-backend repository), then run the suite as usual
mvn test
What AI adds
The tests keep running through markup changes, and the healed locators are reported (and, with the IntelliJ plugin, written back into the Java code in one click), so the fix is a review instead of a morning of triage. Healenium works with any Java and Selenium framework and, through its proxy, with WebdriverIO and Robot Framework.
The risk
Healing is a guess above a probability threshold (score-cap 0.5 means a 50% match is enough), and a button that moved may be a different button. Keep heal-enabled off for tests that guard payments or deletions, read the healing report, and commit the new locator rather than living on the heal. Healenium is Selenium only, Java first, and does not heal elements found through JavascriptExecutor.
Load, stress, spike and soak: a k6 script with thresholds, then a soak profile and an LLM endpoint
k6Locust
The classic test
load/soak.js · javascript
// k6: ramp up to the average load, hold it, ramp down. Thresholds turn the run into a test.
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
stages: [
{ duration: '2m', target: 15 }, // ramp up to average-load level
{ duration: '30m', target: 15 }, // soak plateau: 2 to 8 hours in production soaks
{ duration: '2m', target: 0 }, // ramp down
],
thresholds: {
http_req_duration: ['p(95)<500'],
http_req_failed: ['rate<0.01'],
},
};
export default function () {
const res = http.get('https://quickpizza.grafana.com');
check(res, {
'status is 200': (r) => r.status === 200,
'response body is not empty': (r) => r.body && r.body.length > 0,
});
sleep(1);
}
With AI
load/locustfile.py · python
# The AI-era version: the system under load is a model endpoint, and the metrics that matter
# are per token, not per request. Standard Locust reports request latency; time to first
# token and inter-token latency need an LLM-aware extension or a measurement like the one
# below, which times the first streamed chunk by hand.
import os
import time
from locust import HttpUser, between, task
PROMPTS = [
"Summarize the refund policy in two sentences.",
"List three risks of running an AI agent with database credentials.",
]
class ChatUser(HttpUser):
wait_time = between(1, 3)
@task
def ask(self):
body = {"model": "small", "stream": True,
"messages": [{"role": "user", "content": PROMPTS[int(time.time()) % 2]}]}
headers = {"Authorization": f"Bearer {os.environ['TEST_TOKEN']}"}
started = time.perf_counter()
with self.client.post("/v1/chat/completions", json=body, headers=headers,
stream=True, catch_response=True, name="chat") as resp:
first = None
for chunk in resp.iter_lines():
if chunk and first is None:
first = time.perf_counter() - started # time to first token
if resp.status_code != 200:
resp.failure(f"status {resp.status_code}")
elif first is None or first > 2.0:
resp.failure(f"first token after {first}s") # the SLO you actually promise
Watch it done (0:48)
Load test with Locust, and what changes for an LLM endpoint 0:48
k6 names six test types, each a different question: smoke ("that your script works and that the system performs adequately under minimal load"), average load, stress ("at its limits when load exceeds the expected average"), soak ("reliability and performance of your system over extended periods", hours), spike ("sudden, short, and massive increases in activity") and breakpoint ("gradually increase load to identify the capacity limits"). A threshold breach during a soak plateau usually points to degradation that depends on time, not load. For model endpoints the units change: time to first token, inter-token latency, tokens per second and goodput (the share of requests meeting every SLO); purpose-built tools such as LLMPerf, GenAI-Perf and GuideLLM measure them, and k6 and Locust need extensions to.
The risk
A load test against a paid model API is a bill, and against a shared gateway it is an outage for your colleagues: run it against a test deployment with a spend cap. Averages hide the tail; read p95 and p99, and keep the load pattern simple (ramp, plateau, ramp), not a rollercoaster.
Testing the AI itself: a brittle string match, then evals with a rubric and a judge in CI
promptfooDeepEvalpytest
The classic test
tests/test_bot_answers.py · python
# The first thing every team writes: an exact-match test that breaks on the next rephrasing.
from bot import answer
def test_refund_window():
assert answer("What if these shoes don't fit?") == "We offer a 30-day full refund."
With AI
promptfooconfig.yaml and tests/test_bot_evals.py · yaml
# promptfooconfig.yaml: the same question, graded on what matters. `promptfoo eval` runs the
# matrix across prompts and providers; contains and javascript assertions are deterministic,
# llm-rubric asks a model to grade against a rule, similar compares embeddings.
prompts:
- file://prompts/support.txt
providers:
- openai:gpt-6-luna
defaultTest:
assert:
- type: llm-rubric
value: does not describe self as an AI, model, or chatbot
tests:
- vars:
input: "What if these shoes don't fit?"
assert:
- type: contains
value: "30-day"
- type: javascript
value: output.toLowerCase().includes('refund')
- type: llm-rubric
value: states the refund window and does not invent conditions that are not in the policy
# tests/test_bot_evals.py: the same idea inside pytest with DeepEval
# from deepeval import assert_test
# from deepeval.test_case import LLMTestCase
# from deepeval.metrics import AnswerRelevancyMetric
#
# def test_answer_relevancy():
# metric = AnswerRelevancyMetric(threshold=0.5)
# case = LLMTestCase(
# input="What if these shoes don't fit?",
# actual_output=answer("What if these shoes don't fit?"),
# retrieval_context=["All customers are eligible for a 30 day full refund at no extra cost."],
# )
# assert_test(case, [metric])
# run with: deepeval test run tests/test_bot_evals.py (needs OPENAI_API_KEY for the judge)
Watch it done (0:40)
Evals in CI with promptfoo, no judge model required 0:40
Chapters and transcriptRun it
npx promptfoo@latest eval
pip install -U deepeval && deepeval test run tests/test_bot_evals.py
What AI adds
A test that survives rephrasing and still catches the failures that matter: a missing fact, an invented condition, the bot calling itself a chatbot. promptfoo's config pairs prompts and providers into a matrix with assertions per test (llm-rubric, contains, javascript, similar with a threshold); DeepEval puts the same thing in pytest with assert_test, LLMTestCase and metrics scored 0 to 1 against a threshold, run with deepeval test run.
The risk
A judge model is another non-deterministic component in the test, and it costs money per run: pin the judge model, keep deterministic assertions (contains, javascript) for the facts that must hold, use the rubric for the judgment calls, and sample the judge's verdicts by hand every release. Default metrics need an API key (DeepEval's examples need OPENAI_API_KEY), so the evals read your test prompts to a provider.
MCP servers and agents: a manual click-through, then the Inspector CLI as a CI gate and a harness test
MCP Inspectorpytest
The classic test
docs/release-checklist.md · markdown
# Release checklist (before AI): open the MCP Inspector in the browser, connect to the
# server, click Tools, check that get_weather is listed, call it with a city, read the
# answer, screenshot it for the ticket. Fifteen minutes, once per release, by one person.
With AI
ci/mcp-gate.sh · bash
#!/usr/bin/env bash
# The CI version: the Inspector's CLI connects, runs one request and exits with a code a
# pipeline can branch on (3 needs auth, 4 unreachable, 5 tool error or tool not found,
# 6 schema portability problem under --strict). --stored-auth-only never opens a browser.
set -euo pipefail
# the tool list is the contract the agent reasons over: fail if a tool disappeared
npx @modelcontextprotocol/inspector --cli --config ./ci-servers.json --server weather \
--stored-auth-only --method tools/list --format json \
| jq -e '.result.tools | map(.name) | index("get_weather")' > /dev/null
# schemas every client can read, or exit 6
npx @modelcontextprotocol/inspector --cli --config ./ci-servers.json --server weather \
--stored-auth-only --method tools/list --strict > /dev/null
# one real call; a tool that returns isError: true exits 5 and stops the chain
npx @modelcontextprotocol/inspector --cli --config ./ci-servers.json --server weather \
--stored-auth-only --method tools/call --tool-name get_weather --tool-arg city=Calgary \
--format json | jq -e '.result.content[0].text | test("Calgary")' > /dev/null
# then the harness tests from this center: tool descriptions unchanged, bad token refused,
# agent role cannot delete (recipes below), run with pytest -q
Each CLI run "connects to a server, invokes the single request you name with --method, prints the result, and exits", which the protocol's own docs call "a good fit for CI pipelines, shell one-liners, and coding agents that need to verify a server change immediately". --format json pipes cleanly into jq, --strict turns the schema check into a gate, and every failure class has its own exit code and a one-line JSON error on stderr.
The risk
A green tools/list says the contract is there, not that it is safe: a tool description can change meaning without changing name (snapshot it), and a server can pass every call while leaking credentials through a confused deputy. That is what the harness recipes below are for, and why the agent that runs these tests should hold test credentials only.
Which one to use. These are our picks, not a survey result: the usage numbers next to each are refreshed every morning from GitHub, npm and PyPI, and the reasons come from the tools' own pages linked in the examples above. The rule that beats any pick: use the runner your team already runs in CI, and add the AI layer to it rather than around it.
Unit and property tests
pytest (Python) or Vitest (JavaScript and TypeScript)
The assistant writes tests in whatever runner the repository already has, so the runner choice is about speed and fixtures, not AI. pytest and Vitest are the fast defaults in their ecosystems; Jest and JUnit where a codebase already standardized on them.
The AI move: Have the assistant write the tests, keep the ones mutation testing (mutmut, Stryker) proves, and add Hypothesis or fast-check properties for the invariants.
Schemathesis for anything with an OpenAPI schema; Pact when two teams own the two sides
Schemathesis generates the cases from the contract and reports the rule each failure breaks, which is the cheapest way to check code an agent wrote against the interface a person agreed to. Pact is the answer when the consumer and the provider are different teams and the contract must be versioned between them. Testcontainers gives both a real database to run against.
The AI move: Let the agent write the handler; let the schema fuzzer and the contract decide whether it is right.
Role-based locators and web-first assertions made Playwright tests sturdier before AI; now its planner, generator and healer agents and its MCP server ship with it, in the open and for any coding agent, so the AI layer is a command away and runs on your machine. Cypress's cy.prompt() is the best plain-English authoring experience but routes through Cypress Cloud and Chromium only. Selenium remains the choice for Java shops, existing grids and the widest browser and language coverage, with Healenium for self-healing.
The AI move: Start with the planner so a person approves the plan, let the generator write the tests, and review every healer patch as a pull request; never let a healer touch a test that guards money or deletion.
Grafana k6 for CI gates; Locust when the team writes Python; an LLM-aware tool for model endpoints
k6 scripts are plain JavaScript with thresholds that pass or fail the run, which is what a pipeline needs, and its docs define the six test types clearly. Locust is the Python team's equivalent with a live UI. For model endpoints the metrics change to time to first token, inter-token latency and tokens per second, where LLMPerf, GenAI-Perf and GuideLLM were built for the job and k6 and Locust need extensions.
The AI move: Ask the assistant to draft the scenario and thresholds from your SLOs, then read the p95 and p99, not the average; run against a test deployment with a spend cap.
promptfoo for config-driven evals across providers; DeepEval inside pytest
promptfoo pairs prompts and providers in a matrix with deterministic and model-graded assertions in one YAML file, which suits comparing models and prompts. DeepEval lives where Python teams already are, with assert_test and metrics scored against thresholds.
The AI move: Keep the facts deterministic (contains, javascript), use the rubric for judgment, pin the judge model, and sample its verdicts by hand every release.
The MCP Inspector CLI as the gate, plus the harness tests from this center
It is the protocol's own tool, it exits with a distinct code for each failure class and it needs no browser in CI. The harness tests here add what a contract check cannot: that a bad token is refused, that a tool description has not changed meaning, that the agent's role cannot delete.
The AI move: Snapshot tools/list and diff it on every build; treat a changed description as a changed API.
Usage numbers as of 2026-11-18, from GitHub, npm and PyPI; every tool links to its card on the Test types page.
Shifting left at AI speed
Where this is going
Shift left meant "test earlier". At AI speed it means the checks run before a person ever reads the change, because the change was written by a machine and there are too many of them. The pattern that is emerging: validation that checks, scores and judges each AI output at the moment it is produced, with the person reading the score instead of the diff.
Early validation: check, score, judge, before the diff reaches a person
When an agent writes the code, the tests, the infrastructure and the pull request, the review queue is the bottleneck. The answer that scales is a validation stage in front of the queue: deterministic checks first (does it build, do the tests pass, does mutation testing say the new tests prove anything, does the schema still match, is any secret or CVE in the change), then scores (coverage of the spec, mutation score, eval scores against the thresholds above), then a judgment for what only a model or a person can judge, with the person reading the scores and sampling the judgments. Each example on this page is one of those checks; the shift-left workflow page puts them in order.
Tests as the specification, and the specification as the test
With agents writing the code, a human-owned spec is where intent lives, and the tests are its executable form: property tests state the invariants, contract tests state the interfaces, evals state what a good answer is, and a plan in Markdown states the user journey before Playwright's generator turns it into code. Write those first, and let the agent write the rest to satisfy them; it is cheaper to argue about a property than about a thousand lines.
Healing is maintenance you did not do: Playwright's healer, Cypress's cy.prompt() and Healenium all repair a test that broke on a UI change, and all three can paper over a real regression. The forward-looking setup treats every heal as a pull request: the repaired test runs, the diff is read, and a test that guards money or deletion is never healed without a person.
The agent that failed in production left a trace: the prompt, the tools it called, the answer it gave. With generative-AI tracing conventions and tracing platforms, that trace is a test case: replay it in CI as an eval, add the property it broke, and the suite grows from what actually happened rather than what someone imagined.
An agent that heals a test can loop as happily as an agent that writes code. The same budget tests (tokens, wall clock, tool calls per task) that guard a production agent belong around the planner, generator and healer, and around the judge in your evals, so a flaky UI does not become a surprising bill.
A browser agent with saved logins, a self-healing driver on the payment page, an eval suite with a provider key: every AI version on this page is an agent with credentials and access, and the incidents in the Learning Center show what happens to those. Run them isolated, with test credentials, an egress allowlist and the harness tests from this center around them.
As of 2026-10-25. Every tool fact above links to the page it was read on; the code is ours, written to those pages' APIs, and the Python and JavaScript in it is checked by the build.