AI Test Automation / Learn / Videos
AI Test Automation Preview
Videos: how to test with AI in the loop Short, captioned and real. Each one teaches one move, shows the tool's own output, and ends with the sources.
How they were made
12 short videos, 12:09 in all, and every command in them ran for real
Short, captioned, and real: every command in these videos ran on the date shown, and the output is unedited (the transcripts are in the repository). Each one teaches one move: how to use the tool with AI in the loop, what to read in its output, and the mistake it catches.
Real sessions, replayed Each video replays a session recorded on 2026-10-25: the commands were run, their output was saved unedited (the transcripts are in the repository under data/testlab/transcripts/), and a terminal player types the commands back and shows that output. Nothing in a terminal pane was written by hand.
Captions, chapters, transcript Every video carries a captions track (WebVTT), chapter buttons that seek to the moment, and a full transcript you can read or search instead of watching. The player has no tracking and loads nothing until you press play.
Sources on the last frame The closing card of each video names the documentation its facts came from; the pages linked under each video carry the same facts with the links. When a tool changes, the transcript keeps the date it was true.
Small on purpose 24 frames a second, 1280 by 720, each file under a megabyte and a half: the whole set weighs 6.8 MB, so the page stays light on a phone and nothing loads until you press play.
Play all, in order One video plays at a time; chapters and transcript lines jump to their moment.
Watch
How to, layer by layer
One move per video, with the tool's own output. Each has a worked example on the Examples page.
Your browser does not play this video; download it or read the transcript below.
Grade AI-written tests with mutation testing 1:00
A passing suite proves little. Mutation testing plants bugs and counts how many the tests catch; run it on the human suite, then with the assistant's tests, and read the survivors. The worked example
Chapters and transcript
0:00 Do these tests prove anything?0:04 The human suite passes.0:08 The assistant's suite passes too.0:13 Now plant bugs. mutmut changes one thing at a time and reruns the human suite.0:18 Read the survivors: the first one flips < to <=.0:25 Add the assistant's tests and run again.0:30 Four survive both suites, and they tell you something.0:37 Keep the tests that kill mutants. Delete the rest.
0:00 Do these tests prove anything? pytest, an assistant's tests, and mutmut: a real run, unedited.0:04 The human suite passes. Two tests a person wrote. Green, and that is all it tells you.0:09 The assistant's suite passes too. Six parametrized cases it wrote from the docstring. Also green.0:13 Now plant bugs. mutmut changes one thing at a time and reruns the human suite. 27 mutants of pricing.py. A mutant that survives is a bug the tests cannot see.0:18 Read the survivors: the first one flips < to <=. percent = 0 would be rejected and no human test noticed.0:26 Add the assistant's tests and run again. Its (100, 0) case and its negative-percent case kill two more mutants.0:31 Four survive both suites, and they tell you something. Three reword the error message, which no test should pin. One is min(percent, 101): the clamp hides it.0:38 Keep the tests that kill mutants. Delete the rest. The assistant's volume is worth having when a mutation score says so: 21 of 27 became 23 of 27 here, and the survivors told you which line nobody is really testing.0:42 Run mutation testing before you trust a coverage number0:45 Read every surviving mutant: equivalent, or a gap?0:48 Stryker and mutmut both fail the build below a score you set
Your browser does not play this video; download it or read the transcript below.
Let Hypothesis find the bug 0:46
Property tests state what must always hold; the generator supplies thousands of inputs and shrinks a failure to the smallest case. An assistant is good at proposing properties, and a person must approve them. The worked example
Chapters and transcript
0:00 Three properties, thousands of inputs.0:04 The properties say what must always be true.0:09 Run it. Hypothesis searches, finds a failure, and shrinks it.0:18 The bug: split(" ") counts empty strings. split() does not.0:24 Let the assistant propose properties. You approve them.
0:00 Three properties, thousands of inputs. Hypothesis on a tokenizer an assistant wrote. Real run.0:04 The properties say what must always be true. Never negative, joining never loses tokens, whitespace alone counts as zero.0:09 Run it. Hypothesis searches, finds a failure, and shrinks it. The smallest failing input: a single space. count_tokens(' ') returned 2.0:19 The bug: split(" ") counts empty strings. split() does not. One character of code; a case no example-based test had.0:25 Let the assistant propose properties. You approve them. A wrong property passes as quietly as a right one. Treat each @given test as a line of the specification, because that is what it is.0:29 Hypothesis is an addition to unit tests, not a replacement0:32 Ask for invariants, not examples: never, always, round trips0:35 Keep the shrunken failing case as a regression test
Your browser does not play this video; download it or read the transcript below.
Fuzz the contract: Schemathesis against an OpenAPI schema 0:49
An agent writes a handler that answers, but not the way the contract says. Schemathesis generates the requests from the schema and names the rule each failure breaks. The worked example
Chapters and transcript
0:00 The schema generates the tests.0:04 The bug: a lookup that crashes on an id it has never seen.0:09 One command, no test code written.0:16 Each failure names the rule and gives you the curl to reproduce it.0:24 Let the agent write the handler. Let the schema decide if it is right.
0:00 The schema generates the tests. A small FastAPI orders service with one bug an agent would write. Schemathesis finds it from the OpenAPI document alone.0:04 The bug: a lookup that crashes on an id it has never seen. KeyError becomes a 500. The contract promises 200 or 422.0:09 One command, no test code written. Schemathesis reads the schema, runs examples, coverage, fuzzing and stateful phases.0:17 Each failure names the rule and gives you the curl to reproduce it. ST101 Server error and ST103 Undocumented status code on GET /orders/0.0:25 Let the agent write the handler. Let the schema decide if it is right. 405 generated cases in under five seconds, against a service nobody wrote a test for. Point it at a test instance with test credentials, and ship the JUnit report to CI.0:29 --max-examples caps the cases per operation0:32 --continue-on-failure keeps going after the first failure0:35 --report junit writes the file your pipeline reads0:38 Pact when two teams own the two sides of the contract
Your browser does not play this video; download it or read the transcript below.
Playwright: the failure an agent can read, and the healer 1:08
A role-based test passes, a design change renames the button, the test fails and Playwright writes a failure file addressed to an AI. Then init-agents adds the planner, generator and healer to your repository. The worked example
Chapters and transcript
0:00 The test, the change, the failure an agent can read.0:04 The classic test: role-based locators, web-first assertions.0:09 The design team renames the button. Run again.0:18 Look at that error context file: it is written for an AI.0:29 Add the agents to the repository: one command.0:35 The healer is a subagent with a tool list and a workflow.0:45 Plan first, generate second, heal with a human on the diff.
0:00 The test, the change, the failure an agent can read. Playwright 1.64 against a small checkout page. Real run.0:04 The classic test: role-based locators, web-first assertions. No CSS selectors, no sleeps.0:09 The design team renames the button. Run again. The locator waits the whole timeout for a button that no longer exists.0:19 Look at that error context file: it is written for an AI. Instructions, the error, and the page's accessibility snapshot, with the new button name in it.0:29 Add the agents to the repository: one command. Planner, generator and healer as Claude Code subagents, plus the MCP server they drive the browser with.0:35 The healer is a subagent with a tool list and a workflow. test_run, test_debug, browser_snapshot, browser_generate_locator: it heals with the same tools you would use.0:45 Plan first, generate second, heal with a human on the diff. The planner's Markdown is the review point before any code exists. A healed test is a code change: read it like one, and never heal a test that guards money or deletion.0:50 Regenerate the agent definitions when Playwright updates0:53 Keep the browser agent isolated: no saved logins, test credentials only0:56 A healer that skips a test it believes is broken is doing its job
Your browser does not play this video; download it or read the transcript below.
Load test with Locust, and what changes for an LLM endpoint 0:48
A headless Locust run is a CI step. For model endpoints the metrics change to time to first token and tokens per second, and the risk changes to a bill. The worked example
Chapters and transcript
0:00 Load is a test with a threshold, not a number to admire.0:04 A locustfile is a few lines of Python.0:09 Run it headless, with a user count, a spawn rate and a time limit.0:20 Read p95 and p99, never the average.0:25 The units change: time to first token, tokens per second, goodput.
0:00 Load is a test with a threshold, not a number to admire. Locust headless against the orders API, then the lines that change when the thing under load is a model.0:04 A locustfile is a few lines of Python. Weighted tasks, a think time, a host passed on the command line.0:09 Run it headless, with a user count, a spawn rate and a time limit. The same command runs in CI; the summary table is the result.0:20 Read p95 and p99, never the average. k6 does the same with thresholds that fail the run: p(95)<500, rate<0.01.0:26 The units change: time to first token, tokens per second, goodput. A chat endpoint streams. Request latency hides what users feel, so time the first chunk and set the SLO on it; purpose-built tools (LLMPerf, GenAI-Perf, GuideLLM) measure this out of the box.0:30 Run against a test deployment with a spend cap: a load test on a paid API is a bill0:33 Keep the pattern simple: ramp, plateau, ramp; no rollercoaster0:36 A threshold breach late in a soak points to time, not load
Your browser does not play this video; download it or read the transcript below.
Evals in CI with promptfoo, no judge model required 0:40
An exact-match test breaks on the next rephrasing. promptfoo runs a matrix of inputs with deterministic assertions, and a failure reads like a test failure, not a vibe. The worked example
Chapters and transcript
0:00 Evals that run like tests.0:04 The config is the test plan: inputs, and what a good answer must contain.0:10 Run the matrix.0:18 Facts deterministic, judgment by rubric, judge pinned and sampled.
0:00 Evals that run like tests. promptfoo against a support bot, with contains, not-contains and javascript assertions. No API key, no judge model, real run.0:04 The config is the test plan: inputs, and what a good answer must contain. defaultTest applies to every row; here, the bot must never call itself an AI.0:10 Run the matrix. Three inputs, one provider; every cell passes or fails on the assertions.0:19 Facts deterministic, judgment by rubric, judge pinned and sampled. The failure here is the persona rule, caught by a string check that costs nothing. Add llm-rubric for what only a model can judge, pin the judge model, and read a sample of its verdicts every release.0:23 contains and javascript for the facts that must hold0:26 llm-rubric and similar for the judgment calls0:29 DeepEval does the same inside pytest with assert_test
Your browser does not play this video; download it or read the transcript below.
Four CI gates for an MCP server 1:02
The MCP Inspector's CLI connects, runs one request and exits with a code a pipeline can branch on. Four gates: the tool list, the schemas, one real call, and one call that must be refused. The worked example
Chapters and transcript
0:00 The tool list is an API. Gate it like one.0:04 Gate 1: the contract. Fail the build if a tool disappeared.0:12 Gate 2: schemas every client can read.0:18 Gate 3: one real call, and check the answer.0:27 Gate 4: a call that must be refused. Exit 5 stops the chain.0:37 Green gates say the contract is there, not that it is safe.
0:00 The tool list is an API. Gate it like one. The protocol's own Inspector, in CLI mode, against the reference filesystem server. Real run, real exit codes.0:04 Gate 1: the contract. Fail the build if a tool disappeared. tools/list as JSON, piped to jq -e, which exits non-zero when the tool is missing.0:13 Gate 2: schemas every client can read. --strict turns the portability check into exit code 6 when a schema has an error-severity problem.0:19 Gate 3: one real call, and check the answer. tools/call with a typed argument; the content comes back as JSON.0:27 Gate 4: a call that must be refused. Exit 5 stops the chain. A read outside the allowed directory returns isError: true, and the CLI exits 5 with a one-line JSON error on stderr.0:38 Green gates say the contract is there, not that it is safe. A description can change meaning without changing name, and a server can pass every call while leaking credentials. Snapshot tools/list and diff it, and run the harness tests from this center around the agent.0:42 Exit codes: 3 needs auth, 4 unreachable, 5 tool error, 6 schema problem0:45 --stored-auth-only never opens a browser in CI0:48 The agent that runs these gates holds test credentials only
Watch
When the result is not deterministic
What changes when the thing under test words every answer differently, from unit tests to the eval gate.
Your browser does not play this video; download it or read the transcript below.
Testing results that are not deterministic 1:22
A model words every answer differently. Exact match breaks, properties hold, and the gate is a pass rate with a confidence bound over enough samples. Real runs: promptfoo with repeats, and the same idea in pytest. The page: testing nondeterministic results
Chapters and transcript
0:00 The words change every call. The test cannot.0:04 The thing under test: one of several phrasings, every call.0:11 Exact match versus properties: run every test ten times.0:22 Read it: one instability is the test's fault, the other is the bot's.0:29 The gate: a pass rate the run must clear, not zero failures.0:40 The same idea in pytest: properties on every sample, a bound on the rate.0:50 Enough samples, and the honest result: shipping passes, persona fails.0:58 Remove the randomness you can. Assert properties on the rest. Gate on a rate with a bound.
0:00 The words change every call. The test cannot. A support bot that samples its phrasing, as a model does. Three ways to assert on it, run for real.0:05 The thing under test: one of several phrasings, every call. Same facts, different words. One phrasing in three for the off-topic case breaks persona: a real bug, hidden in the noise.0:12 Exact match versus properties: run every test ten times. promptfoo --repeat 10 reports the stability of each test with a 95 percent confidence interval.0:22 Read it: one instability is the test's fault, the other is the bot's. Exact match fails on wording. The properties test (contains 30, refund, unworn, under 160 characters) passed 10 of 10. Persona at 5 of 10 is the bug.0:29 The gate: a pass rate the run must clear, not zero failures. Drop the exact-match test; keep the properties; set the threshold. Exit code 100 stops the pipeline.0:41 The same idea in pytest: properties on every sample, a bound on the rate. The Wilson lower bound asks how low the true pass rate could be. 40 of 40 is not enough to claim 95 percent.0:50 Enough samples, and the honest result: shipping passes, persona fails. With 80 samples and no failures the bound clears 95 percent. Persona at 56 of 80 could be as low as 59 percent: fix the bot, not the test.0:59 Remove the randomness you can. Assert properties on the rest. Gate on a rate with a bound. Pin the model and the prompt, seed what you own, replay recorded answers in unit tests, and let only the real model's variance reach the eval stage, where the sample size is a decision you write down.1:04 Facts: contains, regex, is-json, schema. Judgment: a pinned rubric judge, sampled by hand1:07 Zero failures in n runs is a bound, not a proof: 73 clean runs for 95 percent1:09 A pass rate that moves between releases is the regression
Watch
Infrastructure and the development lifecycle
The same discipline one level up: validating Terraform for three clouds before a plan, and the repository rules around an agent that writes code.
Your browser does not play this video; download it or read the transcript below.
Terraform for AWS, Azure and GCP: validate before you plan 1:16
An agent writes a storage module for three clouds. Before anything reaches a plan, four validators run without credentials: fmt, tflint, Checkov and a Conftest policy the team owns. Real run on the draft and on the fix. The page: infrastructure as code, in AI Delivery
Chapters and transcript
0:00 The same bucket, three clouds, four validators, zero credentials.0:04 The draft: what an agent writes from "give me a bucket for the export job".0:10 Rung 1: format. Rung 2: lint.0:18 Rung 3: Checkov, 22 failed checks across the three clouds.0:28 Rung 4: the team's own rules, as code. Conftest reads the HCL directly.0:36 Fix, and run the ladder again.0:47 What is left is for people: replication, private endpoints, customer-managed keys.0:52 terraform test with mock providers: plan assertions, no cloud account.
0:00 The same bucket, three clouds, four validators, zero credentials. An agent's first draft of a storage module for AWS, Azure and GCP, and the validation ladder that runs before plan. Real run.0:05 The draft: what an agent writes from "give me a bucket for the export job". Correct, deployable, and missing every control.0:10 Rung 1: format. Rung 2: lint. tflint with the AWS, Azure and Google rulesets: no required_version, and no prevent_destroy on the storage account.0:19 Rung 3: Checkov, 22 failed checks across the three clouds. Public access, TLS, versioning, logging, encryption: each with the resource and the lines.0:28 Rung 4: the team's own rules, as code. Conftest reads the HCL directly. Three rules phrased once per cloud, in Rego. No plan, no credentials.0:37 Fix, and run the ladder again. 44 passed, 16 failed. The policy still catches the one thing the fix forgot.0:47 What is left is for people: replication, private endpoints, customer-managed keys. Checkov's remaining findings are decisions, not omissions. Record them as exceptions with a reason, or fix them.0:53 terraform test with mock providers: plan assertions, no cloud account. From Terraform 1.7, mock_provider lets a .tftest.hcl run plan and apply against generated values and assert on them. It needs the provider binaries, which this machine could not download, so it is shown on the page and not run here.0:58 fmt, lint, scan, policy: all before plan, all in the agent's hook1:01 The agent that writes infrastructure never holds the apply credential1:03 prevent_destroy on anything that holds data, by policy
Your browser does not play this video; download it or read the transcript below.
AI SDLC: Claude Code with GitHub pull requests and Actions 1:13
One repository, one agent writing code, and the gates around it: CLAUDE.md for the rules, hooks for what must always happen, subagents for the tests, Actions for the checks, rulesets so nothing merges without them, and a human at the end. The page: the AI SDLC on GitHub, in AI Delivery
Chapters and transcript
0:00 The agent writes. The repository decides.0:04 CLAUDE.md: the rules the agent reads in every session.0:14 Hooks: what must always happen, whatever the model decides.0:22 Subagents: the testing roles, with their own tools and model.0:32 Actions: the same checks on every pull request, from the repository, not from memory.0:38 Claude on the pull request: reviews, and @claude for the work.0:45 The repository's rules close the loop.0:50 Check, score, judge: before a person reads the diff.
0:00 The agent writes. The repository decides. Claude Code, GitHub pull requests and GitHub Actions on the example project from this center. Every file shown is in the repository.0:05 CLAUDE.md: the rules the agent reads in every session. Build and test commands, conventions, the "always" rules. Under 200 lines; context, not enforcement.0:14 Hooks: what must always happen, whatever the model decides. After every edit, format and run the fast tests. Before any shell command, block the dangerous ones with exit 2.0:23 Subagents: the testing roles, with their own tools and model. Playwright's planner, generator and healer, plus a mutation grader that only reads and runs.0:32 Actions: the same checks on every pull request, from the repository, not from memory. Lint, unit, property and contract tests, Playwright, mutation score, IaC validators. Required checks: nothing merges without them.0:39 Claude on the pull request: reviews, and @claude for the work. The official action, in interactive mode for mentions and with a prompt for automated review.0:45 The repository's rules close the loop. Require a pull request, required status checks, review from code owners, block force pushes. Humans own the merge.0:51 Check, score, judge: before a person reads the diff. Deterministic checks first (format, tests, scanners), then scores (mutation score, eval thresholds, coverage of the spec), then a judgment (Claude's review, then yours). The person reads the scores and samples the judgments, and merges.0:55 CLAUDE.md is context; hooks are enforcement0:58 Secrets stay in GitHub Secrets; the agent's workflow gets the least permission1:01 Review Claude's changes before merging, as the docs say
Watch
Practice and direction
The habits, and where this is going.
Your browser does not play this video; download it or read the transcript below.
AI test automation: best practices 1:21
Eight habits that keep AI speed from costing quality, each tied to a page of this center. Which tool, layer by layer
Chapters and transcript
0:00 Faster is easy. Better takes habits.0:04 Own the oracle. Grade the tests.0:21 Properties and contracts beat examples.0:37 Heal with a human on the diff. Isolate the tester.0:54 Evals are tests. Budgets are tests.1:11 Use the runner your team already runs in CI, and add the AI to it, not around it.
0:00 Faster is easy. Better takes habits. Eight of them, from the examples and the incidents in this center.0:05 Own the oracle. Grade the tests. The expected value is the part only a person can supply; an assistant will happily assert the bug. Then run mutation testing, because a passing test that kills nothing is maintenance with no value.0:09 Check every expected value by hand or by a second source0:13 Mutation score, not coverage, is the number to watch0:21 Properties and contracts beat examples. State the invariants and let Hypothesis invent the inputs. Let the schema generate the API cases with Schemathesis, and keep Pact where two teams meet.0:26 Invariants: never, always, round trip0:29 The contract is the specification an agent cannot argue with0:38 Heal with a human on the diff. Isolate the tester. Playwright's healer, Cypress's cy.prompt() and Healenium all repair tests; all three can paper over a real bug. And a browser agent with saved logins is an attack surface the incidents in this center describe.0:42 Every heal is a pull request; money and deletion are never healed unattended0:45 Test credentials only, fresh profiles, an egress allowlist0:54 Evals are tests. Budgets are tests. Deterministic assertions for the facts, a pinned rubric judge for the judgment, sampled by hand each release. And a token, time and tool-call budget around every agent, including the ones that write and heal your tests.0:59 promptfoo or DeepEval in CI, with thresholds1:02 An agent that loops is a bill; a test catches it1:11 Use the runner your team already runs in CI, and add the AI to it, not around it. Every example on the Examples page follows it: pytest, Vitest, Playwright, k6, the Inspector CLI, with the AI layer a command away.
Your browser does not play this video; download it or read the transcript below.
Where AI test automation is headed 0:44
Six directions, each already visible in a tool or an incident in this center. Where this is going
Chapters and transcript
0:00 Shift left meant test earlier. At AI speed it means test before a person reads the change.0:06 Early validation: check, score, judge.0:12 The spec is the oracle.0:18 Failures written for machines.0:24 Production traces become tests.0:30 Provenance for every test.0:36 The tester is an agent. Secure it.
0:00 Shift left meant test earlier. At AI speed it means test before a person reads the change. Six directions, each already visible somewhere in this center.0:06 Early validation: check, score, judge. A validation stage in front of the review queue. Deterministic checks, then scores (mutation, evals, spec coverage), then a judgment, with the person reading scores and sampling judgments.0:12 The spec is the oracle. When agents write the code, the human-owned spec is where intent lives, and the tests are its executable form: properties, contracts, evals and a plan in Markdown before Playwright generates a line.0:18 Failures written for machines. Playwright already writes an error-context file addressed to an AI, with the page snapshot inside. Expect every tool's failure to arrive this way: structured, reproducible, ready for an agent to act on.0:24 Production traces become tests. The prompt, the tools called and the answer given are a replayable eval. Suites will grow from what happened, not from what someone imagined.0:30 Provenance for every test. Who or what wrote it, from which spec, with what evidence, healed by whom and approved by whom. This center's Test Passport proposes a format and a gate for it.0:36 The tester is an agent. Secure it. Browser agents with credentials, self-healing drivers on payment pages, eval suites with provider keys: the incidents in the Learning Center are the threat model, and the harness tests here are the controls.
Where the tests come from The AI Security Learning Center follows AI security every day: incidents,
weaknesses, fixed CVEs and practices, every claim sourced. A new incident there is a new test here. And
AI Delivery shows the lifecycle these tests run in, before and after.
New to the technology itself? AI Explained shows how it works.
The open-source tools many of these tests use are in Open-source repos .
Open the Learning Center