Skip to the content
AI Test AutomationPreview AI Security Learning Center

AI Test AutomationLearnVideos

AI Test Automation Preview

Videos: how to test with AI in the loop

Short, captioned and real. Each one teaches one move, shows the tool's own output, and ends with the sources.

How they were made

12 short videos, 12:09 in all, and every command in them ran for real

Short, captioned, and real: every command in these videos ran on the date shown, and the output is unedited (the transcripts are in the repository). Each one teaches one move: how to use the tool with AI in the loop, what to read in its output, and the mistake it catches.

Real sessions, replayed

Each video replays a session recorded on 2026-10-25: the commands were run, their output was saved unedited (the transcripts are in the repository under data/testlab/transcripts/), and a terminal player types the commands back and shows that output. Nothing in a terminal pane was written by hand.

Captions, chapters, transcript

Every video carries a captions track (WebVTT), chapter buttons that seek to the moment, and a full transcript you can read or search instead of watching. The player has no tracking and loads nothing until you press play.

Sources on the last frame

The closing card of each video names the documentation its facts came from; the pages linked under each video carry the same facts with the links. When a tool changes, the transcript keeps the date it was true.

Small on purpose

24 frames a second, 1280 by 720, each file under a megabyte and a half: the whole set weighs 6.8 MB, so the page stays light on a phone and nothing loads until you press play.

One video plays at a time; chapters and transcript lines jump to their moment.

Watch

How to, layer by layer

One move per video, with the tool's own output. Each has a worked example on the Examples page.

Grade AI-written tests with mutation testing 1:00

A passing suite proves little. Mutation testing plants bugs and counts how many the tests catch; run it on the human suite, then with the assistant's tests, and read the survivors. The worked example

Chapters and transcript

Let Hypothesis find the bug 0:46

Property tests state what must always hold; the generator supplies thousands of inputs and shrinks a failure to the smallest case. An assistant is good at proposing properties, and a person must approve them. The worked example

Chapters and transcript

Fuzz the contract: Schemathesis against an OpenAPI schema 0:49

An agent writes a handler that answers, but not the way the contract says. Schemathesis generates the requests from the schema and names the rule each failure breaks. The worked example

Chapters and transcript

Playwright: the failure an agent can read, and the healer 1:08

A role-based test passes, a design change renames the button, the test fails and Playwright writes a failure file addressed to an AI. Then init-agents adds the planner, generator and healer to your repository. The worked example

Chapters and transcript

Load test with Locust, and what changes for an LLM endpoint 0:48

A headless Locust run is a CI step. For model endpoints the metrics change to time to first token and tokens per second, and the risk changes to a bill. The worked example

Chapters and transcript

Evals in CI with promptfoo, no judge model required 0:40

An exact-match test breaks on the next rephrasing. promptfoo runs a matrix of inputs with deterministic assertions, and a failure reads like a test failure, not a vibe. The worked example

Chapters and transcript

Four CI gates for an MCP server 1:02

The MCP Inspector's CLI connects, runs one request and exits with a code a pipeline can branch on. Four gates: the tool list, the schemas, one real call, and one call that must be refused. The worked example

Chapters and transcript

Watch

When the result is not deterministic

What changes when the thing under test words every answer differently, from unit tests to the eval gate.

Testing results that are not deterministic 1:22

A model words every answer differently. Exact match breaks, properties hold, and the gate is a pass rate with a confidence bound over enough samples. Real runs: promptfoo with repeats, and the same idea in pytest. The page: testing nondeterministic results

Chapters and transcript

Watch

Infrastructure and the development lifecycle

The same discipline one level up: validating Terraform for three clouds before a plan, and the repository rules around an agent that writes code.

Terraform for AWS, Azure and GCP: validate before you plan 1:16

An agent writes a storage module for three clouds. Before anything reaches a plan, four validators run without credentials: fmt, tflint, Checkov and a Conftest policy the team owns. Real run on the draft and on the fix. The page: infrastructure as code, in AI Delivery

Chapters and transcript

AI SDLC: Claude Code with GitHub pull requests and Actions 1:13

One repository, one agent writing code, and the gates around it: CLAUDE.md for the rules, hooks for what must always happen, subagents for the tests, Actions for the checks, rulesets so nothing merges without them, and a human at the end. The page: the AI SDLC on GitHub, in AI Delivery

Chapters and transcript

Watch

Practice and direction

The habits, and where this is going.

AI test automation: best practices 1:21

Eight habits that keep AI speed from costing quality, each tied to a page of this center. Which tool, layer by layer

Chapters and transcript

Where AI test automation is headed 0:44

Six directions, each already visible in a tool or an incident in this center. Where this is going

Chapters and transcript