What changes
Before and after
The classic way
- Testing is a phase at the end; a QA team finds what the sprint left behind.
- A flaky suite is tolerated and re-run until green.
- Nobody can say who wrote a test, against which requirement, or whether it was ever seen to fail.
The AI way
- Tests are written with the code, by the same agent or a subagent that may not touch the source, and every real incident in the Learning Center becomes a test in the test center that would have caught it [11].
- A test an agent wrote carries a passport: who or what wrote it, to which spec, with what evidence, healed by whom [12].
- AI output is not deterministic, so evals gate on a pass rate with a confidence bound instead of one green run, with promptfoo and pytest shown on real runs [13].
Step by step
The three steps of Test
- 1
Tests arrive with the change AI
The pull request carries unit, property and contract tests written against the spec. A subagent that may only write tests grades them by mutation score, so a test that cannot fail is caught [10].
- 2
Checks run in parallel Gate
Lint, unit, property, contract, the eval suite and the Test Passport check all start on the pull_request event; each is a required status check, so none can be skipped [10].
- 3
The pass rate decides Gate
For the AI parts, the gate is a pass rate with a confidence bound over repeated runs, not a single green [13]. The number is in the repository, where a person set it.
The gate
Required status checks the repository will not merge without, and a pass-rate gate for everything a model touches.
What the AI does here
Writes and heals tests, grades them, writes a regression test for every failure it finds, and leaves the source alone while doing it.
Tools people use at this stage
- pytest and Hypothesis
- mutmut and Stryker
- Schemathesis
- promptfoo and DeepEval
- Playwright test agents
Named as examples, not endorsements; the center is not affiliated with any of them.
What this stage adds
The risks, and the practices that answer them
Each risk is a weakness in the Learning Center, with the incidents where it happened; each practice has its sources there.
Risks
- Overreliance on AI outputA green suite nobody checked is trusted.
- Agent loops and runaway tool callsA test-healing loop never ends.
Read for this stage