AI DeliveryPreview Learning Center

Stage 04 of 07

Test

Every incident earns a test, and every AI-written test carries its provenance

Quality engineering

  1. 01Plan
  2. 02Architect
  3. 03Build
  4. 04Test
  5. 05Release
  6. 06Operate
  7. 07Govern

What changes

Before and after

The classic way

  • Testing is a phase at the end; a QA team finds what the sprint left behind.
  • A flaky suite is tolerated and re-run until green.
  • Nobody can say who wrote a test, against which requirement, or whether it was ever seen to fail.

The AI way

  • Tests are written with the code, by the same agent or a subagent that may not touch the source, and every real incident in the Learning Center becomes a test in the test center that would have caught it [11].
  • A test an agent wrote carries a passport: who or what wrote it, to which spec, with what evidence, healed by whom [12].
  • AI output is not deterministic, so evals gate on a pass rate with a confidence bound instead of one green run, with promptfoo and pytest shown on real runs [13].

Step by step

The three steps of Test

  1. 1

    Tests arrive with the change AI

    The pull request carries unit, property and contract tests written against the spec. A subagent that may only write tests grades them by mutation score, so a test that cannot fail is caught [10].

  2. 2

    Checks run in parallel Gate

    Lint, unit, property, contract, the eval suite and the Test Passport check all start on the pull_request event; each is a required status check, so none can be skipped [10].

  3. 3

    The pass rate decides Gate

    For the AI parts, the gate is a pass rate with a confidence bound over repeated runs, not a single green [13]. The number is in the repository, where a person set it.

The gate

Required status checks the repository will not merge without, and a pass-rate gate for everything a model touches.

What the AI does here

Writes and heals tests, grades them, writes a regression test for every failure it finds, and leaves the source alone while doing it.

Tools people use at this stage

  • pytest and Hypothesis
  • mutmut and Stryker
  • Schemathesis
  • promptfoo and DeepEval
  • Playwright test agents

Named as examples, not endorsements; the center is not affiliated with any of them.

What this stage adds

The risks, and the practices that answer them

Each risk is a weakness in the Learning Center, with the incidents where it happened; each practice has its sources there.

Read for this stage

Sources

  1. [11] AI Test Automation: every incident earns a test (this site)
  2. [12] AI Test Automation: Test Passport, provenance for tests (this site)
  3. [13] AI Test Automation: testing results that are not deterministic (this site)
  4. [10] AI Test Automation: the AI SDLC on GitHub (this site)