Skip to the content
AI Test AutomationPreview AI Security Learning Center

AI Test AutomationTry it yourselfTest Passport

AI Test Automation Preview

Test Passport

Provenance for tests, now that tests have authors that are not people. A format (v0.1), a deterministic grade, a generator and a verifier that run in this page, one file for CI, and our own tests graded by the same rules.

Why

Tests now have authors that are not people. Nothing records it.

A suite can hold a test a person wrote to the spec, one an assistant generated from the implementation (so it asserts the bug), one the healer rewrote last Tuesday, and one nobody has looked at since the model that wrote it was retired. They all look the same in the repository and the same in the report. Software has provenance formats for how an artifact was built and bills of materials for what is in it; we have not found one for the tests themselves: who or what wrote each test, to which specification, where its expected values came from, who reviewed it, and what evidence it has earned. The Test Passport is our proposal. If a format for this exists, tell us and we will link it.

What exists

SLSA provenance describes "where, when, and how something was produced" for build artifacts. in-toto attestations are "authenticated metadata about one or more software artifacts", with predicate types that include SLSA provenance, SPDX and CycloneDX bills of materials, and a Test Result predicate, "a generic schema to express results of any type of tests". Results, not authorship, oracle or review.

SLSA provenance v1: how artifacts were built · in-toto attestation framework: statement, predicate, envelope · in-toto predicate types, including Test Result

What is missing

The three facts that decide whether a green test means anything: who or what wrote it, where its expected values came from, and whether a person has looked. Plus the two that decay: the evidence (mutation score, flakiness) and the heals.

What it is for

A gate in CI (every test file has one, and it clears a grade), a review aid (the pull request shows the passport diff beside the test diff), and an audit answer: which tests in this release were written by a model and never reviewed?

The format

One small YAML file beside each test file

tests/test_pricing_generated.py gets tests/test_pricing_generated.passport.yaml. Eleven fields, five of them required; the rest earn points. The example is from the example project, where this test was written by an assistant, pruned after mutation testing and reviewed by a person.

An AI-written unit test, reviewed

tests/test_pricing_generated.passport.yaml
passport: "0.1"
file: tests/test_pricing_generated.py
spec: docs/pricing.md
authored_by:
  kind: ai
  model: claude-sonnet-4-5
  tool: claude-code
  date: 2026-10-25
  prompt: "Write pytest cases for apply_discount from its docstring and docs/pricing.md; cover the edges."
oracle:
  source: spec
  checked_by: acme developer   # each expected value re-derived from docs/pricing.md by hand
reviewed_by:
  - who: acme developer
    date: 2026-10-25
    what: deleted one case that asserted the implementation's rounding, kept the rest
evidence:
  mutants: { killed: 23, total: 27 }   # mutmut run, 2026-10-25, human and generated suites together
  runs: 12
  flaky: 0
  last_run: 2026-10-25
heals: []
sources:
  - docs/pricing.md

A healed end-to-end test

e2e/tests/checkout.passport.yaml
passport: "0.1"
file: e2e/tests/checkout.spec.ts
spec: docs/checkout-flow.md
authored_by:
  kind: human
  who: acme developer
  date: 2026-10-18
oracle:
  source: spec
  checked_by: acme developer
reviewed_by:
  - who: quality code owner
    date: 2026-10-19
    what: locators are role-based, assertions are web-first
evidence:
  runs: 60
  flaky: 0
  last_run: 2026-10-25
heals:
  - on: 2026-10-25
    by: playwright-test-healer
    change: "getByRole('button', { name: 'Pay $119.00' }) became getByRole('button', { name: 'Place order' })"
    reason: "the design change renamed the button; the page snapshot in error-context.md shows the new name"
    approved_by: acme developer
sources:
  - https://playwright.dev/docs/test-agents
FieldStatusMeaning
passportrequiredThe format version, "0.1".
filerequiredThe test file this passport belongs to.
specrequiredWhere the intent lives: a doc, a ticket, an issue, a schema. Owned by a person.
authored_by.kindrequiredhuman, ai or pair.
authored_by.modelrequired for ai and pairWhich model wrote it. tool, who, date and prompt are welcome.
oracle.sourcerequiredWhere the expected values came from: spec, human, production, or implementation (read off the code under test).
oracle.checked_byscoredThe person who confirmed the expected values.
reviewed_byrequired, may be emptyEach entry: who, date, what. Empty means the review is pending, and the grade says so.
evidencescoredmutation_score or mutants {killed, total}; runs and flaky; last_run; anything else the test can show for itself.
healsscoredSelf-healing changes: date, by, change, reason, approved_by. A heal nobody approved caps the grade at D.
sourcesoptionalThe documentation the test was written to.

A JSON Schema for the format is in the download below. The key is date, not on: YAML reads a bare on as the boolean true, the same trap GitHub Actions workflows live with.

The grade

Ten points, four grades, three caps

The grade is deterministic and the rules are short enough to read in a minute. A from 9 points, B from 7 points, C from 4 points, D below. The caps are the point of the format: no amount of mutation score makes an unreviewed AI-written test better than a C, and a heal nobody approved is a D whatever else is true.

EvidencePointsWhy
oracle.source is spec or human3The expected values came from the specification or a person.
oracle.source is production2Replayed from what really happened: real, but not reviewed intent.
oracle.source is implementation0Read off the code under test. Caps the grade at C: the test proves the code does what it does.
oracle.checked_by1A person confirmed the expected values.
reviewed_by not empty2A person reviewed the test. An AI-written or pair-written test with no review is capped at C.
mutation score 0.80 or more2Most planted bugs are caught.
mutation score 0.60 to 0.791Some are.
10 or more runs, none flaky1It passes for the right reasons.
no heals, or every heal approved1A self-healing change nobody approved caps the grade at D.

The rules live once, in passport_check.py; this page's generator and verifier mirror them in JavaScript, and the site's build runs six fixed passports through both and refuses to publish when they disagree.

Make one

The generator: describe the test, get the passport, the grade and the badge

    tests/test_search.passport.yaml

    Try it: clear the reviewer and watch an A become a C; set the oracle to "running the code" and see why.

    Check one

    The verifier: paste a passport, read the verdict

      Problems make a passport invalid; caps lower the grade; notes are what to improve. The same words passport_check.py prints in CI.

      In CI

      One file, one job, one required check

      passport_check.py finds the test files under the directories you name, expects a passport beside each, verifies and grades them, prints the verdicts, writes a badge, and exits 1 below --min-grade. PyYAML is its only dependency. Make the job a required status check and no test file merges without a passport that clears the bar.

      .github/workflows/passports.yml

      passports.yml
      # .github/workflows/passports.yml: every test file carries a passport that verifies and clears a B.
      # The script is one file with no dependency beyond PyYAML; copy it to scripts/ and make the job a
      # required status check in your ruleset.
      name: Test Passports
      on:
        pull_request:
          paths: ["tests/**", "e2e/tests/**", "scripts/passport_check.py"]
      permissions:
        contents: read
      jobs:
        passports:
          runs-on: ubuntu-latest
          timeout-minutes: 5
          steps:
            - uses: actions/checkout@v6
            - uses: actions/setup-python@v6
              with: { python-version: "3.12" }
            - run: pip install pyyaml
            - run: python scripts/passport_check.py tests e2e/tests --min-grade B --badge passport-badge.svg
            - uses: actions/upload-artifact@v5
              if: always()
              with: { name: passport-badge, path: passport-badge.svg }

      What it prints (the example project, 2026-10-25)

      $ python scripts/passport_check.py tests e2e/tests --min-grade B
              B   8 pts  tests/test_orders_api.py
                 note: no mutation score: nobody has measured what these tests catch
              A   9 pts  tests/test_pricing.py
              A  10 pts  tests/test_pricing_generated.py
              B   7 pts  tests/test_tokens_properties.py
                 note: no mutation score: nobody has measured what these tests catch
              B   8 pts  e2e/tests/checkout.spec.ts
                 note: no mutation score: nobody has measured what these tests catch
      5 test file(s), lowest grade B, minimum B: ok

      passport_check.py, the workflow, the JSON Schema and the two example passports.

      GitHub rulesets: required status checks

      Our own tests

      This center's 18 harness tests carry passports, and they grade C

      Honesty is the point of the format, so here is ours. The harness tests on the home page were written with AI for this site's author, each to a documented incident, and the build checks that they parse and that their incident and OWASP references resolve. No person has reviewed each assertion line by line yet, and no mutation score has been measured, so every one of them is capped at C: exactly the state the passport exists to make visible. The review column is open.

      Test fileSpecificationGradeWhy
      tests/test_gateway_auth.pyincident litellm-mcp-any-bearer-token: LiteLLM: any Bearer token opened the MCP gateway, and attackers used itCwritten with AI and not yet reviewed by a person
      tests/test_deployment.pyincident obot-quickstart-no-auth: Obot: the Docker quickstart gave anyone on the network admin rights and the Docker socketCwritten with AI and not yet reviewed by a person
      tests/test_local_server.pyincident opencode-upgrade-endpoint-rce: OpenCode: a web page could make the coding agent's local server install an attacker's npm packageCwritten with AI and not yet reviewed by a person
      tests/test_sandbox_local_api.pyincident deepseek-harness-sandbox-escape: DeepSeek Harness: a sandboxed agent could switch off its own sandbox through an unauthenticated local APICwritten with AI and not yet reviewed by a person
      tests/test_egress.pyincident openai-agents-55-sites-staging: OpenAI agents reached staging servers among 55 websites, Asymmetric Security findsCwritten with AI and not yet reviewed by a person
      tests/test_untrusted_repo.pyincident gitspawn-git-config-coding-agents: GitSpawn: a repository's .git/config runs code the moment seven coding agents open the folderCwritten with AI and not yet reviewed by a person
      tests/test_injected_email.pyincident manus-agent-single-email-hijack: Manus: one email made the agent run attacker code next to the user's Gmail, Drive and GitHub tokensCwritten with AI and not yet reviewed by a person
      tests/test_hostile_page.pyincident copilot-cli-cryptographic-context-injection: GitHub Copilot CLI: encrypted instructions on a web page made the agent read .env secrets and send them outCwritten with AI and not yet reviewed by a person
      tests/test_output_channels.pyincident echoleak: EchoLeak: zero-click data theft from Microsoft 365 CopilotCwritten with AI and not yet reviewed by a person
      tests/test_generated_code.pyincident langflow-ibm-bulletin-two-dozen-flaws: Langflow: IBM lists more than two dozen flaws, two of them code execution with no login, and an LLM-written lambda that runs on the serverCwritten with AI and not yet reviewed by a person
      tests/test_agent_account.pyincident pixelleak-coding-agents: PixelLeak: coding agents posted 13,000 internal screenshots to public GitHub repositoriesCwritten with AI and not yet reviewed by a person
      tests/test_mcp_config.pyincident postmark-mcp: postmark-mcp: first malicious MCP server found in the wildCwritten with AI and not yet reviewed by a person
      tests/test_instruction_files.pyincident rules-file-backdoor: Rules File Backdoor: invisible instructions in AI coding rules filesCwritten with AI and not yet reviewed by a person
      tests/test_agent_db_role.pyincident replit-agent-database: Replit Agent deletes a production database during a code freezeCwritten with AI and not yet reviewed by a person
      tests/test_approval_gate.pyincident amazon-q-wiper-prompt: Amazon Q Developer: a wiper prompt shipped in the VS Code extensionCwritten with AI and not yet reviewed by a person
      tests/.github/workflows/known-cves.ymlincident mcp-typescript-sdk-oauth: MCP TypeScript SDK: a malicious MCP server could choose where the client's OAuth secrets were sentCwritten with AI and not yet reviewed by a person
      tests/test_skill_hosts.pyincident openai-render-skill-hallucinated-domain: OpenAI's official Render skill sent production apps' tokens to a domain that did not exist, until researchers registered itCwritten with AI and not yet reviewed by a person
      tests/test_mcp_tool_snapshot.pyincident cursor-mcpoison: MCPoison: a swapped MCP config gives persistent code execution in CursorCwritten with AI and not yet reviewed by a person

      Where it goes

      From a file beside the test to a signed statement about it

      Show it where decisions are made

      The grade beside each test in the pull request diff, the suite's lowest grade on the README, and the question "which tests in this release were written by a model and never reviewed?" answered by a query instead of an archaeology.

      Version 0.2

      Open questions we would like help with: a passport per test function rather than per file; how production replays should count as an oracle; and whether a judge model's verdicts deserve their own evidence lines.

      Sources

      Read on these pages, 2026-10-25

      The format, the grading and the checker are ours (v0.1, 2026-10-25); use them freely. Everything on this page that computes was run: the checker on the example project, the six fixed cases through the Python rules and the JavaScript mirror.