Provenance for tests, now that tests have authors that are not people. A format (v0.1), a deterministic grade, a generator and a verifier that run in this page, one file for CI, and our own tests graded by the same rules.
Why
Tests now have authors that are not people. Nothing records it.
A suite can hold a test a person wrote to the spec, one an assistant generated from the implementation (so it asserts the bug), one the healer rewrote last Tuesday, and one nobody has looked at since the model that wrote it was retired. They all look the same in the repository and the same in the report. Software has provenance formats for how an artifact was built and bills of materials for what is in it; we have not found one for the tests themselves: who or what wrote each test, to which specification, where its expected values came from, who reviewed it, and what evidence it has earned. The Test Passport is our proposal. If a format for this exists, tell us and we will link it.
What exists
SLSA provenance describes "where, when, and how something was produced" for build artifacts. in-toto attestations are "authenticated metadata about one or more software artifacts", with predicate types that include SLSA provenance, SPDX and CycloneDX bills of materials, and a Test Result predicate, "a generic schema to express results of any type of tests". Results, not authorship, oracle or review.
The three facts that decide whether a green test means anything: who or what wrote it, where its expected values came from, and whether a person has looked. Plus the two that decay: the evidence (mutation score, flakiness) and the heals.
What it is for
A gate in CI (every test file has one, and it clears a grade), a review aid (the pull request shows the passport diff beside the test diff), and an audit answer: which tests in this release were written by a model and never reviewed?
The format
One small YAML file beside each test file
tests/test_pricing_generated.py gets tests/test_pricing_generated.passport.yaml. Eleven fields, five of them required; the rest earn points. The example is from the example project, where this test was written by an assistant, pruned after mutation testing and reviewed by a person.
An AI-written unit test, reviewed
tests/test_pricing_generated.passport.yaml
passport: "0.1"
file: tests/test_pricing_generated.py
spec: docs/pricing.md
authored_by:
kind: ai
model: claude-sonnet-4-5
tool: claude-code
date: 2026-10-25
prompt: "Write pytest cases for apply_discount from its docstring and docs/pricing.md; cover the edges."
oracle:
source: spec
checked_by: acme developer # each expected value re-derived from docs/pricing.md by hand
reviewed_by:
- who: acme developer
date: 2026-10-25
what: deleted one case that asserted the implementation's rounding, kept the rest
evidence:
mutants: { killed: 23, total: 27 } # mutmut run, 2026-10-25, human and generated suites together
runs: 12
flaky: 0
last_run: 2026-10-25
heals: []
sources:
- docs/pricing.md
A healed end-to-end test
e2e/tests/checkout.passport.yaml
passport: "0.1"
file: e2e/tests/checkout.spec.ts
spec: docs/checkout-flow.md
authored_by:
kind: human
who: acme developer
date: 2026-10-18
oracle:
source: spec
checked_by: acme developer
reviewed_by:
- who: quality code owner
date: 2026-10-19
what: locators are role-based, assertions are web-first
evidence:
runs: 60
flaky: 0
last_run: 2026-10-25
heals:
- on: 2026-10-25
by: playwright-test-healer
change: "getByRole('button', { name: 'Pay $119.00' }) became getByRole('button', { name: 'Place order' })"
reason: "the design change renamed the button; the page snapshot in error-context.md shows the new name"
approved_by: acme developer
sources:
- https://playwright.dev/docs/test-agents
Field
Status
Meaning
passport
required
The format version, "0.1".
file
required
The test file this passport belongs to.
spec
required
Where the intent lives: a doc, a ticket, an issue, a schema. Owned by a person.
authored_by.kind
required
human, ai or pair.
authored_by.model
required for ai and pair
Which model wrote it. tool, who, date and prompt are welcome.
oracle.source
required
Where the expected values came from: spec, human, production, or implementation (read off the code under test).
oracle.checked_by
scored
The person who confirmed the expected values.
reviewed_by
required, may be empty
Each entry: who, date, what. Empty means the review is pending, and the grade says so.
evidence
scored
mutation_score or mutants {killed, total}; runs and flaky; last_run; anything else the test can show for itself.
heals
scored
Self-healing changes: date, by, change, reason, approved_by. A heal nobody approved caps the grade at D.
sources
optional
The documentation the test was written to.
A JSON Schema for the format is in the download below. The key is date, not on: YAML reads a bare on as the boolean true, the same trap GitHub Actions workflows live with.
The grade
Ten points, four grades, three caps
The grade is deterministic and the rules are short enough to read in a minute. A from 9 points, B from 7 points, C from 4 points, D below. The caps are the point of the format: no amount of mutation score makes an unreviewed AI-written test better than a C, and a heal nobody approved is a D whatever else is true.
Evidence
Points
Why
oracle.source is spec or human
3
The expected values came from the specification or a person.
oracle.source is production
2
Replayed from what really happened: real, but not reviewed intent.
oracle.source is implementation
0
Read off the code under test. Caps the grade at C: the test proves the code does what it does.
oracle.checked_by
1
A person confirmed the expected values.
reviewed_by not empty
2
A person reviewed the test. An AI-written or pair-written test with no review is capped at C.
mutation score 0.80 or more
2
Most planted bugs are caught.
mutation score 0.60 to 0.79
1
Some are.
10 or more runs, none flaky
1
It passes for the right reasons.
no heals, or every heal approved
1
A self-healing change nobody approved caps the grade at D.
The rules live once, in passport_check.py; this page's generator and verifier mirror them in JavaScript, and the site's build runs six fixed passports through both and refuses to publish when they disagree.
Make one
The generator: describe the test, get the passport, the grade and the badge
tests/test_search.passport.yaml
Try it: clear the reviewer and watch an A become a C; set the oracle to "running the code" and see why.
Check one
The verifier: paste a passport, read the verdict
Problems make a passport invalid; caps lower the grade; notes are what to improve. The same words passport_check.py prints in CI.
In CI
One file, one job, one required check
passport_check.py finds the test files under the directories you name, expects a passport beside each, verifies and grades them, prints the verdicts, writes a badge, and exits 1 below --min-grade. PyYAML is its only dependency. Make the job a required status check and no test file merges without a passport that clears the bar.
.github/workflows/passports.yml
passports.yml
# .github/workflows/passports.yml: every test file carries a passport that verifies and clears a B.
# The script is one file with no dependency beyond PyYAML; copy it to scripts/ and make the job a
# required status check in your ruleset.
name: Test Passports
on:
pull_request:
paths: ["tests/**", "e2e/tests/**", "scripts/passport_check.py"]
permissions:
contents: read
jobs:
passports:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@v6
- uses: actions/setup-python@v6
with: { python-version: "3.12" }
- run: pip install pyyaml
- run: python scripts/passport_check.py tests e2e/tests --min-grade B --badge passport-badge.svg
- uses: actions/upload-artifact@v5
if: always()
with: { name: passport-badge, path: passport-badge.svg }
What it prints (the example project, 2026-10-25)
$ python scripts/passport_check.py tests e2e/tests --min-grade B
B 8 pts tests/test_orders_api.py
note: no mutation score: nobody has measured what these tests catch
A 9 pts tests/test_pricing.py
A 10 pts tests/test_pricing_generated.py
B 7 pts tests/test_tokens_properties.py
note: no mutation score: nobody has measured what these tests catch
B 8 pts e2e/tests/checkout.spec.ts
note: no mutation score: nobody has measured what these tests catch
5 test file(s), lowest grade B, minimum B: ok
passport_check.py, the workflow, the JSON Schema and the two example passports.
This center's 18 harness tests carry passports, and they grade C
Honesty is the point of the format, so here is ours. The harness tests on the home page were written with AI for this site's author, each to a documented incident, and the build checks that they parse and that their incident and OWASP references resolve. No person has reviewed each assertion line by line yet, and no mutation score has been measured, so every one of them is capped at C: exactly the state the passport exists to make visible. The review column is open.
Test file
Specification
Grade
Why
tests/test_gateway_auth.py
incident litellm-mcp-any-bearer-token: LiteLLM: any Bearer token opened the MCP gateway, and attackers used it
C
written with AI and not yet reviewed by a person
tests/test_deployment.py
incident obot-quickstart-no-auth: Obot: the Docker quickstart gave anyone on the network admin rights and the Docker socket
C
written with AI and not yet reviewed by a person
tests/test_local_server.py
incident opencode-upgrade-endpoint-rce: OpenCode: a web page could make the coding agent's local server install an attacker's npm package
C
written with AI and not yet reviewed by a person
tests/test_sandbox_local_api.py
incident deepseek-harness-sandbox-escape: DeepSeek Harness: a sandboxed agent could switch off its own sandbox through an unauthenticated local API
incident gitspawn-git-config-coding-agents: GitSpawn: a repository's .git/config runs code the moment seven coding agents open the folder
C
written with AI and not yet reviewed by a person
tests/test_injected_email.py
incident manus-agent-single-email-hijack: Manus: one email made the agent run attacker code next to the user's Gmail, Drive and GitHub tokens
C
written with AI and not yet reviewed by a person
tests/test_hostile_page.py
incident copilot-cli-cryptographic-context-injection: GitHub Copilot CLI: encrypted instructions on a web page made the agent read .env secrets and send them out
C
written with AI and not yet reviewed by a person
tests/test_output_channels.py
incident echoleak: EchoLeak: zero-click data theft from Microsoft 365 Copilot
C
written with AI and not yet reviewed by a person
tests/test_generated_code.py
incident langflow-ibm-bulletin-two-dozen-flaws: Langflow: IBM lists more than two dozen flaws, two of them code execution with no login, and an LLM-written lambda that runs on the server
C
written with AI and not yet reviewed by a person
tests/test_agent_account.py
incident pixelleak-coding-agents: PixelLeak: coding agents posted 13,000 internal screenshots to public GitHub repositories
C
written with AI and not yet reviewed by a person
tests/test_mcp_config.py
incident postmark-mcp: postmark-mcp: first malicious MCP server found in the wild
C
written with AI and not yet reviewed by a person
tests/test_instruction_files.py
incident rules-file-backdoor: Rules File Backdoor: invisible instructions in AI coding rules files
C
written with AI and not yet reviewed by a person
tests/test_agent_db_role.py
incident replit-agent-database: Replit Agent deletes a production database during a code freeze
C
written with AI and not yet reviewed by a person
tests/test_approval_gate.py
incident amazon-q-wiper-prompt: Amazon Q Developer: a wiper prompt shipped in the VS Code extension
C
written with AI and not yet reviewed by a person
tests/.github/workflows/known-cves.yml
incident mcp-typescript-sdk-oauth: MCP TypeScript SDK: a malicious MCP server could choose where the client's OAuth secrets were sent
C
written with AI and not yet reviewed by a person
tests/test_skill_hosts.py
incident openai-render-skill-hallucinated-domain: OpenAI's official Render skill sent production apps' tokens to a domain that did not exist, until researchers registered it
C
written with AI and not yet reviewed by a person
tests/test_mcp_tool_snapshot.py
incident cursor-mcpoison: MCPoison: a swapped MCP config gives persistent code execution in Cursor
C
written with AI and not yet reviewed by a person
Where it goes
From a file beside the test to a signed statement about it
Tools write the evidence
Mutation tools already know the score per file; Playwright's healer knows what it changed and why; CI knows the runs and the flakes. Each can update its lines of the passport, leaving people only the lines that need a person: the oracle and the review.
The passport is a natural in-toto predicate: a statement whose subject is the test file's digest and whose predicate is the passport, signed by CI, so the claim "reviewed by a person" cannot be edited after the fact.
The grade beside each test in the pull request diff, the suite's lowest grade on the README, and the question "which tests in this release were written by a model and never reviewed?" answered by a query instead of an archaeology.
Version 0.2
Open questions we would like help with: a passport per test function rather than per file; how production replays should count as an oracle; and whether a judge model's verdicts deserve their own evidence lines.
The format, the grading and the checker are ours (v0.1, 2026-10-25); use them freely. Everything on this page that computes was run: the checker on the example project, the six fixed cases through the Python rules and the JavaScript mirror.