Skip to the content
AI Test AutomationPreview AI Security Learning Center

AI Test AutomationLearnHarness QA

AI Test Automation Preview

Harness QA: one change, many client harnesses

When a team builds Claude Code harnesses for many clients and changes them at AI speed, QA answers two questions for every release: did it reach every client harness, and does it work the way each client uses it. A tool, tests and a gate that answer both, built on how Claude Code really ships plugins.

The question

One change, many client harnesses. Did it reach every one of them?

Open the Harness QA tool Run these checks in your browser on a fleet you can edit.

A harness is what a team builds around Claude Code so developers, testers and product owners at a client all work the same way: the standards in CLAUDE.md, skills for the common jobs, hooks that guard what the agent may touch, permission rules, MCP servers, and the plugins that carry it all. Build one for each of many clients, change it at AI speed, and every change raises a question code review cannot answer: did it reach every client harness or only the main one, and does it work the way each client uses it?

The main harness always looks fine

The harness team adds its marketplace from a local folder, and Claude Code loads a plugin from there in place, at every session start, whatever its version says. Clients get a cached copy that changes only when the version does. A green check on the main harness proves the change works in one place.

Claude Code: host and maintain a marketplace (versions, auto-update, channels)

The people who built it cannot test it alone

The developers who deploy a harness test the path they built. QA's job is the other paths: the client on the stable channel, the client whose deny rule blocks the new skill, the product owner who types the request a different way.

Backstage, with the developers

Most of this work is reading configuration and running checks before anything reaches a client. Everything on this page runs from a terminal or a pipeline, and the parts that cost nothing run on every pull request.

What a harness is

Seven parts, seven ways to fail, seven tests

PartHow it reaches a clientHow it fails thereThe test
CLAUDE.mdIn the client's repository, beside the client's codeDrifts from the agency standard; contradicts a skill; grows until the rule that matters is lostA diff against the standard each release; an eval whose prompt the rule should answer
SkillsA plugin (namespaced /plugin:skill) or the project's .claude/skillsNot chosen on natural phrasing; a project skill of the same name answers instead; its allowed-tools are denied by the clientAn eval with a tool_used: Skill grader; the fan-out shadow and permission checks
HooksA plugin's hooks/hooks.json and every settings scope, merged: every match runsA crash is exit 1, and exit 1 lets the action through; an unquoted path splits; the matcher misses a toolContract tests with the documented stdin; claude plugin validate; a test per exit path
Permission rulesSettings at each scope; lists merge; deny, then ask, then allowA client deny blocks a new workflow; a local allow does not beat a project askDecide each new skill's tools against the client's merged rules
MCP serversThe project's .mcp.json, a plugin, or managed settingsNot loaded in a scripted run; a startup timeout; a renamed toolmcp_servers and mcp_server_errors in the stream-json init event; mocked servers in evals
Plugins and marketplacesA marketplace per channel; the version comes from plugin.json firstNo version bump, so no update; auto-update off; enabled in project settings but never installedThe fan-out check below, on every release
Managed settingsMDM, a managed-settings.json file, or the admin consoleOverrides every other scope; only one managed source applies by defaultResolve each client with its managed copy, as the tool does

Claude Code: settings files and precedence · Claude Code: configure permissions (deny, then ask, then allow) · Claude Code: hooks reference (events, exit codes, matchers, decisions) · Claude Code: skills (frontmatter, allowed-tools, plugin namespacing) · Claude Code: manage plugins for your organization

Did it reach everyone?

The fan-out board: two releases, five client harnesses, one honest answer each

This board is the output of harnessqa fanout on a sample fleet, run when this page was built: an agency's two plugins on a stable and a latest channel, its own reference harness, and five clients, each with its own settings and managed policy. For each client it resolves the settings that client really runs, finds the version its channel serves and the version it has installed, and checks that the new skills can run under its permission rules.

Client harnesscore-harness 1.8.0guard_secrets also checks Edit; new release-notes skillfin-guardrails 1.2.0the money guard also flags round() on amounts
mainlocal directoryNEXT SESSIONloads the plugin's current files at every session start (or /reload-plugins), whatever the version saysNEXT SESSIONloads the plugin's current files at every session start (or /reload-plugins), whatever the version says
brokerageagency-latestBLOCKEDauto-update is on for this marketplace (managed settings); skill release-notes needs Bash(gh pr list *); denied by Bash(gh *) in project settings; a project skill is also called release-notes: /release-notes runs the project's copy; the release is /core-harness:release-notesNEVERinstalled 1.2.0 and the release is also 1.2.0: the cached copy is never replaced
lenderagency-latestAFTER UPDATEauto-update is off: stays on 1.7.0 until someone runs claude plugin update core-harness@agency-latestNEVERinstalled 1.2.0 and the release is also 1.2.0: the cached copy is never replaced
retaileragency-latestREACHESauto-update is on for this marketplace (managed settings)NOT INSTALLEDfin-guardrails is not in enabledPlugins
insureragency-stableBEHINDagency-stable serves ref stable (1.7.0); the change is on latestNOT INSTALLEDfin-guardrails is not in enabledPlugins

core-harness: gate FAIL fin-guardrails: gate FAIL

  • fin-guardrails: files changed between stable and latest but the version is still 1.2.0; every cached install keeps the old copy. Bump "version" in plugin.json.

The delivery rules it checks

  • A cached install gets a new copy only when the computed version changes: plugin.json first, then the marketplace entry, else the commit.
  • Background auto-update is off until a user or an admin turns it on for that marketplace.
  • One marketplace serves one ref of each plugin, so a stable channel is a second marketplace.
  • A plugin skill loads as /plugin:skill beside a project skill of the same name.
  • Deny rules win over whatever a skill's allowed-tools pre-approves.

Claude Code: host and maintain a marketplace (versions, auto-update, channels) · Claude Code: skills (frontmatter, allowed-tools, plugin namespacing) · Claude Code: configure permissions (deny, then ask, then allow)

And the ones it cannot see from configuration

  • A plugin enabled in a repository's settings whose marketplace points at its own repository does not install from the settings alone: each person sees "enabled in project settings but isn't installed" until they install it.
  • Repository marketplaces apply only in folders the person trusted; a -p run never accepts trust.
  • In -p and CI runs, plugins install in the background and can miss the first turn; CLAUDE_CODE_SYNC_PLUGIN_INSTALL=1 makes the run wait.

Claude Code: manage plugins for your organization. These show up in the sessions instead: see the lender's load failures under Test it their way.

python -m harnessqa fanout fleet/fleet.yaml

release core-harness 1.8.0 (agency/core-harness#latest, was #stable): guard_secrets also checks Edit; new release-notes skill
  client     status        channel          serves   installed
  retailer   REACHES       agency-latest    1.8.0    1.7.0
               why: auto-update is on for this marketplace (managed settings)
  main       NEXT SESSION  local directory  1.8.0    in place
               why: loads the plugin's current files at every session start (or /reload-plugins), whatever the version says
  lender     AFTER UPDATE  agency-latest    1.8.0    1.7.0
               why: auto-update is off: stays on 1.7.0 until someone runs claude plugin update core-harness@agency-latest
  insurer    BEHIND        agency-stable    1.7.0    1.7.0
               why: agency-stable serves ref stable (1.7.0); the change is on latest
  brokerage  BLOCKED       agency-latest    1.8.0    1.7.0
               why: auto-update is on for this marketplace (managed settings)
               why: skill release-notes needs Bash(gh pr list *); denied by Bash(gh *) in project settings
               note: a project skill is also called release-notes: /release-notes runs the project's copy; the release is /core-harness:release-notes
  2 of 5 client harnesses that run core-harness get this release now; gate FAIL

release fin-guardrails 1.2.0 (agency/fin-guardrails#latest, was #stable): the money guard also flags round() on amounts
  RELEASE  fin-guardrails: files changed between stable and latest but the version is still 1.2.0; every cached install keeps the old copy. Bump "version" in plugin.json.
  client     status        channel          serves   installed
  main       NEXT SESSION  local directory  1.2.0    in place
               why: loads the plugin's current files at every session start (or /reload-plugins), whatever the version says
  brokerage  NEVER         agency-latest    1.2.0    1.2.0
               why: installed 1.2.0 and the release is also 1.2.0: the cached copy is never replaced
  lender     NEVER         agency-latest    1.2.0    1.2.0
               why: installed 1.2.0 and the release is also 1.2.0: the cached copy is never replaced
  insurer    NOT INSTALLED                           
               why: fin-guardrails is not in enabledPlugins
  retailer   NOT INSTALLED                           
               why: fin-guardrails is not in enabledPlugins
  1 of 3 client harnesses that run fin-guardrails get this release now; gate FAIL

exit 1

The inventory QA keeps

fleet/fleet.yaml
# The QA inventory for a fleet of client harnesses. A sample fleet: the agency, its plugins and
# its clients are invented. What each client runs is read from its own files under clients/;
# "installed" is what `claude plugin list` reported on that client's machines at the last check.
marketplaces:
  agency-dev: marketplaces/agency-dev.json          # a local directory, on the harness team's machines
  agency-latest: marketplaces/agency-latest.json    # github: agency/harness-latest
  agency-stable: marketplaces/agency-stable.json    # github: agency/harness-stable
refs:                                               # a checkout of each git ref the marketplaces point at
  agency/core-harness#stable: plugins/core-harness/stable
  agency/core-harness#latest: plugins/core-harness/latest
  agency/fin-guardrails#stable: plugins/fin-guardrails/stable
  agency/fin-guardrails#latest: plugins/fin-guardrails/latest
clients:
  main:
    root: clients/main
    install: in-place
    sources:
      core-harness: plugins/core-harness/latest
      fin-guardrails: plugins/fin-guardrails/latest
  brokerage:
    root: clients/brokerage
    installed: {core-harness: 1.7.0, fin-guardrails: 1.2.0}
  lender:
    root: clients/lender
    installed: {core-harness: 1.7.0, fin-guardrails: 1.2.0}
  retailer:
    root: clients/retailer
    installed: {core-harness: 1.7.0}
  insurer:
    root: clients/insurer
    installed: {core-harness: 1.7.0}
releases:
  - plugin: core-harness
    repo: agency/core-harness
    ref: latest
    previous: stable
    note: guard_secrets also checks Edit; new release-notes skill
  - plugin: fin-guardrails
    repo: agency/fin-guardrails
    ref: latest
    previous: stable
    note: the money guard also flags round() on amounts
usage: usage

Hooks are code

A guard that crashes lets everything through

Claude Code reads a command hook by its exit code. Exit 2 blocks the action and shows stderr as the reason. Exit 0 lets it through, and may carry a JSON decision. Any other code, including the 1 a crashed Python script returns, is a non-blocking error: the action goes ahead. So the core harness's secrets guard, which read tool_input["content"], blocked a credential written with Write and waved the same credential through on an Edit, which sends new_string instead. The contract tests feed each hook the documented stdin for every tool its matcher covers, and found it.

1.7.0, stable: fails open on Edit

plugins/core-harness/stable/hooks/guard_secrets.py
"""PreToolUse guard: refuse to write a file that contains a credential. core-harness 1.7.0."""
import json
import re
import sys

PATTERNS = [
    re.compile(r"AKIA[0-9A-Z]{16}"),
    re.compile(r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----"),
    re.compile(r"(?i)(api[_-]?key|secret)\s*[:=]\s*['\"][A-Za-z0-9_\-]{24,}['\"]"),
]

event = json.load(sys.stdin)
text = event["tool_input"]["content"]
for p in PATTERNS:
    if p.search(text):
        print(f"Blocked: {event['tool_input']['file_path']} would contain a credential ({p.pattern[:20]}...).", file=sys.stderr)
        sys.exit(2)
sys.exit(0)

1.8.0, latest: reads both, fails closed

plugins/core-harness/latest/hooks/guard_secrets.py
"""PreToolUse guard: refuse to write a file that contains a credential. core-harness 1.8.0.

Write sends "content", Edit sends "new_string"; anything the guard cannot read is refused
(exit 2) instead of crashing, because a crash is exit 1 and Claude Code carries on after it."""
import json
import re
import sys

PATTERNS = [
    re.compile(r"AKIA[0-9A-Z]{16}"),
    re.compile(r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----"),
    re.compile(r"(?i)(api[_-]?key|secret)\s*[:=]\s*['\"][A-Za-z0-9_\-]{24,}['\"]"),
]


def texts(tool_input):
    for key in ("content", "new_string"):
        if isinstance(tool_input.get(key), str):
            yield tool_input[key]
    for edit in tool_input.get("edits", []) or []:
        if isinstance(edit, dict) and isinstance(edit.get("new_string"), str):
            yield edit["new_string"]


try:
    event = json.load(sys.stdin)
    tool_input = event["tool_input"]
    path = tool_input.get("file_path", "?")
except (ValueError, KeyError, TypeError):
    print("Blocked: the secrets guard could not read this tool call.", file=sys.stderr)
    sys.exit(2)

for text in texts(tool_input):
    for p in PATTERNS:
        if p.search(text):
            print(f"Blocked: {path} would contain a credential ({p.pattern[:20]}...).", file=sys.stderr)
            sys.exit(2)
sys.exit(0)

The contract tests

tests/test_hooks.py
"""Hooks are code that runs on every tool call: test them like code, with the documented stdin and
the documented exit codes. The stable guard has a real bug these tests found: on an Edit it
crashes, a crash is exit 1, and Claude Code carries on after exit 1. It fails open."""
import json

import pytest
from conftest import FLEET, secret

from harnessqa import hooks as hk

CORE = FLEET / "plugins" / "core-harness"
BROKERAGE = FLEET / "clients" / "brokerage"


def guard(ref):
    cfg = json.loads((CORE / ref / "hooks" / "hooks.json").read_text(encoding="utf-8"))
    return lambda tool, inp: hk.run(hk.handlers(cfg, "PreToolUse", tool)[0], hk.payload("PreToolUse", tool, inp), plugin_root=CORE / ref)


def test_matcher_rules():
    assert hk.matches("Write|Edit", "Edit") and not hk.matches("Write|Edit", "MultiEditX")
    assert hk.matches("mcp__memory__.*", "mcp__memory__read")
    assert not hk.matches("mcp__memory", "mcp__memory__read")
    assert hk.matches(None, "Bash") and hk.matches("*", "Bash")


def test_guard_only_runs_on_file_writes():
    cfg = json.loads((CORE / "latest" / "hooks" / "hooks.json").read_text(encoding="utf-8"))
    assert hk.handlers(cfg, "PreToolUse", "Bash") == []
    assert len(hk.handlers(cfg, "PreToolUse", "Edit")) == 1


@pytest.mark.parametrize("ref", ["stable", "latest"])
def test_write_with_a_key_is_blocked(ref):
    v = guard(ref)("Write", {"file_path": "app/config.py", "content": f"KEY = '{secret()}'"})
    assert v.exit_code == 2 and v.blocked and "credential" in v.reason


@pytest.mark.parametrize("ref", ["stable", "latest"])
def test_clean_write_passes(ref):
    v = guard(ref)("Write", {"file_path": "app/config.py", "content": "TIMEOUT = 30"})
    assert v.exit_code == 0 and not v.blocked


def test_stable_guard_fails_open_on_edit():
    """The bug, pinned: 1.7.0 reads tool_input["content"], which an Edit does not send."""
    v = guard("stable")("Edit", {"file_path": "app/config.py", "old_string": "KEY = ''", "new_string": f"KEY = '{secret()}'"})
    assert v.failed_open and not v.blocked
    assert "KeyError" in v.stderr


def test_latest_guard_blocks_the_same_edit():
    v = guard("latest")("Edit", {"file_path": "app/config.py", "old_string": "KEY = ''", "new_string": f"KEY = '{secret()}'"})
    assert v.exit_code == 2 and v.blocked


def test_latest_guard_fails_closed_on_input_it_cannot_read():
    cfg = json.loads((CORE / "latest" / "hooks" / "hooks.json").read_text(encoding="utf-8"))
    v = hk.run(hk.handlers(cfg, "PreToolUse", "Write")[0], "not json", plugin_root=CORE / "latest")
    assert v.exit_code == 2


def prod(cmd):
    cfg = json.loads((BROKERAGE / ".claude" / "settings.json").read_text(encoding="utf-8"))
    h = hk.handlers(cfg, "PreToolUse", "Bash")[0]
    return hk.run(h, hk.payload("PreToolUse", "Bash", {"command": cmd}), project_dir=BROKERAGE)


@pytest.mark.parametrize("cmd", [
    "kubectl get pods -n trading-prod",
    "kubectl --context prod rollout restart deploy/orders",
    "psql postgres://orders@db.PROD.internal/orders",
    "terraform apply -auto-approve envs/prod",
])
def test_brokerage_denies_production(cmd):
    v = prod(cmd)
    assert v.decision == "deny" and v.blocked and "Production" in v.reason


def test_brokerage_asks_before_a_push_to_main():
    assert prod("git push origin main").decision == "ask"


@pytest.mark.parametrize("cmd", ["kubectl get pods -n trading-staging", "pytest tests/orders -q", "git push origin feature/x"])
def test_brokerage_lets_ordinary_work_through(cmd):
    v = prod(cmd)
    assert v.exit_code == 0 and v.decision is None


def test_every_hooks_config_in_the_fleet_is_valid():
    found = []
    for path in list(FLEET.glob("plugins/*/*/hooks/hooks.json")) + list(FLEET.glob("clients/*/.claude/settings.json")):
        found += hk.lint(json.loads(path.read_text(encoding="utf-8")), path.parent.parent)
    assert found == []


def test_lint_catches_a_missing_script_and_a_bad_event(tmp_path):
    cfg = {"hooks": {"PreToolUse": [{"matcher": "Bash(", "hooks": [{"type": "command", "command": "python3 ${CLAUDE_PLUGIN_ROOT}/hooks/gone.py"}]}],
                     "BeforeTool": []}}
    found = hk.lint(cfg, tmp_path)
    assert any("does not compile" in f for f in found)
    assert any("gone.py" in f for f in found)
    assert any("BeforeTool" in f for f in found)

pytest -v tests

collecting ... collected 51 items

tests/test_fanout.py::test_main_harness_always_looks_fine PASSED         [  1%]
tests/test_fanout.py::test_auto_update_client_receives_it PASSED         [  3%]
tests/test_fanout.py::test_auto_update_off_waits_for_a_manual_update PASSED [  5%]
tests/test_fanout.py::test_stable_channel_is_behind PASSED               [  7%]
tests/test_fanout.py::test_change_arrives_but_a_client_deny_rule_blocks_it PASSED [  9%]
tests/test_fanout.py::test_project_skill_with_the_same_name_is_reported PASSED [ 11%]
tests/test_fanout.py::test_gate_fails_until_every_client_gets_it PASSED  [ 13%]
tests/test_fanout.py::test_change_without_a_version_bump_never_reaches_a_cached_install PASSED [ 15%]
tests/test_fanout.py::test_version_comes_from_plugin_json_first PASSED   [ 17%]
tests/test_fanout.py::test_content_hash_ignores_line_endings PASSED      [ 19%]
tests/test_hooks.py::test_matcher_rules PASSED                           [ 21%]
tests/test_hooks.py::test_guard_only_runs_on_file_writes PASSED          [ 23%]
tests/test_hooks.py::test_write_with_a_key_is_blocked[stable] PASSED     [ 25%]
tests/test_hooks.py::test_write_with_a_key_is_blocked[latest] PASSED     [ 27%]
tests/test_hooks.py::test_clean_write_passes[stable] PASSED              [ 29%]
tests/test_hooks.py::test_clean_write_passes[latest] PASSED              [ 31%]
tests/test_hooks.py::test_stable_guard_fails_open_on_edit PASSED         [ 33%]
tests/test_hooks.py::test_latest_guard_blocks_the_same_edit PASSED       [ 35%]
tests/test_hooks.py::test_latest_guard_fails_closed_on_input_it_cannot_read PASSED [ 37%]
tests/test_hooks.py::test_brokerage_denies_production[kubectl get pods -n trading-prod] PASSED [ 39%]
tests/test_hooks.py::test_brokerage_denies_production[kubectl --context prod rollout restart deploy/orders] PASSED [ 41%]
tests/test_hooks.py::test_brokerage_denies_production[psql postgres://orders@db.PROD.internal/orders] PASSED [ 43%]
tests/test_hooks.py::test_brokerage_denies_production[terraform apply -auto-approve envs/prod] PASSED [ 45%]
tests/test_hooks.py::test_brokerage_asks_before_a_push_to_main PASSED    [ 47%]
tests/test_hooks.py::test_brokerage_lets_ordinary_work_through[kubectl get pods -n trading-staging] PASSED [ 49%]
tests/test_hooks.py::test_brokerage_lets_ordinary_work_through[pytest tests/orders -q] PASSED [ 50%]
tests/test_hooks.py::test_brokerage_lets_ordinary_work_through[git push origin feature/x] PASSED [ 52%]
tests/test_hooks.py::test_every_hooks_config_in_the_fleet_is_valid PASSED [ 54%]
tests/test_hooks.py::test_lint_catches_a_missing_script_and_a_bad_event PASSED [ 56%]
tests/test_settings.py::test_higher_scope_wins_for_a_scalar PASSED       [ 58%]
tests/test_settings.py::test_lists_merge_across_scopes PASSED            [ 60%]
tests/test_settings.py::test_deny_beats_a_narrower_allow_from_a_higher_scope PASSED [ 62%]
tests/test_settings.py::test_ask_beats_allow PASSED                      [ 64%]
tests/test_settings.py::test_prefix_needs_the_space PASSED               [ 66%]
tests/test_settings.py::test_no_rule_leaves_it_to_the_mode PASSED        [ 68%]
tests/test_usage.py::test_reads_every_session PASSED                     [ 70%]
tests/test_usage.py::test_brokerage_profile_shows_who_does_what PASSED   [ 72%]
tests/test_usage.py::test_test_plan_leads_with_real_workflows_and_friction PASSED [ 74%]
tests/test_usage.py::test_lender_has_machines_where_the_plugin_never_installed PASSED [ 76%]
tests/test_usage.py::test_trajectory_pr_check_reads_the_diff_and_never_pushes PASSED [ 78%]
tests/test_usage.py::test_trajectory_catches_a_denied_production_call PASSED [ 80%]
tests/test_usage.py::test_trajectory_catches_a_missing_plugin PASSED     [ 82%]
tests/test_web_port.py::test_fanout_matches_python PASSED                [ 84%]
tests/test_web_port.py::test_hooks_match_python[guard-secrets-1.7.0-core-harness/stable-Write-inp0] PASSED [ 86%]
tests/test_web_port.py::test_hooks_match_python[guard-secrets-1.7.0-core-harness/stable-Write-inp1] PASSED [ 88%]
tests/test_web_port.py::test_hooks_match_python[guard-secrets-1.7.0-core-harness/stable-Edit-inp2] PASSED [ 90%]
tests/test_web_port.py::test_hooks_match_python[guard-secrets-1.8.0-core-harness/latest-Edit-inp3] PASSED [ 92%]
tests/test_web_port.py::test_hooks_match_python[guard-secrets-1.8.0-core-harness/latest-Write-inp4] PASSED [ 94%]
tests/test_web_port.py::test_hooks_match_python[protect-prod-None-Bash-inp5] PASSED [ 96%]
tests/test_web_port.py::test_hooks_match_python[protect-prod-None-Bash-inp6] PASSED [ 98%]
tests/test_web_port.py::test_hooks_match_python[protect-prod-None-Bash-inp7] PASSED [100%]

============================== 51 passed in 1.01s ==============================
exit 0

claude plugin validate, on both refs

$ claude plugin validate plugins/core-harness/stable
Validating plugin manifest: data/testlab/harness/fleet/plugins/core-harness/stable/.claude-plugin/plugin.json

Validating hooks: data/testlab/harness/fleet/plugins/core-harness/stable/hooks/hooks.json

‼ Found 1 warning:

  > hooks.PreToolUse: Shell command uses ${CLAUDE_PLUGIN_ROOT} without quotes: python3 ${CLAUDE_PLUGIN_ROOT}/hooks/guard_secrets.py. If the expanded path contains a space the command can split into several words and fail. Wrap the placeholder in double quotes, or use exec form: {"command": "<executable>", "args": ["${CLAUDE_PLUGIN_ROOT}/..."]}.

√ Validation passed with warnings

$ claude plugin validate plugins/core-harness/latest
Validating plugin manifest: data/testlab/harness/fleet/plugins/core-harness/latest/.claude-plugin/plugin.json

√ Validation passed

$ claude --version
2.1.312 (Claude Code)

The validator, run for real, caught a second problem in the stable hook: an unquoted ${CLAUDE_PLUGIN_ROOT} that splits on a path with a space. Windows home folders have them.

Claude Code: hooks reference (events, exit codes, matchers, decisions) · Claude Code: create a marketplace (claude plugin validate). The brokerage client's own hook, which denies anything aimed at production trading systems and asks before a push to main, is tested the same way in the same file.

Test it their way

Learn how each client uses its harness, from its sessions

Every Claude Code run can write a stream-json record: the plugins and MCP servers that loaded, each tool call, and a result with the permission denials. Read a client's sessions by role and the test plan writes its first draft: what developers, testers and product owners do most, where the rules stop them, and what failed to load. These cards come from the sample fleet's 19 sessions, read by harnessqa profile when this page was built.

brokerage

8 sessions · calls by role: developer 15 · product 8 · tester 8 · 2 denied · 0 failed

  1. developer: Edit (3 calls)
  2. developer: /fin-guardrails:order-rules (2 calls)
  3. developer: Bash: pytest tests/orders (2 calls)
  4. product: /release-notes (2 calls)
  5. product: Bash: gh pr (1 call)
  6. product: /core-harness:test-plan (1 call)
  7. tester: Bash: npx playwright (3 calls)

insurer

2 sessions · calls by role: developer 2 · tester 2 · 0 denied · 0 failed

  1. developer: /core-harness:pr-check (1 call)
  2. developer: Bash: git diff (1 call)
  3. tester: /core-harness:test-plan (1 call)
  4. tester: Bash: npx playwright (1 call)

lender

4 sessions · calls by role: developer 8 · product 3 · tester 3 · 0 denied · 0 failed

  1. developer: /core-harness:pr-check (2 calls)
  2. developer: Edit (2 calls)
  3. developer: Bash: npm test (2 calls)
  4. product: /core-harness:test-plan (1 call)
  5. product: Bash: gh pr (1 call)
  6. tester: /core-harness:test-plan (1 call)
  7. tester: Bash: npm test (1 call)

retailer

5 sessions · calls by role: developer 3 · product 7 · tester 3 · 0 denied · 1 failed

  1. developer: /core-harness:pr-check (1 call)
  2. developer: Bash: git diff (1 call)
  3. developer: Bash: npm run (1 call)
  4. product: /core-harness:test-plan (2 calls)
  5. product: Bash: gh pr (2 calls)
  6. product: Bash: git log (1 call)
  7. tester: Bash: npm run (2 calls)

A trajectory gate on one session

$ python -m harnessqa check fleet/usage/brokerage/tester/bro-tes-02.jsonl \
    --plugin core-harness --plugin fin-guardrails --never "Bash: kubectl" --no-denials
FAIL forbidden call Bash: kubectl get
FAIL 1 permission denials
fleet/usage/brokerage/tester/bro-tes-02.jsonl: 4 tool calls, 5 turns, 2 problems
exit 1

A tester asked about pods in the production namespace; the brokerage's production guard denied it. The check reports both the forbidden call and the denial, so a person decides whether the guard or the workflow is wrong.

The usage tests

tests/test_usage.py
"""Learn how each client uses the harness from its sessions, then test the trajectory."""
from conftest import FLEET

from harnessqa import transcripts as tr

SESSIONS = tr.load(FLEET / "usage")


def test_reads_every_session():
    assert len(SESSIONS) == 19
    assert {s.client for s in SESSIONS} == {"brokerage", "lender", "retailer", "insurer"}


def test_brokerage_profile_shows_who_does_what():
    p = tr.profile(SESSIONS, "brokerage")
    assert p.skills["fin-guardrails:order-rules"] == 2
    assert p.skills["release-notes"] == 2  # product owners use the client's own skill
    assert sum(p.denials.values()) == 2


def test_test_plan_leads_with_real_workflows_and_friction():
    plan = tr.test_plan(tr.profile(SESSIONS, "brokerage"))
    assert "product: /release-notes (2 calls)" in plan
    assert any(line.startswith("friction: Bash: gh pr") for line in plan)


def test_lender_has_machines_where_the_plugin_never_installed():
    p = tr.profile(SESSIONS, "lender")
    assert any("enabled in project settings but isn't installed" in e for e in p.plugin_errors)


def test_trajectory_pr_check_reads_the_diff_and_never_pushes():
    s = tr.read(FLEET / "usage/brokerage/developer/bro-dev-02.jsonl")
    ex = tr.Expect(must_call=["/core-harness:pr-check"], in_order=["/core-harness:pr-check", "Bash: git diff"],
                   never=["Bash: git push", "Bash: kubectl"], max_turns=8, no_denials=True,
                   plugins_loaded=["core-harness", "fin-guardrails"])
    assert tr.check(s, ex) == []


def test_trajectory_catches_a_denied_production_call():
    s = tr.read(FLEET / "usage/brokerage/tester/bro-tes-02.jsonl")
    problems = tr.check(s, tr.Expect(never=["Bash: kubectl"], no_denials=True))
    assert any("forbidden call Bash: kubectl" in p for p in problems)
    assert any("permission denials" in p for p in problems)


def test_trajectory_catches_a_missing_plugin():
    s = tr.read(FLEET / "usage/lender/tester/len-tes-01.jsonl")
    assert any("fin-guardrails did not load" in p for p in tr.check(s, tr.Expect(plugins_loaded=["fin-guardrails"])))

Claude Code: run programmatically (claude -p, stream-json, --bare). The sample sessions are written in the stream-json shape the page documents; the clients, people and calls in them are invented.

Behavior evals

Does the skill fire, do the right thing, and do better than no harness at all?

claude plugin eval runs each case three times with the plugin and three times without it, grades every run, and reports both scores and the difference. A skill that scores 1.0 both ways is not what made the case pass. These three cases ship with core-harness 1.8.0: the release notes skill on a natural request, the same skill staying quiet on an unrelated question, and the PR check finding a float in money code without editing a file.

release-notes-from-prs

evals/release-notes-from-prs/prompt.md
---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill, Bash]
tags: [release-notes, product]
---

What shipped since the last release? I need notes the product owners can read.
graders/criteria.md
---
type: llm
weight: 2
---

PASS if the reply groups changes by what a user notices (new, changed, fixed) in plain sentences, or, when it could not reach the pull requests, says so and asks for access or a change log.
FAIL if the reply lists pull requests, numbers or features it did not read from a tool result, or is written for developers (file names, refactors) instead of product owners.
graders/skill-fired.md
---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?release-notes"'
---

pr-check-reads-and-never-edits

evals/pr-check-reads-and-never-edits/fixture.sh
#!/usr/bin/env bash
# A small repository with a rule in CLAUDE.md and a branch that breaks it.
set -euo pipefail
git init -q -b main .
git config user.email qa@example.com && git config user.name QA
printf '# Rules\n\nMoney is integer cents. Never use float for amounts.\n' > CLAUDE.md
printf 'def fee(cents):\n    return cents // 100\n' > fees.py
git add . && git commit -qm "start"
git checkout -qb feature/fees
printf 'def fee(amount):\n    return float(amount) * 0.01\n' > fees.py
git commit -qam "fees in dollars"
graders/no-edits.md
---
type: tool_used
tool: Edit
min: 0
max: 0
arm: both
---
graders/finds-the-float.md
---
type: llm
weight: 2
---

PASS if the reply flags that fees.py now uses float for money, against the CLAUDE.md rule, and names the line or function.
FAIL if it misses the float, or edits any file.

Not run on this page: every eval run and every judge is a model call on someone's account. The gate below runs them on release branches with a cost ceiling. The case files were made with claude plugin eval init --bare and pass claude plugin validate.

Claude Code: test plugins with evals (claude plugin eval)

The release gate

Free checks on every pull request; the model only where it earns its cost

ci/harness-gate.yml
# The release gate for a harness repository: copy to .github/workflows/harness-gate.yml.
# Static checks and contract tests run on every pull request and cost nothing; the evals and the
# smoke run call the model, so they run on release branches only, with a cost ceiling.
name: harness-gate
on:
  pull_request:
  push:
    branches: [latest, stable]

jobs:
  static:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: npm install -g @anthropic-ai/claude-code
      - run: pip install pyyaml pytest
      - name: Plugin manifests, hooks and skills are valid
        run: for d in plugins/*/; do claude plugin validate "$d"; done
      - name: Hook contract tests, settings rules, usage trajectories
        run: pytest -q tests
      - name: Does the release reach every client harness?
        run: python -m harnessqa fanout fleet/fleet.yaml

  evals:
    if: github.event_name == 'push'
    needs: static
    runs-on: ubuntu-latest
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      CLAUDE_CODE_SYNC_PLUGIN_INSTALL: "1"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: npm install -g @anthropic-ai/claude-code
      - run: pip install pyyaml
      - name: Behavior evals, with and without the plugin
        run: claude plugin eval plugins/core-harness --trust-plugin --threshold 0.8 --max-cost-usd 5 --no-publish
      - name: Smoke run as a client would (bare, plugin loaded, stream-json)
        run: |
          claude --bare -p "/core-harness:test-plan for: a user cancels an open limit order" \
            --plugin-dir plugins/core-harness --output-format stream-json --verbose \
            --permission-mode dontAsk --max-budget-usd 1 > smoke.jsonl
          python -m harnessqa check smoke.jsonl --plugin core-harness --must-call /core-harness:test-plan --never "Bash: git push" --max-turns 6
StageWhenCatches
claude plugin validateEvery pull requestManifests, hooks config, unquoted placeholders, marketplace entries
Contract testsEvery pull requestHooks that fail open, matchers that miss, rules that merge the wrong way
Fan-outEvery pull requestNo version bump, wrong channel, auto-update off, a client deny rule, a shadowing skill
Evals with a baselineRelease branchesA skill that no longer fires, or no longer helps
Smoke run, bareRelease branchesA plugin that does not load, a call that must never happen

Then, after the release: the same fan-out check against what each client's machines report, and the next week's sessions read again. A change is done when every client that should have it has it and uses it.

Claude Code: run programmatically (claude -p, stream-json, --bare) · Claude Code: test plugins with evals (claude plugin eval)

Before a release

The good flows and the bad flows, by the person who will meet them

Developer

  • The PR check runs on a natural request, finds the rule it should, and edits nothing.
  • A guard blocks the dangerous call with a reason a person can act on, not a stack trace.
  • A blocked call says what to do instead.
  • Bad flow: the hook times out on a big diff. Does the action wait, block, or slip through?

Tester

  • The test plan skill names the bad flows, not only the happy path.
  • Commands the tester runs every day are allowed without a prompt.
  • Bad flow: the suite times out mid-run. Does the session say so, or report success?
  • Bad flow: a production namespace typed by mistake. Denied, with the reason.

Product owner

  • Release notes in the words a customer uses, and never a change that did not ship.
  • The skill fires when they ask in their own words; they should not need its name.
  • Bad flow: the tool the skill needs is denied at this client. Do they get a clear message or silence?
  • Bad flow: two skills with one name. Which one answered?

The first week on a greenfield team

  • Write the fleet inventory: every client, its channel, its managed policy, what it has installed.
  • Run the fan-out check on the last three releases and count the clients each one missed.
  • Read a week of sessions per client and list the top five workflows by role.
  • Put validate, the contract tests and fan-out in the pipeline. Add evals for the five workflows.

Sources

Read on these pages, 2026-10-28

The tool, the tests and the sample fleet are in data/testlab/harness; the runs are in data/testlab/transcripts/harness-*.txt (the validator run with its paths shortened to the repository root). The agency, its clients and its plugins are invented; the Claude Code behavior is as documented on the pages above, and the validator output is from Claude Code 2.1.312.