When a team builds Claude Code harnesses for many clients and changes them at AI speed, QA answers two questions for every release: did it reach every client harness, and does it work the way each client uses it. A tool, tests and a gate that answer both, built on how Claude Code really ships plugins.
The question
One change, many client harnesses. Did it reach every one of them?
A harness is what a team builds around Claude Code so developers, testers and product owners at a client all work the same way: the standards in CLAUDE.md, skills for the common jobs, hooks that guard what the agent may touch, permission rules, MCP servers, and the plugins that carry it all. Build one for each of many clients, change it at AI speed, and every change raises a question code review cannot answer: did it reach every client harness or only the main one, and does it work the way each client uses it?
The main harness always looks fine
The harness team adds its marketplace from a local folder, and Claude Code loads a plugin from there in place, at every session start, whatever its version says. Clients get a cached copy that changes only when the version does. A green check on the main harness proves the change works in one place.
The developers who deploy a harness test the path they built. QA's job is the other paths: the client on the stable channel, the client whose deny rule blocks the new skill, the product owner who types the request a different way.
Backstage, with the developers
Most of this work is reading configuration and running checks before anything reaches a client. Everything on this page runs from a terminal or a pipeline, and the parts that cost nothing run on every pull request.
What a harness is
Seven parts, seven ways to fail, seven tests
Part
How it reaches a client
How it fails there
The test
CLAUDE.md
In the client's repository, beside the client's code
Drifts from the agency standard; contradicts a skill; grows until the rule that matters is lost
A diff against the standard each release; an eval whose prompt the rule should answer
Skills
A plugin (namespaced /plugin:skill) or the project's .claude/skills
Not chosen on natural phrasing; a project skill of the same name answers instead; its allowed-tools are denied by the client
An eval with a tool_used: Skill grader; the fan-out shadow and permission checks
Hooks
A plugin's hooks/hooks.json and every settings scope, merged: every match runs
A crash is exit 1, and exit 1 lets the action through; an unquoted path splits; the matcher misses a tool
Contract tests with the documented stdin; claude plugin validate; a test per exit path
Permission rules
Settings at each scope; lists merge; deny, then ask, then allow
A client deny blocks a new workflow; a local allow does not beat a project ask
Decide each new skill's tools against the client's merged rules
MCP servers
The project's .mcp.json, a plugin, or managed settings
Not loaded in a scripted run; a startup timeout; a renamed tool
mcp_servers and mcp_server_errors in the stream-json init event; mocked servers in evals
Plugins and marketplaces
A marketplace per channel; the version comes from plugin.json first
No version bump, so no update; auto-update off; enabled in project settings but never installed
The fan-out check below, on every release
Managed settings
MDM, a managed-settings.json file, or the admin console
Overrides every other scope; only one managed source applies by default
Resolve each client with its managed copy, as the tool does
The fan-out board: two releases, five client harnesses, one honest answer each
This board is the output of harnessqa fanout on a sample fleet, run when this page was built: an agency's two plugins on a stable and a latest channel, its own reference harness, and five clients, each with its own settings and managed policy. For each client it resolves the settings that client really runs, finds the version its channel serves and the version it has installed, and checks that the new skills can run under its permission rules.
Client harness
core-harness 1.8.0guard_secrets also checks Edit; new release-notes skill
fin-guardrails 1.2.0the money guard also flags round() on amounts
mainlocal directory
NEXT SESSIONloads the plugin's current files at every session start (or /reload-plugins), whatever the version says
NEXT SESSIONloads the plugin's current files at every session start (or /reload-plugins), whatever the version says
brokerageagency-latest
BLOCKEDauto-update is on for this marketplace (managed settings); skill release-notes needs Bash(gh pr list *); denied by Bash(gh *) in project settings; a project skill is also called release-notes: /release-notes runs the project's copy; the release is /core-harness:release-notes
NEVERinstalled 1.2.0 and the release is also 1.2.0: the cached copy is never replaced
lenderagency-latest
AFTER UPDATEauto-update is off: stays on 1.7.0 until someone runs claude plugin update core-harness@agency-latest
NEVERinstalled 1.2.0 and the release is also 1.2.0: the cached copy is never replaced
retaileragency-latest
REACHESauto-update is on for this marketplace (managed settings)
NOT INSTALLEDfin-guardrails is not in enabledPlugins
insureragency-stable
BEHINDagency-stable serves ref stable (1.7.0); the change is on latest
NOT INSTALLEDfin-guardrails is not in enabledPlugins
core-harness: gate FAILfin-guardrails: gate FAIL
fin-guardrails: files changed between stable and latest but the version is still 1.2.0; every cached install keeps the old copy. Bump "version" in plugin.json.
The delivery rules it checks
A cached install gets a new copy only when the computed version changes: plugin.json first, then the marketplace entry, else the commit.
Background auto-update is off until a user or an admin turns it on for that marketplace.
One marketplace serves one ref of each plugin, so a stable channel is a second marketplace.
A plugin skill loads as /plugin:skill beside a project skill of the same name.
Deny rules win over whatever a skill's allowed-tools pre-approves.
A plugin enabled in a repository's settings whose marketplace points at its own repository does not install from the settings alone: each person sees "enabled in project settings but isn't installed" until they install it.
Repository marketplaces apply only in folders the person trusted; a -p run never accepts trust.
In -p and CI runs, plugins install in the background and can miss the first turn; CLAUDE_CODE_SYNC_PLUGIN_INSTALL=1 makes the run wait.
release core-harness 1.8.0 (agency/core-harness#latest, was #stable): guard_secrets also checks Edit; new release-notes skill
client status channel serves installed
retailer REACHES agency-latest 1.8.0 1.7.0
why: auto-update is on for this marketplace (managed settings)
main NEXT SESSION local directory 1.8.0 in place
why: loads the plugin's current files at every session start (or /reload-plugins), whatever the version says
lender AFTER UPDATE agency-latest 1.8.0 1.7.0
why: auto-update is off: stays on 1.7.0 until someone runs claude plugin update core-harness@agency-latest
insurer BEHIND agency-stable 1.7.0 1.7.0
why: agency-stable serves ref stable (1.7.0); the change is on latest
brokerage BLOCKED agency-latest 1.8.0 1.7.0
why: auto-update is on for this marketplace (managed settings)
why: skill release-notes needs Bash(gh pr list *); denied by Bash(gh *) in project settings
note: a project skill is also called release-notes: /release-notes runs the project's copy; the release is /core-harness:release-notes
2 of 5 client harnesses that run core-harness get this release now; gate FAIL
release fin-guardrails 1.2.0 (agency/fin-guardrails#latest, was #stable): the money guard also flags round() on amounts
RELEASE fin-guardrails: files changed between stable and latest but the version is still 1.2.0; every cached install keeps the old copy. Bump "version" in plugin.json.
client status channel serves installed
main NEXT SESSION local directory 1.2.0 in place
why: loads the plugin's current files at every session start (or /reload-plugins), whatever the version says
brokerage NEVER agency-latest 1.2.0 1.2.0
why: installed 1.2.0 and the release is also 1.2.0: the cached copy is never replaced
lender NEVER agency-latest 1.2.0 1.2.0
why: installed 1.2.0 and the release is also 1.2.0: the cached copy is never replaced
insurer NOT INSTALLED
why: fin-guardrails is not in enabledPlugins
retailer NOT INSTALLED
why: fin-guardrails is not in enabledPlugins
1 of 3 client harnesses that run fin-guardrails get this release now; gate FAIL
exit 1
The inventory QA keeps
fleet/fleet.yaml
# The QA inventory for a fleet of client harnesses. A sample fleet: the agency, its plugins and
# its clients are invented. What each client runs is read from its own files under clients/;
# "installed" is what `claude plugin list` reported on that client's machines at the last check.
marketplaces:
agency-dev: marketplaces/agency-dev.json # a local directory, on the harness team's machines
agency-latest: marketplaces/agency-latest.json # github: agency/harness-latest
agency-stable: marketplaces/agency-stable.json # github: agency/harness-stable
refs: # a checkout of each git ref the marketplaces point at
agency/core-harness#stable: plugins/core-harness/stable
agency/core-harness#latest: plugins/core-harness/latest
agency/fin-guardrails#stable: plugins/fin-guardrails/stable
agency/fin-guardrails#latest: plugins/fin-guardrails/latest
clients:
main:
root: clients/main
install: in-place
sources:
core-harness: plugins/core-harness/latest
fin-guardrails: plugins/fin-guardrails/latest
brokerage:
root: clients/brokerage
installed: {core-harness: 1.7.0, fin-guardrails: 1.2.0}
lender:
root: clients/lender
installed: {core-harness: 1.7.0, fin-guardrails: 1.2.0}
retailer:
root: clients/retailer
installed: {core-harness: 1.7.0}
insurer:
root: clients/insurer
installed: {core-harness: 1.7.0}
releases:
- plugin: core-harness
repo: agency/core-harness
ref: latest
previous: stable
note: guard_secrets also checks Edit; new release-notes skill
- plugin: fin-guardrails
repo: agency/fin-guardrails
ref: latest
previous: stable
note: the money guard also flags round() on amounts
usage: usage
Hooks are code
A guard that crashes lets everything through
Claude Code reads a command hook by its exit code. Exit 2 blocks the action and shows stderr as the reason. Exit 0 lets it through, and may carry a JSON decision. Any other code, including the 1 a crashed Python script returns, is a non-blocking error: the action goes ahead. So the core harness's secrets guard, which read tool_input["content"], blocked a credential written with Write and waved the same credential through on an Edit, which sends new_string instead. The contract tests feed each hook the documented stdin for every tool its matcher covers, and found it.
"""PreToolUse guard: refuse to write a file that contains a credential. core-harness 1.7.0."""
import json
import re
import sys
PATTERNS = [
re.compile(r"AKIA[0-9A-Z]{16}"),
re.compile(r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----"),
re.compile(r"(?i)(api[_-]?key|secret)\s*[:=]\s*['\"][A-Za-z0-9_\-]{24,}['\"]"),
]
event = json.load(sys.stdin)
text = event["tool_input"]["content"]
for p in PATTERNS:
if p.search(text):
print(f"Blocked: {event['tool_input']['file_path']} would contain a credential ({p.pattern[:20]}...).", file=sys.stderr)
sys.exit(2)
sys.exit(0)
"""PreToolUse guard: refuse to write a file that contains a credential. core-harness 1.8.0.
Write sends "content", Edit sends "new_string"; anything the guard cannot read is refused
(exit 2) instead of crashing, because a crash is exit 1 and Claude Code carries on after it."""
import json
import re
import sys
PATTERNS = [
re.compile(r"AKIA[0-9A-Z]{16}"),
re.compile(r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----"),
re.compile(r"(?i)(api[_-]?key|secret)\s*[:=]\s*['\"][A-Za-z0-9_\-]{24,}['\"]"),
]
def texts(tool_input):
for key in ("content", "new_string"):
if isinstance(tool_input.get(key), str):
yield tool_input[key]
for edit in tool_input.get("edits", []) or []:
if isinstance(edit, dict) and isinstance(edit.get("new_string"), str):
yield edit["new_string"]
try:
event = json.load(sys.stdin)
tool_input = event["tool_input"]
path = tool_input.get("file_path", "?")
except (ValueError, KeyError, TypeError):
print("Blocked: the secrets guard could not read this tool call.", file=sys.stderr)
sys.exit(2)
for text in texts(tool_input):
for p in PATTERNS:
if p.search(text):
print(f"Blocked: {path} would contain a credential ({p.pattern[:20]}...).", file=sys.stderr)
sys.exit(2)
sys.exit(0)
The contract tests
tests/test_hooks.py
"""Hooks are code that runs on every tool call: test them like code, with the documented stdin and
the documented exit codes. The stable guard has a real bug these tests found: on an Edit it
crashes, a crash is exit 1, and Claude Code carries on after exit 1. It fails open."""
import json
import pytest
from conftest import FLEET, secret
from harnessqa import hooks as hk
CORE = FLEET / "plugins" / "core-harness"
BROKERAGE = FLEET / "clients" / "brokerage"
def guard(ref):
cfg = json.loads((CORE / ref / "hooks" / "hooks.json").read_text(encoding="utf-8"))
return lambda tool, inp: hk.run(hk.handlers(cfg, "PreToolUse", tool)[0], hk.payload("PreToolUse", tool, inp), plugin_root=CORE / ref)
def test_matcher_rules():
assert hk.matches("Write|Edit", "Edit") and not hk.matches("Write|Edit", "MultiEditX")
assert hk.matches("mcp__memory__.*", "mcp__memory__read")
assert not hk.matches("mcp__memory", "mcp__memory__read")
assert hk.matches(None, "Bash") and hk.matches("*", "Bash")
def test_guard_only_runs_on_file_writes():
cfg = json.loads((CORE / "latest" / "hooks" / "hooks.json").read_text(encoding="utf-8"))
assert hk.handlers(cfg, "PreToolUse", "Bash") == []
assert len(hk.handlers(cfg, "PreToolUse", "Edit")) == 1
@pytest.mark.parametrize("ref", ["stable", "latest"])
def test_write_with_a_key_is_blocked(ref):
v = guard(ref)("Write", {"file_path": "app/config.py", "content": f"KEY = '{secret()}'"})
assert v.exit_code == 2 and v.blocked and "credential" in v.reason
@pytest.mark.parametrize("ref", ["stable", "latest"])
def test_clean_write_passes(ref):
v = guard(ref)("Write", {"file_path": "app/config.py", "content": "TIMEOUT = 30"})
assert v.exit_code == 0 and not v.blocked
def test_stable_guard_fails_open_on_edit():
"""The bug, pinned: 1.7.0 reads tool_input["content"], which an Edit does not send."""
v = guard("stable")("Edit", {"file_path": "app/config.py", "old_string": "KEY = ''", "new_string": f"KEY = '{secret()}'"})
assert v.failed_open and not v.blocked
assert "KeyError" in v.stderr
def test_latest_guard_blocks_the_same_edit():
v = guard("latest")("Edit", {"file_path": "app/config.py", "old_string": "KEY = ''", "new_string": f"KEY = '{secret()}'"})
assert v.exit_code == 2 and v.blocked
def test_latest_guard_fails_closed_on_input_it_cannot_read():
cfg = json.loads((CORE / "latest" / "hooks" / "hooks.json").read_text(encoding="utf-8"))
v = hk.run(hk.handlers(cfg, "PreToolUse", "Write")[0], "not json", plugin_root=CORE / "latest")
assert v.exit_code == 2
def prod(cmd):
cfg = json.loads((BROKERAGE / ".claude" / "settings.json").read_text(encoding="utf-8"))
h = hk.handlers(cfg, "PreToolUse", "Bash")[0]
return hk.run(h, hk.payload("PreToolUse", "Bash", {"command": cmd}), project_dir=BROKERAGE)
@pytest.mark.parametrize("cmd", [
"kubectl get pods -n trading-prod",
"kubectl --context prod rollout restart deploy/orders",
"psql postgres://orders@db.PROD.internal/orders",
"terraform apply -auto-approve envs/prod",
])
def test_brokerage_denies_production(cmd):
v = prod(cmd)
assert v.decision == "deny" and v.blocked and "Production" in v.reason
def test_brokerage_asks_before_a_push_to_main():
assert prod("git push origin main").decision == "ask"
@pytest.mark.parametrize("cmd", ["kubectl get pods -n trading-staging", "pytest tests/orders -q", "git push origin feature/x"])
def test_brokerage_lets_ordinary_work_through(cmd):
v = prod(cmd)
assert v.exit_code == 0 and v.decision is None
def test_every_hooks_config_in_the_fleet_is_valid():
found = []
for path in list(FLEET.glob("plugins/*/*/hooks/hooks.json")) + list(FLEET.glob("clients/*/.claude/settings.json")):
found += hk.lint(json.loads(path.read_text(encoding="utf-8")), path.parent.parent)
assert found == []
def test_lint_catches_a_missing_script_and_a_bad_event(tmp_path):
cfg = {"hooks": {"PreToolUse": [{"matcher": "Bash(", "hooks": [{"type": "command", "command": "python3 ${CLAUDE_PLUGIN_ROOT}/hooks/gone.py"}]}],
"BeforeTool": []}}
found = hk.lint(cfg, tmp_path)
assert any("does not compile" in f for f in found)
assert any("gone.py" in f for f in found)
assert any("BeforeTool" in f for f in found)
$ claude plugin validate plugins/core-harness/stable
Validating plugin manifest: data/testlab/harness/fleet/plugins/core-harness/stable/.claude-plugin/plugin.json
Validating hooks: data/testlab/harness/fleet/plugins/core-harness/stable/hooks/hooks.json
‼ Found 1 warning:
> hooks.PreToolUse: Shell command uses ${CLAUDE_PLUGIN_ROOT} without quotes: python3 ${CLAUDE_PLUGIN_ROOT}/hooks/guard_secrets.py. If the expanded path contains a space the command can split into several words and fail. Wrap the placeholder in double quotes, or use exec form: {"command": "<executable>", "args": ["${CLAUDE_PLUGIN_ROOT}/..."]}.
√ Validation passed with warnings
$ claude plugin validate plugins/core-harness/latest
Validating plugin manifest: data/testlab/harness/fleet/plugins/core-harness/latest/.claude-plugin/plugin.json
√ Validation passed
$ claude --version
2.1.312 (Claude Code)
The validator, run for real, caught a second problem in the stable hook: an unquoted ${CLAUDE_PLUGIN_ROOT} that splits on a path with a space. Windows home folders have them.
Learn how each client uses its harness, from its sessions
Every Claude Code run can write a stream-json record: the plugins and MCP servers that loaded, each tool call, and a result with the permission denials. Read a client's sessions by role and the test plan writes its first draft: what developers, testers and product owners do most, where the rules stop them, and what failed to load. These cards come from the sample fleet's 19 sessions, read by harnessqa profile when this page was built.
A tester asked about pods in the production namespace; the brokerage's production guard denied it. The check reports both the forbidden call and the denial, so a person decides whether the guard or the workflow is wrong.
The usage tests
tests/test_usage.py
"""Learn how each client uses the harness from its sessions, then test the trajectory."""
from conftest import FLEET
from harnessqa import transcripts as tr
SESSIONS = tr.load(FLEET / "usage")
def test_reads_every_session():
assert len(SESSIONS) == 19
assert {s.client for s in SESSIONS} == {"brokerage", "lender", "retailer", "insurer"}
def test_brokerage_profile_shows_who_does_what():
p = tr.profile(SESSIONS, "brokerage")
assert p.skills["fin-guardrails:order-rules"] == 2
assert p.skills["release-notes"] == 2 # product owners use the client's own skill
assert sum(p.denials.values()) == 2
def test_test_plan_leads_with_real_workflows_and_friction():
plan = tr.test_plan(tr.profile(SESSIONS, "brokerage"))
assert "product: /release-notes (2 calls)" in plan
assert any(line.startswith("friction: Bash: gh pr") for line in plan)
def test_lender_has_machines_where_the_plugin_never_installed():
p = tr.profile(SESSIONS, "lender")
assert any("enabled in project settings but isn't installed" in e for e in p.plugin_errors)
def test_trajectory_pr_check_reads_the_diff_and_never_pushes():
s = tr.read(FLEET / "usage/brokerage/developer/bro-dev-02.jsonl")
ex = tr.Expect(must_call=["/core-harness:pr-check"], in_order=["/core-harness:pr-check", "Bash: git diff"],
never=["Bash: git push", "Bash: kubectl"], max_turns=8, no_denials=True,
plugins_loaded=["core-harness", "fin-guardrails"])
assert tr.check(s, ex) == []
def test_trajectory_catches_a_denied_production_call():
s = tr.read(FLEET / "usage/brokerage/tester/bro-tes-02.jsonl")
problems = tr.check(s, tr.Expect(never=["Bash: kubectl"], no_denials=True))
assert any("forbidden call Bash: kubectl" in p for p in problems)
assert any("permission denials" in p for p in problems)
def test_trajectory_catches_a_missing_plugin():
s = tr.read(FLEET / "usage/lender/tester/len-tes-01.jsonl")
assert any("fin-guardrails did not load" in p for p in tr.check(s, tr.Expect(plugins_loaded=["fin-guardrails"])))
Does the skill fire, do the right thing, and do better than no harness at all?
claude plugin eval runs each case three times with the plugin and three times without it, grades every run, and reports both scores and the difference. A skill that scores 1.0 both ways is not what made the case pass. These three cases ship with core-harness 1.8.0: the release notes skill on a natural request, the same skill staying quiet on an unrelated question, and the PR check finding a float in money code without editing a file.
release-notes-from-prs
evals/release-notes-from-prs/prompt.md
---
max_turns: 10
allowed_tools: [Read, Glob, Grep, Skill, Bash]
tags: [release-notes, product]
---
What shipped since the last release? I need notes the product owners can read.
graders/criteria.md
---
type: llm
weight: 2
---
PASS if the reply groups changes by what a user notices (new, changed, fixed) in plain sentences, or, when it could not reach the pull requests, says so and asks for access or a change log.
FAIL if the reply lists pull requests, numbers or features it did not read from a tool result, or is written for developers (file names, refactors) instead of product owners.
---
type: llm
weight: 2
---
PASS if the reply flags that fees.py now uses float for money, against the CLAUDE.md rule, and names the line or function.
FAIL if it misses the float, or edits any file.
Not run on this page: every eval run and every judge is a model call on someone's account. The gate below runs them on release branches with a cost ceiling. The case files were made with claude plugin eval init --bare and pass claude plugin validate.
Hooks that fail open, matchers that miss, rules that merge the wrong way
Fan-out
Every pull request
No version bump, wrong channel, auto-update off, a client deny rule, a shadowing skill
Evals with a baseline
Release branches
A skill that no longer fires, or no longer helps
Smoke run, bare
Release branches
A plugin that does not load, a call that must never happen
Then, after the release: the same fan-out check against what each client's machines report, and the next week's sessions read again. A change is done when every client that should have it has it and uses it.
The tool, the tests and the sample fleet are in data/testlab/harness; the runs are in data/testlab/transcripts/harness-*.txt (the validator run with its paths shortened to the repository root). The agency, its clients and its plugins are invented; the Claude Code behavior is as documented on the pages above, and the validator output is from Claude Code 2.1.312.