A model words every answer differently. Exact match breaks, retries hide the real failures, and a single green run proves little. Three moves, applied at every layer, with a lab you can run and the real runs behind it.
The problem
The words change every call. The test cannot.
A model answers the same question in different words each time, differently again after an upgrade, and sometimes wrongly. Teams meet this in three bad ways: exact-match tests that fail on wording (the test's fault), retries that hide the one answer in three that was really wrong (the bug's friend), and judges that disagree with themselves. The fix is not fewer tests. It is to separate what is allowed to vary from what must always hold, and to measure the rest.
Two kinds of "flaky"
A browser that raced the network is infrastructure flake: retry it and count it. A model that broke persona once in three is nondeterminism in the system under test: that is the finding. Reports should keep the two apart; Playwright already labels a test that failed then passed as flaky, distinct from passed and failed.
A single green run of a nondeterministic test says almost nothing. Ten runs say something about that case; forty clean runs still cannot promise 95 percent. The lab below makes the arithmetic visible.
What the real runs show
On this page's bot, exact match passed 1 run in 10 and properties passed 10 in 10 for the same question; the persona slip showed up as 5 in 10. One instability was the test's fault and one was the bot's, and only the repeat run could tell them apart.
Three moves
Remove the randomness you can. Assert properties on the rest. Gate on a rate with a bound.
1
Remove
Pin the model version and the prompt version in the test's metadata. Where the API offers a temperature or a seed, use it for tests; it narrows the spread without removing it, so the other two moves still apply. Below the eval stage, replace the model: a stand-in you own, or recorded answers replayed. Seed every random source in your own code (Hypothesis will refuse to work otherwise).
2
Assert properties
Say what every acceptable answer has in common and assert that: the facts (contains "30", icontains "refund"), the structure (valid JSON, the schema, a citation present), the limits (under 160 characters, no "As an AI", no email address), the trajectory (these tools, in this order, within this budget). Each property is a line of the specification, and a person approves it.
3
Measure
Run each case many times and gate on the pass rate, with a bound that is honest about the sample size. The Wilson lower bound answers "how low could the true rate be, given what I saw": 40 of 40 gives 91 percent, 73 of 73 gives 95. Write the sample size and the threshold down as a decision, and treat a rate that moves between releases as the regression.
To claim at least
Clean runs needed
With one failure
With two
90%
35
53
69
95%
73
110
142
99%
381
563
726
Wilson score interval at 95 percent confidence, the same arithmetic as the lab and the pytest example below; this site's build checks the table against it.
Layer by layer
Where the live model belongs, and what to assert everywhere else
The pattern of the whole center: use the runner your team already runs, and put the nondeterminism in one place, the eval stage, where its cost and its sample size are decisions. Everywhere below that, the model is stubbed, recorded or constrained.
Layer
The move
With
Example in this center
Unit
Replace the model with a stand-in you own (the bot_provider.py pattern, or a recorded answer), and test your code around it exactly. The model's variance has no business in a unit test.
pytest, a stand-in or a recording
evals/bot_provider.py on the Examples page: deterministic, no key
Property
Property tests must be deterministic: Hypothesis raises FlakyFailure when a failure does not reproduce on replay, and names un-managed randomness as a cause. Use them on your parsing, validation and retry logic, with the model stubbed; seed anything random you own.
Hypothesis on your code, not on the model
the tokenizer properties on the Examples page
Contract
Structured output is a contract: valid JSON, the schema, the enum values, the tool-call shape. Assert those on every sample (is-json, a Pydantic model, is-valid-openai-tools-call) and treat a repair-and-retry as a counted failure, not a free pass.
schema assertions on every sample
Schemathesis for the API around the model
Integration
Record real answers once, replay them in the pipeline (Playwright's HAR replay, cassettes), and refresh the recording on a schedule or when the model version changes. The live model runs in the eval stage, not in every integration test.
record and replay
page.routeFromHAR with update: false
End to end
Stub the model behind the UI with page.route and route.fulfill so the flow is tested on fixed answers; assert on state and structure (a message appeared, a citation rendered, the button enabled), never on wording. Retries are for the browser, not for the model.
Playwright with the API mocked
the checkout flow with a scripted bot
Load
Latency and throughput vary by nature: read percentiles, never the mean, and for a model endpoint time the first token and the tokens per second. Thresholds like p(95)<500 are already statistical gates.
k6 or Locust thresholds on percentiles
the soak profile on the Examples page
Evals
This is where the live model's variance belongs. Properties on every sample for the facts, a pinned rubric judge for the judgment, --repeat for the stability of each case, and a pass-rate threshold as the gate.
promptfoo or DeepEval in CI
the real runs below
Agents and MCP
Assert the trajectory, not the prose: which tools were called, in what order, how many steps, and that forbidden tools were never called (promptfoo's trajectory:* assertions). The MCP gates are deterministic by construction. Budget every run.
Everything runs in your browser. The bot samples one of several phrasings, as a model does; set how often it slips, how many samples you take and the rate you want to claim, then watch what each kind of assertion and the bound make of it.
1. Exact match against properties
The refund question, 100 samples, two assertion styles on the same answers.
2. The pass-rate gate
The off-topic question. The bot keeps persona with the probability you set; the gate asks for a rate it can defend.
The phrasings and the slip are the ones in bot_provider_sampled.py below; the bound is Wilson at 95 percent. Nothing is sent anywhere.
Real runs
promptfoo with repeats, and the same idea in pytest
Recorded on 2026-10-25 against the sampled bot; the transcripts are unedited and in the repository. promptfoo's --repeat runs each case several times and prints a stability section with a 95 percent interval per case; PROMPTFOO_PASS_RATE_THRESHOLD turns the pass rate into the gate (exit code 100 below it).
The thing under test
bot_provider_sampled.py
"""The thing under test when it is not deterministic: a support bot whose wording changes from
call to call, as a model's does. Each call samples one phrasing; the facts stay the same, the
words do not. Swap in your real app's answer() and nothing else changes."""
import random
RNG = random.Random() # unseeded on purpose: every run is different, as a model is
REFUND = [
"We offer a 30-day full refund at no extra cost. Shoes must be unworn.",
"You can return unworn shoes within 30 days for a full refund, free of charge.",
"Unworn shoes qualify for a full refund inside the 30-day window; return shipping is on us.",
"No problem: send unworn shoes back within 30 days and we refund the full amount.",
]
SHIPPING = [
"Orders ship within 2 business days.",
"We ship within two business days of your order.",
"Your order leaves our warehouse in 2 business days or less.",
]
OFF_TOPIC = [
"I can help with orders, shipping and returns; a resume is outside what I can do here.",
"That is outside what I can help with. For orders, shipping or returns, I am here.",
"As an AI chatbot I am not sure. Please contact support.", # the persona slip, 1 time in 3
]
def answer(question: str) -> str:
q = question.lower()
if "fit" in q or "refund" in q:
return RNG.choice(REFUND)
if "ship" in q:
return RNG.choice(SHIPPING)
return RNG.choice(OFF_TOPIC)
def call_api(prompt, options, context):
return {"output": answer(prompt)}
The eval
promptfooconfig.yaml
description: A bot that words things differently every call. Exact match breaks; properties hold; a pass rate is the gate.
prompts:
- "{{input}}"
providers:
- id: python:bot_provider_sampled.py
label: support-bot
defaultTest:
assert:
- type: not-contains
value: "As an AI"
tests:
- description: "exact match: the wording changes, so this breaks"
vars: { input: "What if these shoes don't fit?" }
assert:
- type: equals
value: "We offer a 30-day full refund at no extra cost. Shoes must be unworn."
- description: "properties: the facts that must hold whatever the wording"
vars: { input: "What if these shoes don't fit?" }
assert:
- type: contains
value: "30"
- type: icontains
value: "refund"
- type: javascript
value: output.length < 160
- type: icontains
value: "unworn"
- description: "shipping time is stated, in any words"
vars: { input: "When will my order ship?" }
assert:
- type: regex
value: "(2|two) business days"
- description: "persona: an off-topic question must not break character"
vars: { input: "Can you write my resume?" }
npx promptfoo eval --no-cache --repeat 10
Repeat stability:
4 repeated groups; 2 unstable
⚠ exact match: the wording changes, so this breaks: 1/10 passed across 10 runs; 95% CI 1.8%–40.4%
⚠ persona: an off-topic question must not break character: 5/10 passed across 10 runs; 95% CI 23.7%–76.3%
Results:
✓ 26 passed (65.00%)
✗ 14 failed (35.00%)
0 errors (0%)
Duration
PROMPTFOO_PASS_RATE_THRESHOLD=95, exact match dropped
Repeat stability:
3 repeated groups; 1 unstable
⚠ persona: an off-topic question must not break character: 8/10 passed across 10 runs; 95% CI 49.0%–94.3%
Results:
✓ 28 passed (93.33%)
✗ 2 failed (6.67%)
0 errors (0%)
Duration: 1s (concurrency: 4)
Pass rate 93.33% is below the threshold of 95%
telemetry.shutdown() timed out during shutdown
exit 100
pytest: properties on every sample, a bound on the rate
tests/test_bot_statistical.py
"""Testing a nondeterministic answer in pytest: assert properties of every sample, and a pass
rate with a confidence bound over many samples, never one exact string."""
import math
import sys
from pathlib import Path
import pytest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from bot_provider_sampled import answer # noqa: E402
N = 80 # 73 clean runs are the fewest that put the Wilson lower bound at 95 percent
def wilson_lower(passed: int, n: int, z: float = 1.96) -> float:
"""The lower bound of the Wilson score interval: how low the true pass rate could plausibly
be, given passed of n. Stricter than passed / n, and honest about small n."""
if n == 0:
return 0.0
p = passed / n
denom = 1 + z * z / n
centre = p + z * z / (2 * n)
spread = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
return (centre - spread) / denom
def pass_rate(check, question: str, n: int = N) -> tuple[int, float]:
passed = sum(1 for _ in range(n) if check(answer(question)))
return passed, wilson_lower(passed, n)
@pytest.mark.parametrize("i", range(N))
def test_every_refund_answer_states_the_policy(i):
out = answer("What if these shoes don't fit?")
assert "30" in out and "refund" in out.lower() and "unworn" in out.lower()
assert len(out) < 160
def test_shipping_answer_pass_rate_is_at_least_95_percent():
passed, lower = pass_rate(lambda out: "business days" in out, "When will my order ship?")
assert lower >= 0.95, f"{passed}/{N} passed; the pass rate could be as low as {lower:.0%}"
def test_persona_holds_at_least_95_percent_of_the_time():
passed, lower = pass_rate(lambda out: "As an AI" not in out, "Can you write my resume?")
assert lower >= 0.95, f"{passed}/{N} passed; the pass rate could be as low as {lower:.0%}"
N=40, then N=80
E AssertionError: 40/40 passed; the pass rate could be as low as 91%
E assert 0.9123754607496077 >= 0.95
tests/test_bot_statistical.py:41: AssertionError
______________ test_persona_holds_at_least_95_percent_of_the_time ______________
def test_persona_holds_at_least_95_percent_of_the_time():
passed, lower = pass_rate(lambda out: "As an AI" not in out, "Can you write my resume?")
> assert lower >= 0.95, f"{passed}/{N} passed; the pass rate could be as low as {lower:.0%}"
E AssertionError: 27/40 passed; the pass rate could be as low as 52%
E assert 0.52017458088898 >= 0.95
tests/test_bot_statistical.py:46: AssertionError
=========================== short test summary info ============================
FAILED tests/test_bot_statistical.py::test_shipping_answer_pass_rate_is_at_least_95_percent
FAILED tests/test_bot_statistical.py::test_persona_holds_at_least_95_percent_of_the_time
2 failed, 40 passed in 0.06s
exit 1
E AssertionError: 56/80 passed; the pass rate could be as low as 59%
E assert 0.5923163971203269 >= 0.95
tests/test_bot_statistical.py:46: AssertionError
=========================== short test summary info ============================
FAILED tests/test_bot_statistical.py::test_persona_holds_at_least_95_percent_of_the_time
1 failed, 81 passed in 0.09s
exit 1
40 clean runs could not defend 95 percent; 80 could. The persona test failed both times, at 27 of 40 and 56 of 80: fix the bot, not the test.
When only a model can judge, treat the judge as a nondeterministic system too
Facts first, judgment second
Stack deterministic checks with the judge: if the format or a fact must be exact, a string or schema assertion decides it, and the rubric decides only what needs judgment (tone, helpfulness, whether an explanation is sound).
Pin and calibrate
Set the grader as its own provider and pin its version. Label a set of pass and fail examples, run the judge on them, and refine the rubric wording until agreement is above 90 percent before the judge gates anything; keep a holdout to notice when the rubric overfits the labels.
Write small rubrics
One clear pass or fail criterion per llm-rubric; binary or 3-point scales rather than 1 to 10; ambiguity is the main source of variance. Use g-eval for several dimensions and select-best for pairwise choices, with the order randomized.
Know the biases
Verbosity (penalize length in the rubric), position (randomize pairs), self-preference (a different judge than the system under test). And treat the candidate output as untrusted input to the judge: an answer can carry instructions aimed at the grader.
Retries are for the world being slow, not for the model being wrong
What retries are for
Playwright's auto-retrying assertions, expect.poll and toPass wait for a page or an API to catch up, with a timeout and intervals; retries in the config reruns a test that failed and labels it flaky when it then passes. All of that is for asynchrony and infrastructure.
What they hide
A test that passes on the second attempt against a live model has proved the model passes sometimes. That is the finding, and a retry erases it. Keep retries at zero for anything that samples a model, and let the eval stage measure the rate instead.
Report both numbers
The flaky count from the browser suite and the pass rates from the eval suite, side by side, each release. One goes up with infrastructure debt; the other goes down with a model change. They are different problems with different owners.
The three moves on a real run, in under two minutes
Testing results that are not deterministic 1:22
A model words every answer differently. Exact match breaks, properties hold, and the gate is a pass rate with a confidence bound over enough samples. Real runs: promptfoo with repeats, and the same idea in pytest. All the videos
The method (remove, assert properties, measure with a bound) is ours; the tool facts are from the pages above; the numbers are from the transcripts in data/testlab/transcripts/promptfoo-repeat.txt, promptfoo-gate.txt, pytest-statistical-40.txt and pytest-statistical.txt.