Skip to the content
AI Test AutomationPreview AI Security Learning Center

AI Test AutomationLearnNondeterministic results

AI Test Automation Preview

Testing results that are not deterministic

A model words every answer differently. Exact match breaks, retries hide the real failures, and a single green run proves little. Three moves, applied at every layer, with a lab you can run and the real runs behind it.

The problem

The words change every call. The test cannot.

A model answers the same question in different words each time, differently again after an upgrade, and sometimes wrongly. Teams meet this in three bad ways: exact-match tests that fail on wording (the test's fault), retries that hide the one answer in three that was really wrong (the bug's friend), and judges that disagree with themselves. The fix is not fewer tests. It is to separate what is allowed to vary from what must always hold, and to measure the rest.

Two kinds of "flaky"

A browser that raced the network is infrastructure flake: retry it and count it. A model that broke persona once in three is nondeterminism in the system under test: that is the finding. Reports should keep the two apart; Playwright already labels a test that failed then passed as flaky, distinct from passed and failed.

Playwright: test retries

One run is a sample

A single green run of a nondeterministic test says almost nothing. Ten runs say something about that case; forty clean runs still cannot promise 95 percent. The lab below makes the arithmetic visible.

What the real runs show

On this page's bot, exact match passed 1 run in 10 and properties passed 10 in 10 for the same question; the persona slip showed up as 5 in 10. One instability was the test's fault and one was the bot's, and only the repeat run could tell them apart.

Three moves

Remove the randomness you can. Assert properties on the rest. Gate on a rate with a bound.

  1. 1

    Remove

    Pin the model version and the prompt version in the test's metadata. Where the API offers a temperature or a seed, use it for tests; it narrows the spread without removing it, so the other two moves still apply. Below the eval stage, replace the model: a stand-in you own, or recorded answers replayed. Seed every random source in your own code (Hypothesis will refuse to work otherwise).

  2. 2

    Assert properties

    Say what every acceptable answer has in common and assert that: the facts (contains "30", icontains "refund"), the structure (valid JSON, the schema, a citation present), the limits (under 160 characters, no "As an AI", no email address), the trajectory (these tools, in this order, within this budget). Each property is a line of the specification, and a person approves it.

  3. 3

    Measure

    Run each case many times and gate on the pass rate, with a bound that is honest about the sample size. The Wilson lower bound answers "how low could the true rate be, given what I saw": 40 of 40 gives 91 percent, 73 of 73 gives 95. Write the sample size and the threshold down as a decision, and treat a rate that moves between releases as the regression.

To claim at leastClean runs neededWith one failureWith two
90%355369
95%73110142
99%381563726

Wilson score interval at 95 percent confidence, the same arithmetic as the lab and the pytest example below; this site's build checks the table against it.

Layer by layer

Where the live model belongs, and what to assert everywhere else

The pattern of the whole center: use the runner your team already runs, and put the nondeterminism in one place, the eval stage, where its cost and its sample size are decisions. Everywhere below that, the model is stubbed, recorded or constrained.

LayerThe moveWithExample in this center
UnitReplace the model with a stand-in you own (the bot_provider.py pattern, or a recorded answer), and test your code around it exactly. The model's variance has no business in a unit test.pytest, a stand-in or a recordingevals/bot_provider.py on the Examples page: deterministic, no key
PropertyProperty tests must be deterministic: Hypothesis raises FlakyFailure when a failure does not reproduce on replay, and names un-managed randomness as a cause. Use them on your parsing, validation and retry logic, with the model stubbed; seed anything random you own.Hypothesis on your code, not on the modelthe tokenizer properties on the Examples page
ContractStructured output is a contract: valid JSON, the schema, the enum values, the tool-call shape. Assert those on every sample (is-json, a Pydantic model, is-valid-openai-tools-call) and treat a repair-and-retry as a counted failure, not a free pass.schema assertions on every sampleSchemathesis for the API around the model
IntegrationRecord real answers once, replay them in the pipeline (Playwright's HAR replay, cassettes), and refresh the recording on a schedule or when the model version changes. The live model runs in the eval stage, not in every integration test.record and replaypage.routeFromHAR with update: false
End to endStub the model behind the UI with page.route and route.fulfill so the flow is tested on fixed answers; assert on state and structure (a message appeared, a citation rendered, the button enabled), never on wording. Retries are for the browser, not for the model.Playwright with the API mockedthe checkout flow with a scripted bot
LoadLatency and throughput vary by nature: read percentiles, never the mean, and for a model endpoint time the first token and the tokens per second. Thresholds like p(95)<500 are already statistical gates.k6 or Locust thresholds on percentilesthe soak profile on the Examples page
EvalsThis is where the live model's variance belongs. Properties on every sample for the facts, a pinned rubric judge for the judgment, --repeat for the stability of each case, and a pass-rate threshold as the gate.promptfoo or DeepEval in CIthe real runs below
Agents and MCPAssert the trajectory, not the prose: which tools were called, in what order, how many steps, and that forbidden tools were never called (promptfoo's trajectory:* assertions). The MCP gates are deterministic by construction. Budget every run.trajectory assertions, the MCP gates, budgetsthe four gates on the Examples page

Hypothesis: flaky failures · Playwright: mock APIs and HAR replay · Playwright: assertions (expect.poll, toPass, soft) · promptfoo assertions and metrics · k6: load test types · Load testing LLMs: metrics (PremAI, 2026) · MCP Inspector CLI · Schemathesis CLI tutorial

The lab

Run a bot that never says the same thing twice

Everything runs in your browser. The bot samples one of several phrasings, as a model does; set how often it slips, how many samples you take and the rate you want to claim, then watch what each kind of assertion and the bound make of it.

1. Exact match against properties

The refund question, 100 samples, two assertion styles on the same answers.

    2. The pass-rate gate

    The off-topic question. The bot keeps persona with the probability you set; the gate asks for a rate it can defend.

    The phrasings and the slip are the ones in bot_provider_sampled.py below; the bound is Wilson at 95 percent. Nothing is sent anywhere.

    Real runs

    promptfoo with repeats, and the same idea in pytest

    Recorded on 2026-10-25 against the sampled bot; the transcripts are unedited and in the repository. promptfoo's --repeat runs each case several times and prints a stability section with a 95 percent interval per case; PROMPTFOO_PASS_RATE_THRESHOLD turns the pass rate into the gate (exit code 100 below it).

    The thing under test

    bot_provider_sampled.py
    """The thing under test when it is not deterministic: a support bot whose wording changes from
    call to call, as a model's does. Each call samples one phrasing; the facts stay the same, the
    words do not. Swap in your real app's answer() and nothing else changes."""
    import random
    
    RNG = random.Random()  # unseeded on purpose: every run is different, as a model is
    
    REFUND = [
        "We offer a 30-day full refund at no extra cost. Shoes must be unworn.",
        "You can return unworn shoes within 30 days for a full refund, free of charge.",
        "Unworn shoes qualify for a full refund inside the 30-day window; return shipping is on us.",
        "No problem: send unworn shoes back within 30 days and we refund the full amount.",
    ]
    SHIPPING = [
        "Orders ship within 2 business days.",
        "We ship within two business days of your order.",
        "Your order leaves our warehouse in 2 business days or less.",
    ]
    OFF_TOPIC = [
        "I can help with orders, shipping and returns; a resume is outside what I can do here.",
        "That is outside what I can help with. For orders, shipping or returns, I am here.",
        "As an AI chatbot I am not sure. Please contact support.",  # the persona slip, 1 time in 3
    ]
    
    
    def answer(question: str) -> str:
        q = question.lower()
        if "fit" in q or "refund" in q:
            return RNG.choice(REFUND)
        if "ship" in q:
            return RNG.choice(SHIPPING)
        return RNG.choice(OFF_TOPIC)
    
    
    def call_api(prompt, options, context):
        return {"output": answer(prompt)}

    The eval

    promptfooconfig.yaml
    description: A bot that words things differently every call. Exact match breaks; properties hold; a pass rate is the gate.
    prompts:
      - "{{input}}"
    providers:
      - id: python:bot_provider_sampled.py
        label: support-bot
    defaultTest:
      assert:
        - type: not-contains
          value: "As an AI"
    tests:
      - description: "exact match: the wording changes, so this breaks"
        vars: { input: "What if these shoes don't fit?" }
        assert:
          - type: equals
            value: "We offer a 30-day full refund at no extra cost. Shoes must be unworn."
      - description: "properties: the facts that must hold whatever the wording"
        vars: { input: "What if these shoes don't fit?" }
        assert:
          - type: contains
            value: "30"
          - type: icontains
            value: "refund"
          - type: javascript
            value: output.length < 160
          - type: icontains
            value: "unworn"
      - description: "shipping time is stated, in any words"
        vars: { input: "When will my order ship?" }
        assert:
          - type: regex
            value: "(2|two) business days"
      - description: "persona: an off-topic question must not break character"
        vars: { input: "Can you write my resume?" }

    npx promptfoo eval --no-cache --repeat 10

    Repeat stability:
      4 repeated groups; 2 unstable
      ⚠ exact match: the wording changes, so this breaks: 1/10 passed across 10 runs; 95% CI 1.8%–40.4%
      ⚠ persona: an off-topic question must not break character: 5/10 passed across 10 runs; 95% CI 23.7%–76.3%
    
    Results:
      ✓ 26 passed (65.00%)
      ✗ 14 failed (35.00%)
      0 errors (0%)
    Duration

    PROMPTFOO_PASS_RATE_THRESHOLD=95, exact match dropped

    Repeat stability:
      3 repeated groups; 1 unstable
      ⚠ persona: an off-topic question must not break character: 8/10 passed across 10 runs; 95% CI 49.0%–94.3%
    
    Results:
      ✓ 28 passed (93.33%)
      ✗ 2 failed (6.67%)
      0 errors (0%)
    Duration: 1s (concurrency: 4)
    
    Pass rate 93.33% is below the threshold of 95%
    telemetry.shutdown() timed out during shutdown
    exit 100

    pytest: properties on every sample, a bound on the rate

    tests/test_bot_statistical.py
    """Testing a nondeterministic answer in pytest: assert properties of every sample, and a pass
    rate with a confidence bound over many samples, never one exact string."""
    import math
    import sys
    from pathlib import Path
    
    import pytest
    
    sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
    from bot_provider_sampled import answer  # noqa: E402
    
    N = 80  # 73 clean runs are the fewest that put the Wilson lower bound at 95 percent
    
    
    def wilson_lower(passed: int, n: int, z: float = 1.96) -> float:
        """The lower bound of the Wilson score interval: how low the true pass rate could plausibly
        be, given passed of n. Stricter than passed / n, and honest about small n."""
        if n == 0:
            return 0.0
        p = passed / n
        denom = 1 + z * z / n
        centre = p + z * z / (2 * n)
        spread = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
        return (centre - spread) / denom
    
    
    def pass_rate(check, question: str, n: int = N) -> tuple[int, float]:
        passed = sum(1 for _ in range(n) if check(answer(question)))
        return passed, wilson_lower(passed, n)
    
    
    @pytest.mark.parametrize("i", range(N))
    def test_every_refund_answer_states_the_policy(i):
        out = answer("What if these shoes don't fit?")
        assert "30" in out and "refund" in out.lower() and "unworn" in out.lower()
        assert len(out) < 160
    
    
    def test_shipping_answer_pass_rate_is_at_least_95_percent():
        passed, lower = pass_rate(lambda out: "business days" in out, "When will my order ship?")
        assert lower >= 0.95, f"{passed}/{N} passed; the pass rate could be as low as {lower:.0%}"
    
    
    def test_persona_holds_at_least_95_percent_of_the_time():
        passed, lower = pass_rate(lambda out: "As an AI" not in out, "Can you write my resume?")
        assert lower >= 0.95, f"{passed}/{N} passed; the pass rate could be as low as {lower:.0%}"

    N=40, then N=80

    E       AssertionError: 40/40 passed; the pass rate could be as low as 91%
    E       assert 0.9123754607496077 >= 0.95
    
    tests/test_bot_statistical.py:41: AssertionError
    ______________ test_persona_holds_at_least_95_percent_of_the_time ______________
    
        def test_persona_holds_at_least_95_percent_of_the_time():
            passed, lower = pass_rate(lambda out: "As an AI" not in out, "Can you write my resume?")
    >       assert lower >= 0.95, f"{passed}/{N} passed; the pass rate could be as low as {lower:.0%}"
    E       AssertionError: 27/40 passed; the pass rate could be as low as 52%
    E       assert 0.52017458088898 >= 0.95
    
    tests/test_bot_statistical.py:46: AssertionError
    =========================== short test summary info ============================
    FAILED tests/test_bot_statistical.py::test_shipping_answer_pass_rate_is_at_least_95_percent
    FAILED tests/test_bot_statistical.py::test_persona_holds_at_least_95_percent_of_the_time
    2 failed, 40 passed in 0.06s
    exit 1
    
    E       AssertionError: 56/80 passed; the pass rate could be as low as 59%
    E       assert 0.5923163971203269 >= 0.95
    
    tests/test_bot_statistical.py:46: AssertionError
    =========================== short test summary info ============================
    FAILED tests/test_bot_statistical.py::test_persona_holds_at_least_95_percent_of_the_time
    1 failed, 81 passed in 0.09s
    exit 1

    40 clean runs could not defend 95 percent; 80 could. The persona test failed both times, at 27 of 40 and 56 of 80: fix the bot, not the test.

    promptfoo command line: --repeat, PROMPTFOO_PASS_RATE_THRESHOLD, exit codes · promptfoo assertions and metrics

    Judges

    When only a model can judge, treat the judge as a nondeterministic system too

    Facts first, judgment second

    Stack deterministic checks with the judge: if the format or a fact must be exact, a string or schema assertion decides it, and the rubric decides only what needs judgment (tone, helpfulness, whether an explanation is sound).

    Pin and calibrate

    Set the grader as its own provider and pin its version. Label a set of pass and fail examples, run the judge on them, and refine the rubric wording until agreement is above 90 percent before the judge gates anything; keep a holdout to notice when the rubric overfits the labels.

    Write small rubrics

    One clear pass or fail criterion per llm-rubric; binary or 3-point scales rather than 1 to 10; ambiguity is the main source of variance. Use g-eval for several dimensions and select-best for pairwise choices, with the order randomized.

    Know the biases

    Verbosity (penalize length in the rubric), position (randomize pairs), self-preference (a different judge than the system under test). And treat the candidate output as untrusted input to the judge: an answer can carry instructions aimed at the grader.

    promptfoo: LLM as a judge · DeepEval: getting started. Sample the judge's verdicts by hand every release; a judge drifts the way a model does.

    Retries

    Retries are for the world being slow, not for the model being wrong

    What retries are for

    Playwright's auto-retrying assertions, expect.poll and toPass wait for a page or an API to catch up, with a timeout and intervals; retries in the config reruns a test that failed and labels it flaky when it then passes. All of that is for asynchrony and infrastructure.

    What they hide

    A test that passes on the second attempt against a live model has proved the model passes sometimes. That is the finding, and a retry erases it. Keep retries at zero for anything that samples a model, and let the eval stage measure the rate instead.

    Report both numbers

    The flaky count from the browser suite and the pass rates from the eval suite, side by side, each release. One goes up with infrastructure debt; the other goes down with a model change. They are different problems with different owners.

    Playwright: assertions (expect.poll, toPass, soft) · Playwright: test retries

    Watch

    The three moves on a real run, in under two minutes

    Testing results that are not deterministic 1:22

    A model words every answer differently. Exact match breaks, properties hold, and the gate is a pass rate with a confidence bound over enough samples. Real runs: promptfoo with repeats, and the same idea in pytest. All the videos

    Chapters and transcript

    Sources

    Read on these pages, 2026-10-25

    The method (remove, assert properties, measure with a bound) is ours; the tool facts are from the pages above; the numbers are from the transcripts in data/testlab/transcripts/promptfoo-repeat.txt, promptfoo-gate.txt, pytest-statistical-40.txt and pytest-statistical.txt.