Skip to the content
AI Test AutomationPreview AI Security Learning Center

AI Test AutomationAI, cloud and DevOpsShift left

Shift left Preview

Shift left at AI speed

AI writes a growing share of new code and finds vulnerabilities faster than ever, while the time people have to review, test and fix stays the same. Testing has to start earlier, decide what comes first on evidence, and keep running after release.

The speed

AI writes more code, faster, than teams can review by hand

AI tools are now in nearly every team, and they write much of the new code. The output grows faster than the time people have to read it, and a lot of it is almost right.

90%

of nearly 5,000 technology professionals use AI at work. DORA found AI adoption linked to higher delivery throughput but, still, to lower delivery stability.

Google Cloud DORA report, 2025

66%

of developers name AI solutions that are almost right, but not quite, as their biggest frustration; more distrust AI output (46%) than trust it (33%).

Stack Overflow Developer Survey, 2025

19% slower

In a randomized trial, 16 experienced open-source developers took 19% longer with early-2025 AI tools, while believing they had been 20% faster.

METR, July 2025

+91%

pull request review time on teams with high AI adoption, which also merged 98% more pull requests, across over 10,000 developers.

Faros AI, July 2025

+23%

pull requests merged per month on GitHub, 43.2 million on average, with nearly 1 billion commits pushed in 2025 (up 25.1%).

GitHub Octoverse, October 2025

The backlog

The findings pile up faster than they can be fixed

More code, and AI that hunts for bugs, mean more vulnerabilities published, more reports to sort and more fixes waiting. The public systems that used to help sort them are falling behind too.

67,000+

new CVEs published in 2026 by September 18. CISA says automated and AI-enabled technologies add pressure from discovery to response, straining triage.

CISA, September 2026

263%

rise in CVE submissions from 2020 to 2025. Since April 15, 2026 NIST's vulnerability database enriches priority CVEs first (those in CISA's KEV catalog, in software the federal government uses, and in critical software); the rest are listed but not scheduled for immediate enrichment.

NIST National Vulnerability Database, April 2026

26%

of vulnerabilities in CISA's Known Exploited catalog were fully fixed by organizations in 2025, with a median of 43 days, CISA wrote, citing Verizon's 2026 DBIR.

CISA, June 2026

75 of 530

high or critical bugs (Anthropic's estimate) that Anthropic disclosed to open-source maintainers after scans with Mythos Preview had been patched, early in a 90-day disclosure window; some maintainers asked Anthropic to slow its disclosures.

Anthropic, May 2026

Below 5%

of curl's bug bounty reports were real vulnerabilities from 2025, down from above 15%, amid AI-written reports; curl ended its bounty on January 31, 2026.

Daniel Stenberg, curl, January 2026

In the Learning Center this month

Prioritize

When everything is urgent, decide the order on evidence

When thousands of findings are "critical", severity alone cannot set the order. Public, free signals say what attackers use now and what they are likely to use next, and your own setup says what can actually be reached. Use them in this order, and test the same way: the riskiest first.

  1. Exploited now: CISA's Known Exploited Vulnerabilities catalog

    A vulnerability joins the catalog when it has a CVE ID, there is reliable evidence it has been actively exploited in the wild, and there is a clear remediation action, such as a vendor update. Fix these first. Since June 2026, CISA's BOD 26-04 sets federal deadlines from four questions: is the asset exposed, is the CVE in the catalog, can exploitation be automated, and how much control does it give. Only the highest-risk combinations must be fixed within three days.

    CISA: Known Exploited Vulnerabilities catalog · CISA: the criteria for the KEV catalog · CISA: BOD 26-04, prioritizing security updates based on risk

  2. Likely next: EPSS

    FIRST's Exploit Prediction Scoring System estimates, every day, the probability that each CVE will be exploited in the next 30 days. Pick a threshold and fix what is above it. FIRST warns not to multiply it by CVSS: EPSS is a calibrated probability, CVSS an ordinal ranking with no empirical calibration, so the product has no meaning.

    FIRST: using EPSS

  3. Reachable in your setup: exposure and blast radius

    The same flaw matters more on an exposed asset, or in an agent that can reach your data and send it out. CISA's SSVC decision tree sorts a long list into four decisions (Track, Track* for closer monitoring, Attend, Act) from exploitation status, technical impact, whether exploitation is automatable, mission prevalence, and public well-being impact.

    CISA: Stakeholder-Specific Vulnerability Categorization

  4. Severity, read properly: CVSS

    Read the CVSS Base score as a reasonable worst case for the flaw itself. In version 4.0, FIRST's Threat metrics (the renamed Temporal metrics) adjust it with what is known about exploitation, and Environmental metrics adjust it for your own deployment.

    FIRST: CVSS v4.0 user guide

  5. Test in the same order: risk-based testing

    ISTQB defines risk-based testing as choosing, prioritizing and managing test activities on risk analysis: keep a register of risks with their likelihood and impact, start early, and run the tests that cover the biggest risks first.

    ISTQB Certified Tester Foundation Level syllabus 4.0.1, section 5.2

The reworked workflow

Shift left, keep testing right

Shift left means testing earlier, and ISTQB adds that it does not mean neglecting testing later. At AI speed both ends matter more: requirements and tests have to exist before an agent writes the code, and checks have to keep running after release because new attacks and new incidents arrive every week. Five stages, the tests from this center that belong in each, where AI helps, and the guardrail that keeps it honest.

  1. 1

    Before the sprint: the definition of ready

    Refinement, before anyone builds

    Acceptance criteria and the tests that will check them are written with the story, not after it. A requirement that cannot be tested goes back to its owner.

    Where AI helps: Drafting acceptance criteria and test cases from the story, and flagging words that cannot be tested.

    The guardrail: A person who owns the risk approves the criteria; the agent proposes, it does not decide.

  2. 2

    While coding: tests first

    In the editor, on every change

    Unit tests and the security tests for the feature are written before or with the code, and run on every change.

    Where AI helps: An agent writes the failing tests first, then the code that makes them pass.

    The guardrail: The agent may not change a test to make it pass; a changed test is a reviewed change.

    Code a model writes is rejected unless it runs in a sandbox

  3. 3

    At the pull request: the fast gates

    Every pull request, in minutes

    Static checks, secrets, dependency advisories, pinned components and instruction files are checked before review, so a person reviews only what passed.

    Where AI helps: Review agents point out missing tests and risky changes; they comment, they do not approve.

    The guardrail: Required checks block the merge; nobody, human or agent, can skip them.

    The build fails on a known, fixed CVE in your AI stackEvery MCP server is pinned to an exact versionAgent instruction files hold no invisible charactersNothing in the deployment mounts the Docker socket or turns authentication off

  4. 4

    Before release: the slow, thorough tests

    On every release candidate

    Integration, API, end-to-end and the agent security tests run against a production-like environment, each agent test many times.

    Where AI helps: Agents explore the application, write end-to-end tests from a plan and repair broken locators, for a person to review.

    The guardrail: An agent test passes only when every run passes, and a healed test is a reviewed change.

    An email's text is read, never obeyed, run after runReading a web page never leads to reading .envA high-risk action waits for a person, is logged, and the wait cannot be skippedThe agent's sandbox reaches the allowlist and nothing elseA wrong or default token is refused on every routeA web page cannot drive your local agent server

  5. 5

    In production: keep testing right

    Continuously, after release

    Monitoring, the audit trail and scheduled re-runs catch what no test before release could: new attacks, new incidents, drift.

    Where AI helps: Triaging alerts and failed runs, and turning a new incident into a new test.

    The guardrail: Every agent action is recorded where the agent cannot change it, and someone reads the alerts.

    The agent's account owns no public repositoryThe agent's sandbox reaches the allowlist and nothing else

AI pipeline: the tools and the files

The free tools at each stage, the four files that wire them in (pre-commit, the pull request gate, the agent's definition of done, a hardened agent workflow) and GitHub's hardening guidance are on the AI pipeline page, under AI, cloud and DevOps.

Design time

Threat-model the agent before it exists

Shift left ends at design. Before an agent exists, answer four questions about it, the same four the Learning Center's Blast radius page asks, and read the incidents that answer them: what it reads, what it can reach, what it can do, and what it is built on. Each answer is a threat and a test.

  1. What it reads is who can instruct it

    Every input the agent reads is a place an attacker can write: web pages, email, tickets, documents, repositories, logs. Decide which ones it needs, treat all of them as untrusted, and write the prompt-injection tests for each before the agent reads its first page.

    Weaknesses: Agent goal hijack via untrusted content (prompt injection)Hidden instructions in invisible or disguised text

    Tests: An email's text is read, never obeyed, run after runOpening an unknown repository runs nothingA web page cannot drive your local agent server

  2. What it can reach is what it can lose

    Secrets, private data, production systems. The design question is which of these the task needs; the default is none. An agent with a narrow identity of its own, never a person's tokens, limits what a hijack can take.

    Weaknesses: Identity and privilege abuse by agentsLong-lived credentials in agent and MCP configuration

    Tests: Reading a web page never leads to reading .envThe agent's database role cannot delete or drop

  3. What it can do is the blast radius

    Reaching the internet, sending messages, running commands, pushing code. Each capability needs a gate: an allowlist for destinations, a person's approval for consequences, a sandbox for commands. The gate is designed, then tested, then enforced where the agent cannot change it.

    Weaknesses: Unrestricted network egress for agentsNo human approval for consequential actionsAgent sandbox escape

    Tests: The agent's sandbox reaches the allowlist and nothing elseA high-risk action waits for a person, is logged, and the wait cannot be skippedFrom inside the sandbox, localhost is not a way out

  4. What it is built on is your supply chain

    Packages, MCP servers, skills, model files, the vendors that see your data. Pin them, sign what you can, scan model files before loading, and write down the bill of materials, so that a change reaches production only through a reviewed pull request.

    Weaknesses: Agentic supply chain compromiseMCP server changes after approval (rug pull)Unsafe model serialization (pickle)

    Tests: Every MCP server is pinned to an exact versionSkills and instruction files name only hosts you ownAn MCP server's tools match the snapshot you reviewedThe build fails on a known, fixed CVE in your AI stack

What comes next

Beyond security: the same machinery for everything else

Security is where testing AI systems started, because the incidents forced it. The same machinery, tests that run in CI and fail for a reason, is about to cover the rest of what makes an agent fit for production. Six directions, each with the open tools that exist today.

Observability-driven testing: production traces become the next test

OpenTelemetry's semantic conventions for generative AI, now kept in their own repository, define spans, metrics and events for model calls, agents and MCP, and tracing platforms such as Langfuse (open source, MIT) and Arize Phoenix (source available under the Elastic License) store, replay and score the traces. When every agent run is a trace, a failed run in production is a test case waiting to be copied into the suite, and an eval can run against last week's real traffic rather than a hand-written sample.

Sources: OpenTelemetry GenAI semantic conventions (repository) · OpenTelemetry: generative AI conventions, moved · Langfuse on GitHub · Arize Phoenix on GitHub

Budgets as tests: tokens, time and money per task

An agent that loops burns money, and an agent that is slow is not used. A test that runs the task and fails when it exceeds a token budget, a wall-clock limit or a number of tool calls turns a bill into a regression, caught before release; the cloud spend caps the providers now offer are the backstop, not the test.

In the Learning Center

Sources: Simon Willison: default hard budget caps on pretty much everything

Testing the tests: do AI-written tests prove anything?

Agents write tests cheaply, and a test that passes against broken code is worse than none. Mutation testing introduces changes to the code and runs the tests against the changed code: a mutant is killed when a test fails, and a surviving mutant means the tests did not cover that code. Stryker does this for JavaScript, TypeScript, C# and Scala, mutmut for Python. A pull request from an agent can be required to kill the mutants in the code it touched.

Sources: Stryker Mutator · mutmut on GitHub

Spec-driven development: the spec is the oracle

When an agent writes most of the code, the specification is where a person's intent lives, and the tests are its executable form. GitHub's Spec Kit (open source, MIT) gives coding agents a process that turns requirements into a specification, a technical plan and actionable tasks before implementation, then checks the result until it converges, which is the definition of ready from this center's principles page, made into a tool.

On the principles page

Sources: GitHub Spec Kit

Quality in the same pipeline: accessibility, resilience, privacy

The pipeline that gates on security can gate on the rest. Playwright runs axe-core accessibility checks in the same end-to-end tests; a resilience test makes a tool fail and checks that the agent degrades gracefully rather than improvising; a privacy test scans prompts, traces and test data for personal data that should not be there.

Sources: Playwright: accessibility testing · OWASP AI Testing Guide

Official guidance

What the standards ask for

NIST SP 800-218: Secure Software Development Framework 1.1

NIST, 2022; version 1.2 in draft since December 2025

Secure development practices for any lifecycle. For testing: review the code (PW.7) and test the executable (PW.8) so vulnerabilities are fixed before release, and record and triage every issue in the team's tracker.

NIST SP 800-218A: secure development for generative AI

NIST, July 2024

Adds AI-specific practices to the SSDF. Its practices do not distinguish human-written from AI-generated source code, because all source code should be evaluated for vulnerabilities and other issues before use.

Secure by Design

CISA, 2023

Asks software makers to treat customer security as a core business requirement and to significantly decrease the number of exploitable flaws before they reach the market.

OWASP SAMM: Security Testing

OWASP, SAMM version 2

A maturity path for security testing: from automated tools, to application-specific automation, to automated security testing built into build and deploy, alongside manual expert testing.

ISTQB Foundation Level 4.0.1: shift left

ISTQB, September 2024

Defines shift left as testing earlier in the lifecycle, and says it does not mean testing later should be neglected. Good practice: review specifications as a tester, write test cases before the code, use continuous integration for fast feedback.