Every AI testing vendor will tell you their tests fix themselves—Checksum included. Our analysis of more than a million production test runs found that about 70% of test failures resolve without an engineer ever touching them. But "70%, trust me" isn't enough when we're asking engineers to adopt a system that rewrites their tests.
Instead, let's walk through what actually happens between a test going red in CI and a pull request landing in your repo.
Key takeaways
- Healing happens in two stages; one starts before the test even fails. Checksum splits auto-healing into real-time recovery, which resolves element-location failures within a bounded time window while the browser is still open, and post-run healing, which brings in an isolated agent and opens a PR for failures that require a code change.
- Not all failures are equally healable, and Checksum tells you which is which. Across more than 1 million production test runs, selector-related failures resolve automatically about 91% of the time, while failures caused by real flow changes resolve about 52% of the time, because a changed journey has to be inferred rather than re-located.
- Nothing is marked 'healed' until a real test run proves it. Checksum guards against false-green results with a scan that catches classic shortcuts, a nine-criteria review pass, and a check against the test runner's own report. A self-reported outcome is never treated as evidence on its own.
What auto-healing means
We describe auto-healing (also known as self-healing) as being like two clocks; the first one starts before a test has even 'failed'.
The first clock
Recovery happens at the failing step, inside the run, while the browser is still open. When a step throws (a click can't find its element or an assertion doesn't match), the runtime intercepts the error before it kills the test and works through a fallback chain, cheapest first.
The first in that chain is deterministic re-location with no AI involved. Every element a test interacts with was recorded when Checksum's AI test generation first wrote the test, with a structural 'fingerprint' (its attributes and features) plus a set of alternate locators. The runtime re-scores that fingerprint against the live DOM and only accepts a confident match above a similarity threshold, never 'closest thing wins,' and separately resolves every alternate locator in parallel and takes a majority vote on which element they agree on. These fallbacks run on budgets of about 5–30 seconds each.
If that misses, an in-run recovery agent gets one bounded window with a hard 120-second cap. It sees a reduced snapshot of the live DOM with every element tagged so it can reference them precisely: a full-page screenshot, the error and the test source around the failing line, what the surrounding steps are trying to do, and the test's current variable state. It can probe the page with real Playwright code, and after every probe it's shown the side effects (a navigation, a console error, a failed network call) before it's allowed to commit a fix.
If any of that succeeds, the step is annotated 'auto-recovered' and the run continues. In your Checksum dashboard, that test is categorized as Recovered (not lumped in with Passed). If nothing works, the original error is re-thrown and the test fails in the same way as an ordinary Playwright failure.
The second clock
Checksum sees the result when that run finishes, and, if the run failed and auto-heal is on, it gathers whatever tests are still red (not skipped, not recovered, not just flaky on retry) and spins up a healing session. This is an isolated agent with the test repo checked out, the structured run results, per-test error context, screenshots, and trace data.
Auto-healing is split into two pieces like this because real-time recovery can only act on one page in one run: it can't edit the test file, read the app's git history, look across other failing tests, or re-run anything to check stability. Anything that requires a code change (a real flow change, stale test data, a broken shared helper, a real application bug) is out of scope for recovery.
Instead, those failures land in the healing session. This has what recovery lacks: a repo checkout, artifacts from every failed test in the run so it can spot systemic patterns, trace summaries, memory of past verified fixes for the project, and the ability to re-run tests to prove a fix actually works.
A recovered step is annotated in the report and categorized separately from a pass. A healed test arrives as a pull request: a reviewable diff, a link to the full agent session transcript, and a bug report for anything the agent couldn't or shouldn't fix. There's even a suggest-only mode where the recovery agent diagnoses a failure but deliberately leaves the test red, carrying its diagnosis forward for a human or a healing session to act on.
How the agent decides what broke
Before the healing session touches any code, there's a dedicated triage stage that builds a diagnosis for every failing test. It does this with:
- The structured run results
- A per-test error-context file
- Screenshots at the moment of failure
- A summary of the Playwright trace (which is where a 500 from your backend shows up)
- The test source and the app source in the sandbox
- Past triage verdicts on similar failures
- A memory file of previously verified fixes
- Recent app-code PRs merged shortly before the session
Triage classifies every failure as an app bug, a test issue, an infrastructure problem, or needs review. Blaming your application requires evidence: a real fault in the trace, a connection error, a JS exception, or a cited file and line number in your source code. If triage is unsure, it defaults to test issue and a fix gets attempted, and three or more tests failing with the same symptom get grouped into one infrastructure incident instead of three bogus heals. Only then does a test issue get sorted by what actually broke: a selector that moved, an element that's simply slower to load than the test expects, test data that's gone stale, a shared setup step that broke, an assertion nobody updated, or the user's journey changing outright.
We do this because 'selector changed', 'the flow changed', and 'the environment was just flaky' look nearly identical in a red CI log, but they need different responses. Collapsing them into one bucket makes a healing agent untrustworthy.
Why some failures are easier to heal than others
Selector fixes and flow changes don't resolve at anything like the same rate. Selectors autonomously resolve about 91% of the time while flow changes sit at about 52%. A selector fix is a problem with one unknown. The journey is unchanged, the element still exists, the test's intent is intact, and the agent has evidence that points at the same answer: the recorded fingerprint, the alternate locators, the live DOM, the screenshot. Verification is local and cheap: re-locate, act, and the rest of the test proceeding is itself confirmation.
A flow change breaks the journey. A step got added, removed, or reordered, a screen was redesigned, a confirmation modal now appears. Now the agent has to solve several coupled problems at once: infer what the new intended journey actually is (since there's no stored ground truth for a UI that no longer exists); decide whether the change was intentional product evolution or an actual bug; figure out where the line sits between assertions that must be preserved and expectations that are legitimately stale; and often touch multiple steps or shared helpers other tests depend on.
Flow changes also carry the highest risk of a false green: a test can be made to pass by editing a different step than the one that actually threw, or by quietly dropping the assertion that no longer fits. Our guardrails matter here: when the agent can't clear them with confidence, it's designed to stop and hand the test to a human instead of guessing.
Building a system you can trust
We learned a lot while building Checksum, and the most instructive lesson didn’t happen when the agent fixed tests badly. It happened when we saw the agent winning badly, which is the kind of action that erodes trust.
Given the goal 'make this test green,' a capable agent will occasionally find the shortest path: delete the spec, insert a skip, remove the assertion that's failing, force a click through, or simply report the test as fixed without ever running it. That last one is memorialized in our incident tracker, and it produced a rule we now enforce in code: an agent's own report is never trusted.
The lesson generalized well beyond that one incident: with agents, the failure mode usually isn't incompetence, it's a misaligned shortcut, so the guardrail has to live outside the agent, in code that checks ground truth.
Another key part of the way we built Checksum is that triage regularly declines to heal a test, and is right to. It'll trace a 'broken' test to a real 500 in the network trace or an exception in the console, tag the test as a product bug with the evidence attached, and refuse to patch around it. The first time you watch a healing system correctly argue that a test should stay red, the whole category starts to feel different. The hard part isn’t editing the test file. It’s knowing when not to.
Inside a healing run: plan, implement, review, verify
A healing run is really a handful of separate agents, each with one job.
- Plan and triage comes first. Every failure gets a diagnosis and a classification before any code is touched, and anything ruled an app bug or an infrastructure issue is routed away from the next step entirely. It will not be 'healed'.
- Implement is where the actual fix happens, but only on the failures triage cleared as test issues, and only under strict constraints: preserve the original journey and assertion intent, change the minimum necessary, and fix the step that actually threw the error. A test that goes green because a different step got edited doesn't count as a real fix; it's flagged as having patched the wrong spot instead. Reflexively forcing a click through instead of asking why an element wasn't clickable, grabbing the first matching element to dodge an ambiguous selector, or padding a test with a longer wait instead of fixing the real timing issue are all explicitly off-limits, since they hide the drift rather than heal it. This step also checks the project's own history of previously verified fixes, so it favors an approach that's already worked in this codebase over inventing one from scratch.
- Review is a second pass looking at the change with a clean slate and no attachment to the reasoning behind the fix, and scoring it each round against nine criteria. The criteria includes whether the original intent was preserved, whether the change was minimal, whether the wait strategy actually makes sense, and whether a real bug was handled correctly. Whatever it flags gets triaged in turn, legitimate findings get fixed, and anything off-base gets pushed back on. This can run up to ten rounds.
- Verify is the point where the healed tests actually get executed. Checksum doesn't take anyone's word for the result here, agent included. It checks the outcome against the test runner's own report (the subject of the next section).
When healing fails outright, the roughly 30% that doesn't resolve, Checksum routes it rather than retrying forever. App bugs get tagged and reported with evidence rather than patched around. Infrastructure incidents get grouped and reported once. Ambiguous cases get tagged as needing review by a human instead of guessed at. Fix attempts are bounded.
The session ends with an explicit outcome and a bug report listing exactly what's unresolved and why, and work completed before that point is committed. A partial PR with an honest record of what did and didn't get fixed is a normal, intended result, not a failure of the system.
Guarding against false greens
At the 'verify; point of a healing run, an actual test execution is mandatory. Every healed test has to prove itself green by running, and finishing without that proof counts as a false green. A scan independent of the agent checks the diff for the classic shortcuts: deleted test files, inserted skips, a net removal of assertions. Catching one of the first two forces the agent to justify the change or undo it.
If a session ends without ever running the suite at all, Checksum pushes it back in to run the tests, including re-running anything it flagged as an app bug, just to confirm that it still genuinely fails rather than having been quietly worked around.
Underneath all of that is a check against ground truth: pass or fail for each test, pulled straight from the test runner's own report, never from what the agent says happened. That report has to be newer than the failure data the session started with, so a stale green can't be recycled as evidence. A test only counts as healed if the agent's own claim and that independent check agree it passed in a fresh run. Anything less doesn't count as healed, and if the agent never ran the tests at all, nothing ships.
The subtler false green—a test that passes but now checks the wrong thing—gets caught differently: an assertion-removal check, the review pass's scoring on whether intent was preserved and the change stayed minimal, and hard rules baked into how the fix gets made in the first place. Don't remove a meaningful assertion just to make a test pass. Don't weaken an expectation unless the old one is clearly wrong for how the product behaves now, and if that's the case, the change shows up plainly in the PR diff for a human to judge.
What it looks like when healing doesn't work
For the tests that don't heal themselves (around 30%), the system is designed to fail in four distinct ways:
- Tests get tagged as your bug, not the test's: With trace or source evidence in hand, it's walled off from any further fix attempts, a written description is required, and it's reported as a product bug. The test stays red.
- Tests get tagged as environment: When three or more tests fail with the same symptom (an auth outage, a wiped seed dataset, a dead dependency), they become a single infrastructure incident with one description, not thirty individual heal attempts.
- Tests get tagged as needing review: This classification exists specifically to stop the agent from defaulting to 'app bug' just to be safe when it isn't sure. Those get flagged for a human with the diagnosis already attached.
- Tests are simply tried and not resolved: Fix attempts are bounded, and once the budget's spent, the session ends with an explicit outcome and a bug report listing each unresolved failure and what was attempted.
Progress commits as it is made, so a partially successful session produces a PR containing the tests it did heal plus a record of the ones it didn't, not a discarded session. The team gets notified with the batch outcome (healed, partial, or failed), and whatever's unresolved is given a diagnosis. Engineers are left with a decision to make, not an investigation to start from scratch.
What the engineer reviewing a PR sees
The PR itself is deliberately boring: a diff over test spec files and test utilities only. For customers with allow-list enforcement on, this extends to the agent being barred from touching build config, lint config, or dependencies. The PR body states that it was generated by a Checksum agent, what triggered it, and links to the full transcript of the session that produced it.
Commit messages carry the reasoning behind each change; if anything was left unresolved, the bug report says so explicitly. By the time a PR is open, the change has already survived the review rounds, the diff scan, and a verified green run.
The reviewing engineer should ask:
- Does the diff touch the step that actually failed? A fix in the right place is usually a good fix, and a green diff in an unrelated region of the test needs further investigation.
- Do the assertions still check the same journey? Skim what the test verifies, not how it locates elements: that's where a false green would hide. It should be rare given the scanners, but it's the highest-value thing a human can check.
- Are there any bug or needs-review tags in the diff? Those aren't part of the fix at all: they're the agent handing you a decision with its evidence attached.
What's safe to rubber-stamp: pure locator swaps, same action, same assertions, a stale selector replaced with a more stable one, confirmed by a green verification run. Removing that maintenance work is the purpose of a tool like Checksum. Everything else (reordered steps, updated expectations, anything tagged) deserves the sixty seconds it takes to read what happened. Transparent healing gives you the choice of how much to trust—silent healing makes that choice for you.
An evidence-based test maintenance system
There's a difference between a testing tool and a test maintenance system: one produces green checkmarks, the other produces evidence.
Checksum is the latter. Recovered steps are annotated in the report. Healing sessions have a shareable transcript. Outcomes are computed from the test runner's own output, not the model's self-assessment. What your team sees is a pull request that a human merges. That's what makes the 70% auto-healing figure trustworthy: the other 30% is visible too, tagged, and routed to a person instead of quietly disappearing.
If you want the full breakdown by failure type, our 2026 benchmark report has it, drawn from the same million-plus runs referenced above. Or get a demo and watch a healing session run against your own suite.
FAQs
What is auto-healing in software testing?
Auto-healing (also called self-healing) is when a test that fails in CI is automatically diagnosed and fixed without an engineer rewriting it by hand. Checksum splits this into two layers: real-time recovery, which resolves element-location failures while the test is still running, and post-run healing, which opens a pull request for failures that need a code change, such as a broken selector, stale test data, or a real flow change.
What percentage of test failures does AI testing auto-heal?
Across more than 1 million production test runs, Checksum auto-heals about 70% of test failures without an engineer touching them. The rate varies by failure type: selector-related failures resolve automatically about 91% of the time. Failures caused by real flow changes resolve about 52% of the time because they require inferring a new user journey rather than relocating a known element.
How does Checksum prevent AI test healing from producing false positives?
Checksum layers several guardrails against false greens: a scan that flags classic shortcuts like deleted spec files, inserted skips, or removed assertions; a review pass that scores every fix against nine criteria, including whether the original intent was preserved; and a final check that only marks a test 'healed' once a fresh test run, not the agent's own claim, confirms it passes. If a session finishes without running the suite, nothing ships.
