41.9% of engineering leaders have rolled back an AI change that broke in a way unit tests didn't catch in the last 90 days. 32.4% have done it more than once.
Those were the results from Checksum’s research into the state of AI code, which found that the testing infrastructure most teams are still relying on isn’t fit for purpose. It’s time to adopt AI testing, but now the question is: "How do we actually adopt it without breaking everything we have?"
That's what we’re covering here.
Still weighing whether AI testing is right for your team? Start with our complete guide to AI testing. This practical guide picks up where that one leaves off.
Step 1: Run a pre-adoption audit
You need to understand what you're working with before you touch any tooling. Skipping this step means having no baseline to measure improvement against, and no way to demonstrate ROI when it matters.
Here are the three questions to answer, supported by data and analysis from Checksum's analysis of 1M+ production test runs:
- How much time does your team spend on test maintenance?
Teams spend around 1.3 engineering hours per failure and 41% of that is investigation alone, before anyone writes a fix. For a 500-test suite, where the median team sees 14.8 failures per 100 runs, that works out to roughly 1,850 failures a month and 2,405 engineering hours: $360,750 a month, or $4.3M a year, in maintenance cost.
Assuming an engineer rate of $150/hour, that's the equivalent of roughly 14 full-time engineers just keeping tests green. Run the same math on your own suite size and failure rate to see where you land.
- What share of your failures are real bugs?
The answer is probably lower than you think. Selector changes account for 32% of failures and loading/timing issues another 19%: mechanical breakage where the test's intent hasn't changed, only the implementation details underneath it. Flow changes (27%) are the category most likely to signal an actual product change worth a second look.
True flakiness (e.g. environment instability, bad deployments, flaky networks ) accounts for under a quarter of failures overall, and under 10% in mature, stable systems. If your team is triaging ten failures a week, the data says most of them are selector or timing noise, not real regressions. That's the maintenance tax AI testing is designed to absorb.
- Which flows break most often?
Selector changes account for 32% of failures; flow changes account for another 27%. Map those percentages to your own suite. Which tests fail most frequently? Which ones get disabled rather than fixed? Which critical flows does nobody trust enough to gate a release on? Use that list as your starting point.
The audit output is two things: a baseline to measure against, and a prioritized target list. We’ll return to both in step six.
Step 2: Start where failures live
A common adoption mistake is treating AI testing like a greenfield project: mapping the entire application, generating tests for everything, and targeting full coverage on day one. That approach takes longer and makes it harder to build the internal trust that sustains adoption.
The right starting point is your ten to twenty highest-churn, highest-value flows.
Look for these in:
- Signup and onboarding
- Checkout and payment
- Core feature paths that ship on every sprint
- Anything that has broken three or more times in the last 90 days
- Any flow that currently gates a manual release checklist
These flows are typically consuming the most maintenance hours. They're also the ones where a failure is most likely to reach production. Fixing them first means your team feels the improvement immediately, giving you concrete data to show within the first few weeks.
Step 3: Decide what to do with your existing test suite
One of the most common questions asked when adopting an AI testing tool like Checksum: do we throw away what we have?
You don’t, but you do need a plan for it. Here’s what we advise, based on what you already have in place:
- You have a brittle Cypress suite
If you're on Cypress and your suite breaks constantly, or if you're in the middle of a migration to Playwright, AI testing can handle the transition without a months-long rewrite. Engagement Agents faced exactly this situation: a brittle Cypress suite and a major UI redesign on a tight deadline. Rewriting every test manually would have taken months. Instead, they used Checksum to migrate 500 existing tests in a week and launched the redesign 30% faster. An anticipated two-month delay became a one-week project.
- You have Playwright but tests keep breaking
If you already have Playwright coverage but the suite demands constant attention, you need to take the maintenance burden off your engineers with AI testing that runs alongside your existing suite. Auto-healing handles the selector changes, flow updates, and UI churn that previously landed on a person. Your tests stay in your repo as standard Playwright code and the system keeps them current.
- You have minimal coverage
If you're starting from near-zero—either because tests were never written or because the suite was abandoned—AI testing generates coverage from scratch by mapping your real application flows. ClearPoint Strategy built 250+ E2E tests in under a month, coverage that would have taken a team months to author manually.
Whatever your starting point, the goal is consolidation, not parallel maintenance. You're not building a second suite alongside the first; you're transitioning to infrastructure that maintains itself.
Step 4: Plan for week one
Setup has four parts. Know what's involved before you start so you're not surprised mid-week.
Environment access. Checksum needs a staging URL and login credentials i.e. test user accounts it can use to authenticate against your application. If your staging environment requires specific setup or your test credentials are managed separately, get these in order before you start to save time.
Git integration. You'll connect your GitHub or GitLab account so Checksum can read your codebase and open pull requests with generated or healed tests. Changes come back as PRs your team reviews in the same place as everything else, which makes the workflow feel native rather than bolted on.
CLI and test repository setup. A short initialization step (npx checksumai init) creates the configuration files, Playwright config, and an example test in your repository. If you're adding Checksum to an existing repo rather than creating a new one, there's a setup path for that too.
Verification. Before any test generation happens, Checksum runs a validation test against your login flow to confirm the environment is correctly configured. If it passes, you're ready to start detecting flows and generating tests.
Here’s how to handle questions that commonly come up in the first week:
- "How do I know these tests are checking the right things?"
They're standard Playwright code in your repo. Read them, run them, push back on the PR if something looks off.
- "What happens when the AI gets it wrong?"
Nothing is silently applied. Every auto-heal opens a PR so your engineers can see the diff, understand what changed, and approve or reject it.
- "Who owns this now?"
The tests live in your repository and run in your CI pipeline. Checksum maintains them, but you own the output.
Step 5: Get your team on board
The engineering leader who evaluates and buys an AI testing platform, then announces it in a Slack message, will have a harder time than the one who brings the relevant people into the evaluation early.
Here are three conversations to have before you start.
With your QA engineer or QA manager
Your QA team will want to know: "Is this going to replace me?"
The answer is no. AI testing removes the low-leverage work: selector archaeology, triage of stale failures, writing boilerplate test cases for flows that haven't changed. It doesn't replace the judgment about what edge cases matter, what exploratory testing should probe, or how to interpret ambiguous failure signals.
Söderberg & Partners, a financial services firm, had a team running manual E2E testing before every release. It was necessary work, but releases became rare and painful because someone had to run through a long testing checklist before every ship.
After deploying Checksum, Söderberg & Partners reclaimed 90 hours of manual testing per month. The team didn't shrink but the work changed: less time running repetitive release checklists, more time on the quality decisions that actually require human judgment.
With engineering skeptics on your team
The concern most often here is reliability: "AI-generated tests sound like they'd be noisy and wrong half the time."
The answer is that reliability is a system property, not a property of any single AI output. Generated tests are validated by running against the real application before they're committed. Auto-healed tests are re-run after updating to confirm they still check the intended behavior. All changes surface as PRs, not silent rewrites. And every test is standard, inspectable Playwright code in your repo so engineers can read exactly what's being checked.
Shift the question asked by these skeptics towards "do you trust a system with these safeguards in place?" and confidence should rise.
With your engineering leader or budget holder
This audience wants to know about ROI; Checksum’s data from 1M+ production test runs has useful benchmarks for framing the conversation.
Teams spend around 1.3 engineering hours per failure with an average of ten failures per sprint. That's 13 hours per sprint on maintenance, about 1.5 engineers per month.
Checksum customers see an 82% failure rate reduction. Roughly 70% of failures resolve without any engineer touching them. The math on engineering time reclaimed is straightforward; the only variable is your team's current failure rate.
One more data point for your engineering leader or budget holder: in Checksum's AI code survey of 105 engineering leaders, 70.5% said more headcount alone wouldn't solve the review and maintenance burden. The leaders who know this problem best don't think hiring is the answer. AI testing is.
Step 6: Set up success metrics
Using a baseline from your audit and set these up before you start.
Failure rate
Track weekly. Checksum’s benchmark is an 82% reduction compared to manual maintenance. If you start at 14 failures per 100 runs, the target is around 2–3. This is the headline metric and the one that speaks most directly to your engineering leader.
Auto-heal rate
What percentage of failures is the system resolving without engineer intervention? Checksum’s benchmark is roughly 70%. If your auto-heal rate is significantly lower, look at the failure types: environment instability and complex flow changes are harder to resolve autonomously than selector changes.
Time to green after a product change
How long does it take from a UI or flow change shipping to the suite going green again? Before AI testing, this is often measured in days. With AI testing, it should be measured in hours or less.
Engineer hours on test maintenance
This is the number that shows up in sprint capacity. Track it against your pre-adoption baseline. At 1.3 hours per failure, an 82% reduction means your team gets back roughly one full engineering hour per failure, per sprint.
Production incidents from AI-generated code
61% of engineering leaders surveyed in Checksum's State of AI code report shipped an AI-originated production incident in the last 90 days. Track this number for your team quarter over quarter. It's a lagging indicator, but it's the one that reflects the actual cost of the verification gap.
Suite trust
This one is qualitative but important. Are engineers actually investigating failures when they surface, or dismissing them as likely noise? A suite nobody trusts is a suite that provides no signal. Watch whether the culture around test failures changes as the false positive rate drops.
Bonus: What not to do
Don't treat what the system detects as complete
Checksum automatically identifies the flows it considers most important and maps your application's user journeys. That detection is a strong starting point, but it won't know which flows carry the highest business risk, which edge cases have burned you before, or what ‘passing’ actually means for a feature with complex business logic.
It remains vital to review the detected flows, add the ones it missed, and remove the ones that don't matter. While the system handles authorship, the coverage strategy is still yours.
Don't treat setup as a one-time project
AI testing is infrastructure, not a deployment. It runs continuously, adapts to product changes, and generates new tests as your application grows. It needs the same ongoing attention you'd give to any part of your CI pipeline: reviewing auto-healed PRs, validating new coverage, expanding to new flows as the product ships them.
Don't skip defining success criteria upfront
Use the metrics from step six as your framework: pick two or three that matter most to your organization and document them before the first test runs. If you don’t agree on what ‘working’ looks like before you start, a skeptic can always find a reason to call the results inconclusive. A shared definition of success removes ambiguity from the evaluation.
Don't exclude QA from the process
Bring QA in at the evaluation stage, not the rollout stage. The teams that move fastest with AI testing treat QA as the domain expert who shapes what the system covers: the person who knows which flows need the most protection, which edge cases get missed, and which failure signals are meaningful.
What getting it right looks like
Checksum customers demonstrate the impact of having AI testing in place. Different industries, team structures, and starting points but the same outcome: engineers spending their time building, not maintaining.
Stellic is an edutech company that provides academic planning software for higher education. E2E testing was becoming a challenge as their platform scaled and the team needed a solution that could generate and auto-heal E2E tests, not another tool to babysit. Checksum gives Stellic broader coverage with the same headcount and, with manual testing down 40%, the team can now focus on scaling the platform and delivering value.
Reservamos is a travel tech platform serving multiple enterprise clients. Real-time market data and multi-tenant deployments made their tests flaky and required constant manual upkeep. The cost of that maintenance wasn't just money, it was engineering time that should have been going elsewhere. Within a month of adopting Checksum, they had a complete test suite, had reclaimed 20% of engineering time, and were saving $200K annually. The suite now updates itself as each client environment changes.
Closing the verification gap
The gap between writing and verifying code was always there. AI coding tools made it visible by widening it faster than teams could compensate manually. Every quarter there's more code, more surface area, and engineers struggling to plug the gap.
Checksum's E2E Agent automatically generates, runs, and auto-heals a production-ready test suite: AI code gets verified before anyone reviews it, not debugged after it ships. Every test is standard Playwright code committed to your repo. No lock-in.
Request a demo to see coverage in your actual application before you commit.
FAQs
Do you need to replace your existing test suite to adopt AI testing?
No. AI testing works with what you already have. It can migrate a brittle Cypress suite, run alongside existing Playwright coverage to absorb maintenance, or generate coverage from near-zero. The goal is to consolidate onto self-maintaining infrastructure, not build another suite to maintain in parallel.
Will AI testing replace QA engineers?
No. AI testing removes low-leverage work such as selector fixes, stale-failure triage, and boilerplate test authoring for flows that haven't changed. It doesn't replace the judgment QA brings to prioritizing edge cases, running exploratory testing, or interpreting ambiguous failure signals. This is why teams that adopt it fastest bring QA in at the evaluation stage, not after rollout.
How do you measure whether AI testing adoption is working?
Track failure rate, auto-heal rate, time to green after a product change, and engineer hours spent on test maintenance against a pre-adoption baseline. Checksum's benchmark is an 82% failure rate reduction and roughly a 70% auto-heal rate; if a team's numbers land well below that, the failure-type breakdown (environment instability vs. selector changes) usually explains why.
