There's a concept in climbing that resonates with where many engineering teams find themselves today.
A climber who trusts their gear moves completely differently than one who doesn't. Trust your protection and you commit. You go fast. You try moves you're not sure you can do. But if you don't trust it, you hesitate on everything—including the easy moves. The fear of falling slows you down far more than the actual difficulty of the climb.
In software, an E2E test suite is supposed to be that protection. Lose the trust and it starts slowing you down.
Equally, protection only works if it covers the actual route. Gear that's solid but anchored to the wrong rock face gives you the same hesitation as gear you don't trust. Worse, it gives false confidence. A suite that's green but isn't testing the conditions your code will actually run in is exactly that: protection that looks solid but isn't covering the route.
Key takeaways
- When engineers can't rely on their E2E suite, they hesitate. This shows up as engineers re-running CI, merging with doubt, and avoiding risky-but-necessary changes. The slowdown creates behavioural drag across every sprint.
- A flaky, decaying suite is not the only source of doubt. Misplaced confidence is dangerous when even an accurate, well-maintained suite can miss failures that only show up against real user flows, real data, and real third-party APIs.
- Auto-healing tests change the economics of test maintenance and coverage. Selector changes and flow changes each cause around a third of E2E failures. With auto-healing, around 70% of failures resolve without engineer intervention. This keeps the suite accurate even as PR volume is up 23% year over year.
The cost of a broken E2E suite
The obvious cost of a flaky test suite is the time spent fixing it. Selector changes, flow changes, tests that go red for reasons that have nothing to do with the code that triggered them. Checksum analysis of 18,000 test failures found teams spend roughly 1.3 hours on every failure. Selector changes (32%) and flow changes (27%) make up almost two thirds of test-suite failures.
Those costs are real and measurable. It's the behavioral cost of a broken E2E suite that's harder to quantify.
When engineers can't tell whether a failed E2E test means something actually broke, they make a choice. Most of the time, they re-run CI and hope it passes. Sometimes they merge anyway in the hope that the failure is unrelated. Sometimes they slow down, dig in, and spend an hour on selector archaeology before concluding the test was wrong, not the code.
None of these actions show up in your incident log.
What they show up as is hesitation. A team that's spent enough cycles on selector archaeology starts treating every red build with suspicion—re-running it, merging anyway, or quietly avoiding the components most likely to break the suite again.
But the maintenance burden is only half of the trust problem. The other half is what happens when the suite is green and the code still breaks in production anyway.
Why CI/CD alone won't solve a flaky test suite
For the last decade, the answer to shipping faster has been CI/CD. Automate the delivery pipeline, remove the manual gates, and let code flow from commit to production with minimal friction.
CI/CD solved the delivery problem. It didn't solve the verification problem.
What most CI/CD pipelines do is run the existing test suite faster and more frequently. But if the suite is flaky, running it on every commit just surfaces flakiness more often. If coverage has gaps, automating deployment doesn't close them. If tests break every time a selector changes, faster pipelines mean faster alerts about tests that may or may not mean anything.
CI/CD moved the bottleneck from delivery to verification. And most teams have been living in that bottleneck ever since, patching it with manual effort that doesn't scale and a growing tolerance for red builds that nobody's quite sure how to interpret.
There's a growing need for continuous verification.
Defining the value of continuous verification
CI/CD automated how code gets delivered. Continuous verification automates how it gets proven.
The distinction matters because delivery and verification are different problems. Delivery is about moving code from a repository to a running environment. Verification is about truly knowing that the code behaves the way it's supposed to in that environment—against real user flows, your actual database, and the third-party APIs that behave differently under load than they do in a mock.
That's the gap a unit-test-and-review pipeline can't close on its own. Code can look correct in isolation and still fail the moment it meets production load, real data shapes, or an integration nobody wrote a test for. Closing that gap is what continuous verification is for.
Continuous verification means this kind of testing runs permanently, in the background, on every change, without manual effort to set it up or keep it working. It means tests that are generated automatically and executed on every PR. Tests that auto-heal when they break, without anyone filing a ticket or scheduling a sprint to fix the suite.
It's not a tool you use when you remember to. It's infrastructure. Like CI/CD, it runs whether you're thinking about it or not.
The goal is a permanent, reliable signal that your product works against real conditions. It needs to stay accurate as the product changes and scale as your team ships faster.
The trust problem isn't only a maintenance problem
Most E2E suites lose trust for two different reasons, and they tend to get treated as one.
The first is decay. Tests are written to pass against a specific version of the UI. When the UI changes, tests that were accurate six months ago now fail intermittently for reasons that have nothing to do with the code being tested. The suite goes from green to amber to permanently red-ish, and at some point the team makes an unspoken decision to stop fully believing it.
Continuous verification solves decay by making maintenance automatic. When a UI change breaks a selector, an auto-healing agent identifies the breakage, updates the test, and opens a PR for review. Around 70% of failures resolve without human intervention. The suite stays accurate.
The second reason is different, and accuracy alone doesn't fix it: a suite can be perfectly in sync with the UI and still not be testing the conditions that matter. Real user flows, production-scale data, third-party APIs under real load—none of that is guaranteed just because every selector resolves correctly. An accurate suite that's only ever run against a thin staging environment can be green for the wrong reasons.
The gear gets checked before every climb, not once at the start of the season—and it’s checked against the actual route, not a practice wall.
Creating engineering behavioral change
A lot can shift when your E2E suite is both accurate and testing the right things.
Refactors that used to feel risky become routine. UI changes go out without the usual anxiety about what they might have broken downstream. New features get shipped with confidence because they've been tested against real flows and real data, not just reviewed against the diff. The team commits to hard moves because they trust what catches them if something goes wrong. That trust isn't misplaced because the suite is actually looking at the conditions that matter.
This is what continuous verification enables that a static test suite can't. It enables teams to ship faster because they can trust what the tests tell them.
From CI/CD to continuous verification
Engineering teams that can move quickly in an AI-enabled world aren't just using AI to write code. They're using it to solve the second half of the problem: making sure that code works, against real conditions, before it ships.
CI/CD gave teams the infrastructure to deliver continuously. Continuous verification gives them the infrastructure to prove, continuously, that what they're delivering actually works. Together, they close the loop from commit to confident deployment, without the manual effort that used to sit in between.
If hesitation is the symptom—whether it's caused by a suite you've stopped trusting, or one that's accurate but isn't testing the conditions that matter—that's not a testing problem in the old sense. It's a continuous verification problem. And it has a solution that doesn't require a sprint, a QA hire, or another round of selector archaeology.
Checksum's E2E Agent automatically generates, runs, and auto-heals your test suite so your team ships with confidence and speed. See how it works →
FAQs
What is continuous verification, and how is it different from CI/CD?
Whereas CI/CD automates how code gets delivered (from commit to running environment), continuous verification automates how it gets proven to work. Tests are generated automatically, run on every change against real flows and data, and healed automatically when the UI changes. The goal is a permanent, reliable signal that your product works under real conditions, without the manual effort of test upkeep.
Why do E2E test suites become unreliable over time?
E2E tests are written against a specific version of the UI. As the product evolves, selectors change (responsible for 32% of failures) and user flows change (27% of failures). Since static tests don't update automatically, the result is a suite that fails intermittently for reasons unrelated to actual bugs. This erodes trust until teams stop fully believing what the suite tells them. That’s before you factor in whether the suite was testing the right things in the first place.
Can a green E2E suite still miss real problems?
Yes, if it isn't testing against production-like conditions. A suite can be perfectly up to date with the UI and still pass code that fails under real load, against real third-party APIs, or with data shapes it was never tested against. Accuracy and coverage are different problems. Continuous verification is designed to address both.
What does "auto-healing" mean in the context of E2E testing?
An auto-healing agent monitors your test suite continuously to:
- Identify failures caused by UI changes rather than real bugs
- Update affected selectors or flows
- Open pull requests for review
With Checksum's E2E Agent, around 70% of failures resolve without any engineer intervention. The suite stays accurate without maintenance sprints or selector archaeology.
