AI-enabled engineering teams are shipping faster than ever—but something else is happening too. Software is becoming less stable, not because engineers are writing worse code, but because the combination of AI code and its volume is making verification harder at precisely the moment it’s needed most.
Key takeaways
- More features means more bugs because the math changed, not the code quality. When teams ship one feature a week at a 10% bug rate, that's one manageable bug every ten weeks. AI lets teams ship ten features a week at the same bug rate, and customers experience that as a broken product rather than a percentage.
- Teams spend significantly each year to resolve test failures. Checksum’s analysis of 1M+ production test runs revealed that a 100-test suite costs over $500,000 a year to maintain; that rises to almost $4.5 million a year for a 500-test suite.
- When production bugs come from system interactions and not code logic, AI can't see them. Enterprise bugs commonly trace back to third-party services, deployments, and infrastructure behaving unexpectedly rather than logic errors in a single codebase. AI models can't catch these because they read code without knowing the runtime context, past incidents, or failure modes senior engineers have learned the hard way.
AI development is creating a QA headache
Speaking to AlphaSignal, Checksum CEO and Co-Founder Gal Vered described the problems he sees engineering teams face in the shift to AI-assisted development.
- Context is being lost: “Writing is thinking. When you’re not writing the code, you’re thinking less. The context is often in the senior developer's head, they’re thinking ‘I know about this outage, so next time it happens I'll know what to do’. When you’re not writing the code, you’re more likely to miss the relevance of the context for a specific change you’re making.”
- The bug rate math has changed. “You ship more features with AI. If you used to ship a feature a week, and every feature had a 10% chance of being buggy, then every ten weeks you'd have a bug and that’s manageable. But if you ship 10 features a week, now you have 10 bugs in production. Customers don't do percentages. They encounter 10 bugs in one day and think your app is buggy. The difference is velocity.”
These compounding problems reveal the hidden cost of AI-assisted development that most teams haven't fully reckoned with.
Checksum's 2026 QA Benchmark Report, which was drawn from over one million production test runs, puts numbers on this. Teams see a median of 14.8 failures per 100 test runs, which each take an average of 1.3 hours to fix. In a small test suite of 100 tests, that adds up to $519,480 annually; for a test suite of 500 tests the cost is $4,329,000.
When AI code meets enterprise systems
Loss of context and shipping velocity are problems that become acute in complex enterprise systems. AI is capable of writing syntactically correct code. But enterprise software environments are systems of systems where most bugs have nothing to do with logic errors in a single codebase.
Vered explained, "Most bugs are not 'you did a plus b instead of a minus b.' It's usually dependencies on third-party services that aren't behaving correctly. Deployments, services, servers all working together. Once your software is dependent on a thousand different things, and one of those things doesn't work as expected, things start to break."
To demonstrate what that looks like in practice, Vered used X (Twitter) as an example. "In essence, Twitter is an extremely simple product. You can probably build it in a weekend. But Twitter itself is extremely complex because it's deployed on thousands of machines across the world, all geared to give you a real-time image of what's going on in milliseconds."
AI doesn't understand any of that context. It reads your code but it doesn't know the edge cases, the data shape, the third-party integration behaviour, or the failure modes that senior engineers carry in their heads from past incidents.
A real-world example: when three things go wrong at once
Checksum's own production incident illustrates the kind of multi-dependency failure many enterprise teams will recognise.
The team made changes to their agents, tested thoroughly in local environments, and deployed to production. What happened next was a confluence of three independent, low-probability failures: a sandbox provider with an undocumented two-minute request timeout; an Anthropic outage causing response times to stretch from 30 seconds to three minutes; and an upstream bug in a library that caused timeouts when streamed responses exceeded a threshold.
"All three happened at the same time," Vered explained. "That caused a bug to one percent of our agent sessions."
No LLM could have predicted this. Vered continued, "They have no idea about the sandbox environment and the behaviour. They have no idea about the library bug and whether it's relevant. And they have no idea that Anthropic is going to take three minutes to stream a response that usually takes thirty seconds."
But those failure patterns are knowable if you have a system that collects production run data over time, identifies patterns, and generates tests from them. "These things are knowable. They're just not knowable by LLMs."
Building a world model for verification
The core idea behind Checksum's approach is that rather than testing code in isolation, the best way to verify is to build a model of the production environment—its APIs, databases, third-party dependencies, real user behaviour—and run every change against that simulation before it ships.
"The fastest way to ship code to your customers is to ship it to production as fast as possible and see what breaks," said Vered. "That's also the fastest way to lose your customers' trust. We think about production environments as a world, right? We’re building this world model that essentially you ship your software to and we simulate everything else. That gives you a concrete signal of everything that’s working and not working in your app.”
He continued, “If I have a PR and I’m not sure if it’s working or not, I can figure it out by shipping to production and then waiting ten minutes to see if any customer calls to shout at me. Basically, we’re allowing you to ship to production without actually shipping to production because it’s a simulation.”
In practice, this means Checksum runs on every PR and every deploy, auto-healing tests when the product changes and surfacing real regressions with a reliable signal.
ClearPoint Strategy, an enterprise SaaS company, used Checksum to build 250+ end-to-end tests in under a month. The result: catching six critical bugs per week and saving $500,000 annually.
Checksum sits at the infrastructure layer, similar to how CI/CD automated software delivery without replacing engineers. QA engineers retain ownership of test strategy, coverage decisions, and failure triage. What Checksum eliminates is the maintenance burden: the hours spent debugging broken selectors, updating tests after UI changes, and manually triaging failures that turn out not to be real regressions.
No new tooling is required as Checksum runs on every PR in your existing CI pipeline. Checksum is SOC 2 and ISO 27001 certified, with all test code staying in customer infrastructure — a requirement most enterprise security teams will need before any external testing layer touches production workflows.
Quality is becoming a competitive differentiator
The teams that treat verification as an afterthought are going to feel it. "Quality becomes the bottleneck," Vered explained. "Think about how you ship high quality code. How do you make sure the code you ship to your customers is actually working? Reliability becomes a differentiator."
Velocity without verification just means broken code ships faster. Continuous verification is how you keep both. See how it works →
FAQs
Why is software becoming less reliable even as AI helps teams ship faster?
Software is becoming less stable because shipping velocity has outpaced verification. Teams that once shipped one feature a week now ship ten; the same per-feature bug rate produces far more bugs in production. Engineers are writing less of the code themselves, which means they have less of the operational context to make quick fixes.
What causes bugs in AI-assisted software?
Many bugs come from dependencies rather than logic errors, i.e. third-party services, deployments, servers, and integrations that don't behave as expected once an app depends on enough of them. AI models can write syntactically correct code, but when they lack visibility into edge cases, data shapes, third-party integration behavior, or past production incidents, they can't predict multi-system failures.
What is continuous verification and how does Checksum's approach work?
Continuous verification means running real end-to-end tests against your application on every PR and every deploy, not testing code in isolation or waiting to see what breaks in production. Checksum's E2E Agent automatically detects critical user journeys, generates production-ready Playwright tests, and delivers them as pull requests to a test repository you own. Those tests then run in your existing CI pipeline on every change.
When tests break, Checksum doesn't just flag the failure— it fixes the code and opens a pull request with the healed tests for your team to review.
