There's a version of QA that looks great in a sprint review and does nothing in production.
It lives in dashboards, tools someone set up two years ago, or perhaps no-code testing platforms. Those ones that require you to log into a separate app before you know whether your code is safe to ship. On the face of it, the tests exist and they’re organized.
But nobody looks at that tab when shipping velocity goes up.
Key takeaways
- Shadow QA gives false confidence. Tools that rely on engineer or QA team maintenance create the illusion of coverage while drifting further from production with every sprint.
- Context-switching kills test coverage. When tests require manual intervention to run, they get skipped as shipping velocity rises. This makes test infrastructure that depends on human discipline structurally incompatible with AI-enabled development.
- “Does this tool work when nobody is paying attention to it?” Engineering leaders should be asking their teams this question and evaluating tools that generate Playwright tests as PRs and auto-heal when the product changes.
The impact of shadow QA
Fast-moving engineering teams will be familiar with a scenario like this.
A senior engineer merges a PR on Thursday afternoon. The change touches an auth flow that has corresponding tests in a separate testing platform—carefully organized by a QA engineer who left last quarter. Someone needs to log in, update the tests to reflect the new flow, run them against staging, and verify the results before the release.
But in reality, there are four more PRs queued for review, a production incident from Tuesday that's still being monitored, and a sprint demo in two days. Nobody logs in, the tests run stale, and the release ships.
Shadow QA shows up in a few different ways:
- Test suites that live in separate platforms outside your development workflow
- No-code testing platforms with their own dashboards
- Visual regression tools that require someone to approve screenshots
- Test suites that run when someone remembers to run them.
What these tools have in common is they exist alongside your real development process but they're not integrated into it. Instead, they accumulate coverage that nobody maintains—a test state that drifts further from production with every sprint.
While these tools aren't bad at testing per se, the problem is structural. When they live outside your development workflow, they're too easy to overlook.
Why context-switching doesn’t work
When you're moving slowly, running tests manually is more manageable. But the velocity of AI-enabled teams makes this impossible. And it’s not about discipline—context-switching is a design problem. Gal Vered, Co-Founder and CEO at Checksum, explains that when you have to trigger and maintain tests from a separate place, they get ignored:
“Especially when you're shipping fast. Nobody is going to context-switch into another app to check if things are okay while an agent is writing features all day. If a test doesn't run where your code runs, it may as well not exist."
The QA perspective
Senior engineers know that QA tools get ignored when shipping velocity goes up. But QA managers have watched this unfold from a different angle.
They built the collection, documented the process, and onboarded engineering colleagues to keep it updated. But then the coverage gap widened as features shipped faster than the collection could be maintained. The tool was fine. The problem was that maintaining it was a manual job that competed with every other priority on the board.
When a QA team gets labeled a bottleneck, it's usually not because the team is slow. It's because the testing infrastructure requires human intervention when humans are most stretched. The solution isn't to work harder. It's to stop depending on a tool that requires manual effort to stay relevant.
A compounding problem
Here's what makes shadow QA worse than having no tests at all: it creates the impression of coverage without the substance of it.
A team that knows it has no API tests will be cautious about API changes. But a team that has a test suite with 300 tests feels covered, even when the collection hasn't been touched in six months.
Stale tests don't fail loudly. They pass against the old state, or nobody runs them, or the results show green because the collection is testing a flow that no longer exists. The gap between what your tests say and what your production system does widens invisibly.
The teams that have solved this have moved to testing infrastructure that's structurally integrated with the work—infrastructure that runs automatically, heals automatically, and fails visibly when something is actually wrong.
Tests that don't need anyone to run them
It’s time for a different QA model: Tests that live in your repository, run automatically, and maintain themselves as the product changes.
Checksum's E2E Agent maps your entire application—every screen, interaction, and user flow—and generates production-ready Playwright tests as pull requests to your repository. There’s no proprietary format and, with tests that run automatically in CI, nobody needs to remember to run them or log into anything to keep them current.
When the product changes and tests break, the agent identifies the failures, repairs them, and opens a pull request. Around 70% of failures resolve without any engineer intervention. For those that do, reviewing a PR is far lighter work than rebuilding a collection.
The question worth asking
Next time you're thinking about how your team handles E2E coverage, ask one question: does this tool work when nobody is paying attention to it?
If the answer requires human discipline or a manual maintenance process, it won't scale to where your team is heading.
Checksum's E2E Agent generates production-ready Playwright tests, delivers tests as pull requests to your repository, and auto-heals them as your product changes. Most teams have first tests running within a day. See how it works →
FAQs
Why do shadow QA tools create false confidence?
The problem with shadow QA tools is that the coverage they offer looks real. Tests exist, they're organized, and nobody has explicitly switched them off. But a test suite that requires manual effort to stay current will gradually drift from production. Stale tests don't fail loudly—they pass against the old state, or nobody runs them, or the results show green because the tests cover flows that no longer exist. A team that knows it has no coverage will be cautious. A team that thinks it has coverage won't be—and that's the more dangerous position.
What's the difference between E2E testing and API testing?
E2E tests implicitly cover API behavior for every call within a user journey. They pass real values between requests and assert on real responses in the sequences real users execute. Dedicated API tests are warranted for endpoints consumed by mobile clients, third-party integrations, or other backend services that operate outside the UI, and for testing contract behavior like error handling and schema validation. Engineering leaders should ask whether a comprehensive collection maintained across every sprint is the right tool for those specific gaps.
How does Checksum keep E2E tests up to date automatically?
Checksum's E2E Agent maps your application, generates production-ready Playwright tests that deliver as pull requests to your repository, and auto-heals them when the product changes. Around 70% of test failures resolve without any engineer intervention. When manual review is needed, it comes as a pull request—far lighter work than rebuilding a test collection from scratch.
