Within a month of activation, the team at Clearpoint Strategy built a suite of over 250 tests that runs seamlessly in their existing CI/CD. The time investment was only a few hours per week – and the test suite is completely maintained by AI.
Background
Clearpoint Strategy is a leader in strategic planning and business reporting software – with large, enterprise customers like AT&T, Kimberly-Clark, and the New York Power Authority.
These users rely on Clearpoint’s software – primarily accessed through a web application – for data collection, custom reporting, automation, and real time collaboration. The software is designed for power users and is very feature-rich, which made automated testing a significant challenge.
Problem
When Clearpoint Strategy engaged with Checksum, their customers were experiencing bugs frequently. Their existing test suite needed better coverage (to cover the many potential user paths) and better maintenance (as tests would frequently break, and therefore not detect bugs), despite significant efforts by the engineering team to write and maintain tests.
Understandably, the engineering team did not trust their existing test suite to catch bugs before deployment.
In an effort to supplement the test suite, Clearpoint used manual QA – which was often done by the engineering team and product team – to check the health of core flows. This worked to catch _some _bugs, but slowed down the engineering team significantly, since each new release involved extensive manual testing.
Engineers were spending way too much time writing and maintaining tests, new releases were constantly delayed, and avoidable bugs were still impacting clients in the production environment.
Results
Checksum transformed Clearpoint Strategy’s bug detection, test coverage, test maintenance, and engineering velocity – all within a month.
With the investment of a few hours per week on Checksum, Clearpoint Strategy built a suite of over 250 tests that covered all the “happy path” user flows and a significant number of edge cases. These tests run seamlessly in the existing Clearpoint CI/CD environment on a nightly basis.
The maintenance burden essentially disappeared. In the past, small changes to the app (for instance, altering the inner text of a form) would cause a test to fail, even if the meaning and user flow was essentially unchanged. Now, Checksum auto-healed and maintained these tests, allowing engineers to move faster.
On average, 6 critical bugs were discovered and prevented weekly. This resulted in higher customer satisfaction, significantly reduced tech debt, and a relieved customer support team.
Quality of life also improved for engineers. Significantly less time was spent writing and maintaining tests, as well as firefighting, which meant more developer time for new features. Engineers were empowered to take bolder bets because, for the first time, they actually trustedt their automated test suite.
Solution
You’re probably wondering: did Checksum really manage to do all that in less than a month?
In actuality, it took years
- of training specialized models,
- building the intelligent orchestration layer that deploys the right model for the right task, and
- intimately understanding the QA problem as career technologists and entrepreneurs.
Clearpoint’s results are representative of what all Checksum customers experience. How does it work?
Automatic Test Detection
Checksum needs 1-3 days to “understand” your application. Put slightly more technically, we need to fine-tune our AI to understand the unique rhythm and flow of your application.
We do this by generating synthetic data, and in the case of clients like Clearpoint, Checksum also detects edge cases by training our AI on (anonymized) user data.
After this brief training period, Checksum detects tests, identifies test steps, names the tests, categorizes the tests, and queues them up for review by a knowledgeable human (usually an engineer) with strong knowledge of the application.
1-click Test Generation
It only takes a single click to generate a test with Checksum.
A human still needs to review the test – but even so, this process only takes minutes. On average, it would take hours to write a single end-to-end test.
We obsess over test quality, and our tests don’t just mindlessly describe user flows; they also validate that frontend elements reflect the desired state of the application at every step of the way. This is, once again, something that’s slow and painstaking for a engineer to do – time not spent building new features or capabilities.
Test Maintenance with AI
Maintaining tests is an ongoing burden for virtually every organization with a test suite. Here’s a common case: the author of a test describes a locator, then something trivial about that locator (e.g. its position on the page, its inner text, or another identifying element) changes, and the test breaks.
Fixing a broken test is usually a multi-step process, with DMs flying back and forth, and engineers trying to remember the conditions before and after the test broke.
But with Checksum, tests can typically be healed in a matter of seconds: essentially, in real-time.
A huge benefit is that engineers don’t get distracted – instead, they get a neat report explaining why the test broke and how it was fixed. And most importantly, this ensures that tests are always fresh and accurate, and never flaky.
Conclusion
Checksum is a game-changer… It saves me so much time writing tests so I can deploy my engineering resources to building tomorrow’s technology today - not fixing yesterday’s release over and over again.
— Ted Jackson, Co-founder, ClearPoint Strategy
Clearpoint Strategy’s annual savings on testing was estimated at $500,000 per year. This savings was comprised of:
- Significant reduction in the time engineers spent writing tests
- Elimination of most manual QA processes
- Decreased time spent fixing bugs due to early detection and accurate tests
- Enhanced customer satisfaction and retention, as a result of fewer bugs in production
Clearpoint Strategy also reported a number of qualitative benefits:
- Improved engineering velocity, since engineers get immediate feedback from automated tests
- Higher willingness to innovate, because engineers have confidence that issues will be caught early
- Better quality of life for engineers, primarily due to less time writing tests and maintaining broken tests
Interested?
If these problems sound familiar, we are available to chat. Book a demo with us at checksum.ai and reference this case study. We would love to help.
Gal Vered
Gal Vered is a Co-Founder at Checksum where they use AI to generate end-to-end Cypress and Playwright tests, so that dev teams know that their product is thoroughly tested and shipped bug free, without the need to manually write or maintain tests.
In his role, Gal helped many teams build their testing infrastructure, solve typical (and not so typical) testing challenges and deploy AI to move fast and ship high quality software.