A few weeks back I wrote about why "point an LLM at your OpenAPI spec and get a test suite" produces tests that pass and catch nothing. The short version: generating a test that passes and generating a test that finds a bug are two different goals, and they conflict. A model optimizing for green will quietly shrink a five-endpoint journey down to two, turn a mutation into a read, and call it a happy path. You never see the retreat. You see a clean folder of passing tests, right up until a broken release sails through it.
Today we are shipping the thing that closes that gap. The Checksum API Agent is live.
Capability journeys, not endpoint pings
A test that calls a list endpoint and checks that the list came back is not a test. It is a ping. It passes whether the feature works or is completely broken.
The API Agent does not generate pings. It generates journeys: create a resource, capture the ID the API returns, read it back, update it, confirm the update held, and check that the side effects landed where they should. Values produced in one step flow into the next, so the suite exercises the behavior your product actually depends on instead of confirming that endpoints return 200. That is the difference between a suite that proves your API responds and a suite that proves your product works.
It ships the tests it cannot pass
This is the part I care about most, because it is the part everyone else hides.
When a generated test cannot be made to pass cleanly, because a credential is missing, or a second tenant was never provisioned, or the environment has a quirk, the easy move is to drop it and say nothing. That is how you get a demo where everything is green. It is also how untested surface area piles up until it surfaces in production.
The API Agent does the opposite. A test it cannot pass ships as a declared, skipped test, with a plain-English reason and a specific next step, something like "add a second tenant key to your environment." Failures you can see are failures you can fix. Failures that get filtered become rot you find out about later.
You can read all of it off a coverage matrix: every endpoint against every failure mode, auth mechanism, and cross-cutting concern like pagination or concurrency. Each cell is either covered by a real test or explicitly not covered for a stated reason. "Why isn't cross-tenant access tested on the admin endpoints" is a question you can ask and we can answer. "We generated 437 tests" is not.
Auth as its own dimension
Roughly a third of production API failures trace back to authentication or authorization. A spec tells the model the required shape, send a bearer token, and nothing about the shapes of failure: missing, malformed, expired, wrong scope, cross-tenant, or the worst one, a 401 that still leaks the data the caller was never allowed to read.
So the agent builds a dedicated set of failure tests for every auth mechanism in your spec, and asserts on the response body, not just the status code. The status code is the easy half. The leak is the bug.
It keeps the suite alive
APIs change. Endpoints get added, reshaped, deprecated. When that happens, the agent identifies the journeys the change affects and proposes updates through a reviewable session, so you approve a diff instead of digging through a wall of red trying to work out which failures are real. Across our platform, teams run with 82% lower test failure rates than they get maintaining suites by hand.
Execution happens in a deterministic runner, not inside an agent session where model variance can make the same test pass and fail on back-to-back runs. Flaky tests train teams to ignore failures, so we keep the reasoning and the running apart. The model only shows up when something genuinely breaks and needs explaining.
What you can do today
Point the API Agent at a spec or captured traffic; no repository connection required. Within days, it maps your API surface and builds meaningful, journey-based coverage even across large APIs. Tests are written as readable PyTest and maintained in a dedicated Checksum repository for your organization, without relying on a proprietary test format.
If you already run our end-to-end agent, this is the same approach extended down to the API layer, so end-to-end and API coverage sit in one place, maintained the same way.
The one thing to take away
The model underneath is the same one everyone else can call. That was never the hard part. The hard part is the planner that thinks in journeys, the review gate that fails a run when the implementer drops what the planner proposed, the skipped test that shows up with a reason instead of disappearing, the matrix that turns "not tested" into a visible decision. That is the product. The model is a good collaborator inside it.
Point any API tester at your spec and ask it one question: show me a test you proposed and could not pass, and tell me where it went. If the answer is "we filter those," you are looking at a clean demo. If the answer is "it ships as a skipped test with a reason and a next step," you are looking at something built to catch bugs.
That is what we shipped. Go try it.
