Checksum’s AI agent writes production-ready Playwright tests for you and delivers them as a pull request. You can start it from your own tooling with the REST API, from a pull request comment, from your coding agent, from Slack, from the web app, or by recording a walkthrough with the Chrome extension. Wherever you start it, the agent plans, writes, reviews, and runs the tests before handing them over.
Every trigger starts an agent session, which runs this pipeline:
1 · PlanDeep mode only: interview + plan
→
2 · ImplementStory .checksum.md + test .checksum.spec.ts
→
3 · ReviewSelf-review for quality and correctness
→
4 · VerifyActually runs the tests against your environment
→
5 · DeliverPull request to your tests repo
The test generation pipeline, from trigger to pull request.
Checksum’s world model is built from real user-behavior patterns, and the agent reaches ~97% test accuracy. Generated tests include data setup and cleanup, selectors grounded in your source code, and a production-ready structure. Unlike local coding agents, Checksum runs in the cloud 24/7, generating and verifying tests autonomously at scale. You can start many sessions in parallel and come back to finished PRs.
Detect first for broad coverageTo cover a whole feature area, run detection first. You can then review and prioritize the proposed flows before any code is written. For one quick, targeted test, generate straight from a manual flow or a PR.
Pick where you want to start. Every option starts the same agent session and pipeline. They differ in what you give the agent and whether a PR opens automatically.
Use the API when a release bot, an internal tool, or a custom CI step should kick off test generation. You tell Checksum which pull request to cover. It replies right away with a batchId, works in the background, and opens a pull request in your tests repository when the tests are ready. Progress also appears as a single comment on the source PR, posted by Checksum, so your CI step doesn’t need to comment. To have a coding agent build this CI step for you, use CI Setup Prompts → Prompt 2.
The batch moves from pending to in_progress, then ends as completed or failed. It has the same shape as a healing batch, without the healing-only fields testRunId and sessions[].checksumTestId.
Example: a script that generates tests for a PR and waits for the result
Needs curl, jq, and the variables CHECKSUM_API_KEY, PR_NUMBER, REPO (owner/repo), and HEAD_BRANCH.
#!/usr/bin/env bashset -euo pipefailBASE="https://api.checksum.ai/public-api/v1"# 1. Start generation for the PRBATCH=$(curl -sf -X POST "$BASE/auto-generate" \ -H "Authorization: Bearer $CHECKSUM_API_KEY" -H "Content-Type: application/json" \ -d "$(jq -n --argjson pr "$PR_NUMBER" --arg repo "$REPO" --arg br "$HEAD_BRANCH" \ '{prNumber: $pr, repoName: $repo, branch: $br}')" | jq -r .batchId)echo "Started batch: $BATCH"# 2. Poll until every session is finisheduntil [[ "$(curl -sf "$BASE/auto-generate/batch/$BATCH" \ -H "Authorization: Bearer $CHECKSUM_API_KEY" | tee /tmp/batch.json | jq -r .allTerminal)" == "true" ]]; do sleep 30done# 3. Print each session's result and PR linkjq -r '.sessions[] | "\(.status) \(.prUrl // "no PR")"' /tmp/batch.json
POST https://api.checksum.ai/public-api/v1/auto-generate
Starts a generation agent session for a pull request. Opens a PR in the tests repository when finished. Progress is posted as one sticky comment on the source PR. Generates from a PR only: saved test flows require the web app, and description-only runs require MCP.
Header
Value
Authorization
Bearer $CHECKSUM_API_KEY
Content-Type
application/json
Body field
Type
Required
Description
prNumber
number
Yes
Pull request to generate tests for. Used as agent context and for the sticky progress comment.
repoName
string
Yes
Repository that hosts the PR, <owner>/<repo>. Must be connected in Git Integration.
branch
string
Yes
The PR’s head branch. The agent diffs against it.
metadata
object
No
Free-form key/value pairs forwarded to the agent as context.
envOverrides
object (string → string)
No
Per-session environment variables layered on the project environment. Keys named CI or starting with CHECKSUM_ are rejected with 400.
Response field (202)
Type
Description
batchId
string
Pass to GET /auto-generate/batch/{batchId}
sessionId
string
The generating agent session
links.session
string (URL)
Live session in the web app
Error
Cause
Fix
400
Reserved key in envOverrides (CI, CHECKSUM_*)
Pass only application variables, e.g. BASE_URL
401
Missing or invalid API key
Send Authorization: Bearer $CHECKSUM_API_KEY. Verify with GET /me.
Next: poll GET /public-api/v1/auto-generate/batch/{batchId}.
GET https://api.checksum.ai/public-api/v1/auto-generate/batch/{batchId}
Path parameter
Type
Description
batchId
string
From POST /auto-generate
Response field (200)
Type
Description
batchId
string
The batch being reported
status
enum
pending (not started), in_progress (at least one session running), completed (all sessions succeeded), failed (one or more sessions failed)
totalSessions
number
Sessions in the batch
completedCount
number
Sessions that finished successfully
failedCount
number
Sessions that failed
allTerminal
boolean
true once every session has finished. Stop polling.
sessions[].sessionId
string
Agent session ID
sessions[].status
string
Session state, e.g. running, completed
sessions[].prUrl
string | null
Tests-repo PR URL once opened, otherwise null
Unlike a healing batch, a generation batch has no testRunId and no sessions[].checksumTestId.Polling rule: every 10–30 seconds until allTerminal is true. On failed, open links.session from the start response.
If the Checksum GitHub App is installed on a repository, anyone with write access can ask for tests right on the pull request. Comment:
/checksum generate
Add instructions on the same line to steer the agent:
/checksum generate focus on the new SSO login flow and error states
Checksum reacts with 👀 while it starts, 🚀 once the session is running, or 👎 if it can’t start (usually missing permissions or the app isn’t installed). Progress goes in one sticky comment that is updated in place, so the thread stays tidy. The agent always works from the PR’s head branch, so feature-branch PRs get tests for the right code.
Automation welcomeBots and automation accounts work just like teammates. PRs opened by Dependabot, Renovate, or your own release bot can get generated tests. Any bot account with write access can post /checksum generate, or call POST /auto-generate, to start the agent whenever you need it.
Enabled by Checksum Checksum can turn on automatic generation for every non-draft PR. It works the same as /checksum generate, without anyone commenting. Draft PRs are skipped. Ask your Checksum team to enable it for your project.
Reference for AI: /checksum generate
Item
Value
Syntax
/checksum generate [free-text instructions on the same line] as a PR comment
Requirement
Checksum GitHub App installed on the repository (Git Integration)
Requirement
Commenter (person or bot account) has write access to the repository
Target code
Always the PR head branch. Checksum fetches the head ref from GitHub.
Progress
One sticky PR comment, edited in place
Output
A PR in the tests repository
Automatic variant
Auto-generate on PR open: non-draft PRs only, enabled per project by Checksum
Reaction
Meaning
👀
Command received, session starting
🚀
Session started successfully
👎
Command rejected (missing permissions, or the app isn’t installed)
With the Checksum MCP server connected, you can ask Claude Code, Cursor, or VS Code for tests in plain English:
Generate Checksum tests for PR 142 in acme/web on branch feature/checkout.
Give your agent a pull request (number, repository, and branch) and Checksum opens a PR with the tests. If you ask twice for the same PR, it joins the run already in progress. Or describe a flow, such as “the login flow with an invalid password”, and the tests stay in the session for you to review without a PR. Or target specs you already have, such as a collection or the flows detection just found, and Checksum generates tests for exactly those. For a description or targeted run, ask for a PR up front, or once you’ve reviewed the session. Your agent can also point a run at a preview deployment, switch on Deep mode, report progress, and pass along follow-up instructions while the run is going.
Push firstCloud generation checks out your repo from your Git provider, so it only sees pushed commits. Commit and push before you ask. The Claude Code plugin does this for you.
Reference for AI: checksum_test_generate
MCP server: https://api.checksum.ai/public-api/mcp (Streamable HTTP). Tool: checksum_test_generate. Returns a batchId (a sessionId for a targeted run). It changes things: it starts a billable cloud run and can open a PR.
Input
Required
Behavior
PR number + repository + branch
All three together, or one of the options below
Generates for the PR and opens a PR with the tests. A repeat request for the same PR joins the run in progress.
extraInstructions
Instead of a PR
Describes the flow to cover. Tests stay in the session unless autoCreatePR is true.
userStoryIds and/or collectionNames
Instead of a PR
Targeted generation for exactly those specs. Can’t be combined with a PR. Returns a sessionId.
codeRepoNames, repoBranches
No
Which connected code repositories (and branches) a targeted run reads
autoCreatePR
No
true opens a PR when a description or targeted run finishes
deepMode
No
true for Deep mode: the session pauses for plan approval (checksum_session_approve). Default: Standard
envOverrides
No
Points the run at another environment, e.g. a preview deployment
Related tool
Use
checksum_session_status
Progress, the agent’s latest messages, changed files, and prUrl. Poll until the run finishes.
Once the Checksum Slack app is installed (Notifications & Slack), mention it in a thread in a channel it’s been invited to:
@checksum add E2E coverage for the new checkout discount flow
Checksum reads the whole thread, not just your message, and uses it as the agent’s instructions. The bot replies with a link to follow the session. If the thread includes pull-request context, the tests arrive as a PR; otherwise they wait in the session for you to review. Mention the bot in a thread rather than a top-level message, and be specific: “Add tests for guest checkout with an expired card” works much better than “add tests.”
Beta The Checksum Capture Chrome extension turns one walkthrough of your app into a batch of tests. Click through the flow you want covered, and talk through it if you like. Checksum hands the recording to the agent, which starts a generation session from it.It’s not a 1:1 recorder like Playwright’s codegen, which turns each click into one line of one test. Checksum treats your recording as training material for the agent. The agent learns the flow, the screens, and what you care about, then writes the tests it implies. When you split the recording into flows, one recording can produce 10, 20, 30 or more tests, depending on what you ask for and how deep and broad the flow is.
Open the extension and paste your project’s API key, which you’ll find on the Settings page in the Checksum web app. The key tells Checksum which project the recording and the generated tests belong to.
3
Go to your app
In the same browser, go straight to the page of your app where the flow starts.
4
Set up the session
Name the session, then choose how the agent should turn your recording into tests:
Exactly what I recorded (one test): one test that follows your walkthrough.
Split into flows (one test per flow): the agent breaks the recording, and your narration, into separate flows and writes a test for each. This is how one recording becomes a batch of tests.
To talk while you record, turn on Record microphone narration. Chrome asks for microphone access the first time.
5
Record and click through
Start recording and use the app the way a user would. The extension captures your actions from the page’s DOM as you go. If you turned on narration, talk the agent through what matters (see below).
6
Stop, and let the agent generate
When you stop, the recording is sent to Checksum and an agent session starts generating tests from it.
7
Review and open a PR
The generated tests appear on the session’s page in the Checksum web app. Review them there, then open a pull request to your tests repository from the session when you’re ready.
Narration is the best way to tell the agent things it can’t learn from clicks alone. Use it to:
Call out edge cases: “An expired card should show an error here, not a spinner.”
State assertions: “After saving, the total at the top should update and the badge should say Draft.”
Share app knowledge: requirements, business rules, or behavior that a user wouldn’t notice, such as “Only admins see this tab” or “This field is required for EU customers.”
Ask for variations: “There are four plan types and two billing periods. Capture every combination.” The agent expands the one path you showed into a test for each variation.
Say how much you wantWith Split into flows, the size of the batch follows your ask. One quiet walkthrough gives a focused set of tests. Narrating variations, roles, and error cases gives the agent the breadth to write many more.
Reference for AI: Checksum Capture Chrome extension
Connected to a project with the project’s API key, found on the Settings page of the web app (API keys)
Session setup
Session name · output mode: Exactly what I recorded (one test) or Split into flows (one test per flow) · optional Record microphone narration
Captures
User actions from the DOM while recording, plus optional microphone narration
Output
Starts a generation agent session that uses the recording as context. “Exactly what I recorded” gives one test. “Split into flows” gives a batch, one test per flow (often 10–30+).
Delivery
Tests stay on the agent session page in the web app. No PR is opened automatically: open one from the session.
Not
A 1:1 action-to-code recorder (unlike Playwright codegen)
Narration uses
Edge cases, assertions, requirements and app knowledge, and requests to cover variations or combinations
Select a collection and find the test flow(s) you want. You can select several flows in one collection and generate them together.
3
Click Generate
Choose Standard or Deep mode (see Deep vs Standard). Standard is the default.
4
Watch or walk away
An agent session starts. Standard sessions finish without any input from you. Deep sessions wait for you to approve the plan first. The flow’s status updates as the agent works.
Test Generation: a collection's flows with per-flow Generate actions.
For each flow, the agent writes two files to a new branch in your tests repository:
File
Purpose
*.checksum.md
The story: a human-readable spec with frontmatter, data setup, data cleanup, and numbered steps with verifications. It is the source of truth for what the test does.
*.checksum.spec.ts
The test: Playwright TypeScript using Checksum fixtures. Pure Playwright underneath, so you can run it anywhere.
The branch is named like ChecksumAI-generated-test-20260315143022, and the PR description explains the flow being covered. See Story & Test Format for both file formats.
Each test flow in the web app shows where its generation stands:
Status
Meaning
Idle
No generation in progress for this flow
Queued
Generation is queued and will start shortly
Running
The agent is planning, implementing, or reviewing
Verifying
The agent is executing the tests to prove they pass
Completed
Tests are generated and the PR is ready
Failed
Generation hit an error. Open the session for details.
Aborted
Generation was cancelled
Edited
The generated tests were manually edited afterward
PR Merged
You merged the generated-tests PR
PR Closed
You closed the PR without merging
API and MCP callers see the equivalent state on the batch (pending → in_progress → completed / failed) and on each session. See Agent Sessions for the session lifecycle.
Generated tests arrive as a normal pull request. Checksum’s agent behaves like another developer on your team. By default, nothing reaches your main branch without your review. If you turn on auto-merge, Checksum’s PRs merge without a human review and the steps below are optional.
1
Read the PR
Review the story file first, since it’s the fastest way to confirm the agent understood the flow. Then review the test code.
2
Run it yourself (optional)
Check out the branch and run npx checksumai test -g "<test title>". See Running Tests.
3
Request changes
Edit the files directly, or re-trigger generation with more specific instructions. A flow you edit shows the Edited status.
4
Merge
Once merged, the tests join your suite and are covered by auto-maintenance.
envOverrides only accepts your application’s own variables. Names reserved by Checksum, including CI and anything starting with CHECKSUM_, are rejected.
Tests were generated but no PR was opened
Checksum opens a PR only when it knows the pull request number, repository, and branch. Slack mentions without PR context, MCP runs from a description or targeted specs (without autoCreatePR), and Chrome extension recordings leave the tests in the session. Open the session and create the PR from there (PR Eligible state), ask your coding agent to call checksum_session_create_pr, or re-run against a PR.
The tests cover old code
Cloud sessions clone from your Git provider. Push your commits, then trigger generation again.
/checksum generate got a 👎 reaction
The commenter needs write access, and the Checksum GitHub App must be installed on that repository. Check Settings → Git Integration for an Active status.
The session failed with "application unreachable"
The agent couldn’t load your environment URL. Check that the URL in Environments is correct and up. If the environment is behind a VPN or firewall, confirm Checksum’s IP addresses are allowlisted (see Security & Access).
