Running Tests - Checksum AI

At a glance

You want to	Use	Section
Run tests on your machine or your own CI runner	`npx checksumai test`	Run tests with the CLI
Start a cloud run from a GitHub workflow	The Checksum GitHub Action	GitHub Actions
Start a cloud run from any tool or CI system	The REST API	Start a cloud run
Start a run from your coding agent	The MCP `checksum_test_run` tool	Coding Agents & MCP
Find out whether a run passed	The run status endpoint	Check whether a run passed
Run fewer tests (affected, failed, by name)	CLI flags or API options	Pick which tests to run

Reference for AI: running tests at a glance

Interface	Command / endpoint	What it does	Key constraint
CLI	`npx checksumai test`	Runs tests on your machine or CI runner	Run `npx checksumai dotenv --download --api-key=$CHECKSUM_API_KEY` first
CLI	`--cksm-auto-heal`, `--cksm-affected`, `--cksm-rerun-failed`	Heal failures, run affected tests only, re-run failures	`--cksm-affected` and `--cksm-rerun-failed` can’t be combined with each other or with `-g`
GitHub Action	`checksum-ai/test-run-action@v2`	Starts a cloud run from a workflow	Wraps the REST endpoints
REST API	`POST https://api.checksum.ai/public-api/v1/execution/suite`	Runs the full suite in Checksum’s cloud	Returns `runId`
REST API	`POST https://api.checksum.ai/public-api/v1/execution/collection/{id}`	Runs one collection	—
REST API	`POST https://api.checksum.ai/public-api/v1/execution/tests`	Runs specific test IDs	Body: `testIds`
REST API	`POST https://api.checksum.ai/public-api/v2/execution/grep`	Runs tests matching a name/tag pattern	Only endpoint with `branch` + `envOverrides`
REST API	`POST https://api.checksum.ai/public-api/v1/affected-tests`	Returns test IDs affected by changed files	Max 1,000 files
REST API	`GET https://api.checksum.ai/public-api/v1/execution/status/run/{runId}`	Run status + CI verdict	Gate on `verdict`, not `status`
MCP	`checksum_test_run` (`mode`: `suite`, `collection`, `tests`, `grep`)	Starts a cloud run from a coding agent; returns `testRunId`	Optional `envOverrides` (grep), environment overrides (collection/tests), `shardCount`, `autoHeal`

Base URL: https://api.checksum.ai/public-api/v1/. Only grep (and its job-name status) uses https://api.checksum.ai/public-api/v2/.
Auth: every REST call sends Authorization: Bearer $CHECKSUM_API_KEY. Key location: Settings → Project Settings.
Gating CI: poll GET /execution/status/run/{runId} until isTerminal is true, then pass only when verdict is "pass".
Heal on failure: add an autoHeal object to any execution request, or pass --cksm-auto-heal to the CLI.
No cancel: API-triggered runs can’t be cancelled through the public API.
Sharding: shardCount 2–40. Requires checksumai 4.4.0+ on the tests branch.

Developer guide

The CLI, GitHub Actions, and the REST API

Before you run the CLI

Checksum sets up your tests repository during onboarding. To run it with the CLI, clone the tests repo and prepare it once:

npm install
npx playwright install --with-deps
npx checksumai dotenv --download --api-key=$CHECKSUM_API_KEY

dotenv --download writes a .env file with your project’s environment settings: environment URL, login URL, credentials, and custom variables. Your API key is in Settings → Project Settings (see API Keys). Checksum maintains the file’s values, so to change them, contact your Checksum team and then download it again. Any automation that starts a Checksum test run or AI generation should download the latest .env first (see Environments). To check your setup, run the built-in example test. It’s a quick check that your login works:

npx checksumai test -g "example"

Run tests with the CLI

The Checksum CLI (checksumai) runs your generated Playwright tests. It downloads your environment configuration, runs the tests, uploads the results to the dashboard, and can hand failures to auto-healing. It ships in the @checksum-ai/runtime npm package, and every command runs with npx checksumai. Because it wraps Playwright, your tests also run under plain Playwright.

Everyday commands

# Run all tests
npx checksumai test

# Run tests whose name matches a pattern
npx checksumai test -g "User Can Create Opportunity"

# Run only tests affected by your changes vs. main
npx checksumai test --cksm-affected=origin/main

# Re-run only what failed last time
npx checksumai test --cksm-rerun-failed

# Run, then auto-heal any failures (repo/branch/PR auto-detected in CI)
npx checksumai test --cksm-auto-heal

# Heal failures but don't open a PR
npx checksumai test --cksm-auto-heal --cksm-auto-heal-create-pr=false

# Point at a local build instead of the downloaded environment URL
BASE_URL=http://localhost:3000 npx checksumai test

In CI (where CI=true), results upload to the dashboard automatically. See What gets reported.

Useful flags

Most runs need no flags at all. The ones you’ll reach for:

-g "pattern" runs only the tests whose name matches.
--cksm-auto-heal sends failures to auto-healing when the run ends. In GitHub Actions and GitLab CI, Checksum detects the repository, branch, and pull request on its own, so this flag alone is usually enough.
--cksm-affected runs only the tests a change affects (below).
--cksm-rerun-failed re-runs only what didn’t pass last time (below).

The full flag list, with defaults, is in the Reference for AI block at the end of this CLI section.

Run only the tests a change affects

For pull-request checks, you usually don’t need the whole suite. --cksm-affected looks at which files changed since a base branch, asks Checksum which tests those files affect, and runs just those:

# Compare against origin/main
npx checksumai test --cksm-affected=origin/main

# In GitHub Actions / GitLab MRs, the base ref is detected automatically:
#   GitHub Actions reads GITHUB_BASE_REF
#   GitLab merge requests read CI_MERGE_REQUEST_TARGET_BRANCH_NAME
npx checksumai test --cksm-affected

# Print the changed files and affected test IDs without running anything
npx checksumai test --cksm-affected-dry-run

It needs enough git history to find where your branch split off, so use a full checkout in CI (for example fetch-depth: 0 in actions/checkout). It can’t be combined with -g or --cksm-rerun-failed.

Try it with a dry run firstRun --cksm-affected-dry-run in CI for a while to see which tests would run, before you turn the flag on for PR workflows.

Re-run only what failed

After a run with failures, you can retry just the test files that didn’t pass (failed or recovered), without running everything again:

# From the latest completed non-manual run
npx checksumai test --cksm-rerun-failed

# From a specific run (UUID)
npx checksumai test --cksm-rerun-failed=$TEST_RUN_ID

# After a CI run failed, retry only the failing specs locally
npx checksumai test --cksm-rerun-failed

If every test in that run passed, the command finishes successfully with nothing to run.

Update the CLI

Installing your tests repository’s dependencies installs the CLI (see Before you run the CLI). To update to the latest release, which you need before using sharding (minimum 4.4.0), run this and commit the result:

npm install checksumai@latest
# commit the updated package.json / lockfile to your tests branch

Other CLI commands

Command	What it does
`npx checksumai dotenv`	Downloads your environment variables into `.env` (Environment variables)
`npx checksumai show-report`	Opens the HTML report from your most recent local run in your browser (Results, Reports & Traces)
`npx checksumai init`, `tsconfig`, `eslint`, `postinstall`	Set up or refresh the `checksum/` folder. Checksum runs `init` for you during onboarding (Repository tooling).

Reference for AI: Checksum CLI (npx checksumai)

Package: checksumai (npm dependency of the tests repository). Invocation: npx checksumai <command>. Wraps Playwright; generated tests also run under plain Playwright. Update: npm install checksumai@latest, then commit package.json / lockfile to the tests branch; sharding requires 4.4.0+.Setup per checkout: npm install, npx playwright install --with-deps, npx checksumai dotenv --download --api-key=$CHECKSUM_API_KEY. Verify with npx checksumai test -g "example" (checks login). Re-download after Checksum updates the variables, and in any automation that starts a Checksum test run or AI generation.

Commands

Command	Purpose	Details
`npx checksumai test`	Run tests, optionally healing failures, running affected tests only, or re-running failures	Flags below
`npx checksumai dotenv`	Download environment variables into `.env` (`--download`, `--api-key=<KEY>`)	Environment variables
`npx checksumai show-report`	Open the HTML report from the most recent local run in the default browser	Results, Reports & Traces
`npx checksumai init`	Create the `checksum/` folder. Checksum runs this during onboarding.	Repository tooling
`npx checksumai tsconfig`	Add or refresh `checksum/tsconfig.json`	Repository tooling
`npx checksumai eslint`	Add an ESLint configuration to `checksum/`	Repository tooling
`npx checksumai postinstall`	Post-install setup. Runs automatically after `npm install`.	Repository tooling

`npx checksumai test` flags

Flag	Default	Description
`-g "pattern"`	All tests	Run only tests whose name matches the pattern
`--cksm-auto-heal`	Off	After the run finishes, heal any failing tests automatically. If the run has failures, Checksum analyzes them and (by default) opens a PR with the fixes.
`--cksm-auto-heal-create-pr[=false]`	`true` when `--cksm-auto-heal` is set	Whether healing opens a pull request with the fixes. Pass `=false` to start healing without creating a PR.
`--cksm-auto-heal-repo-name=<owner>/<repo>`	Auto-detected in CI	Repository that hosts the source PR for healing progress comments (usually the app-code repo). Auto-detected from the CI environment when omitted.
`--cksm-auto-heal-branch=<branch>`	Auto-detected in CI	Branch in the tests repository that the healed PR targets. Auto-detected from the CI environment when omitted.
`--cksm-auto-heal-pr-number=<number>`	Auto-detected for PR events	Existing pull request to post healing progress on. Auto-detected for pull-request CI events when omitted.
`--cksm-affected[=<ref>]`	PR/MR target branch in CI	Run only tests affected by files changed since a git ref (branch, tag, or SHA). In GitHub Actions / GitLab CI, defaults to the PR/MR target branch when no ref is passed (`GITHUB_BASE_REF` / `CI_MERGE_REQUEST_TARGET_BRANCH_NAME`).
`--cksm-affected-dry-run`	Off	Print the resolved changed files and affected test IDs, then exit without running Playwright. Works without `--cksm-affected`.
`--cksm-rerun-failed[=<id>]`	Latest completed non-manual run	Re-run the non-passing test files from the latest completed non-manual run, or from the given test run UUID.

CI auto-detection: in GitHub Actions and GitLab CI, repository, branch, and pull-request number are read from the CI environment, so --cksm-auto-heal alone is usually enough.

`--cksm-affected` behavior and constraints

Three steps: (1) compute changed files between the current checkout and the base ref; (2) call POST /public-api/v1/affected-tests; (3) run Playwright with an internal --grep over the returned test IDs.

Constraint	Details
Can’t be combined with `-g` / `--grep`	Affected mode sets grep internally
Can’t be combined with `--cksm-rerun-failed`	Pick one selection mode per run
Needs git history	Shallow clones without enough history may fail. The fetch depth must include the merge base with the target ref (e.g. `fetch-depth: 0` in `actions/checkout`).

`--cksm-rerun-failed` behavior

Re-runs only test files that didn’t pass (failed or recovered). Without an ID, fetches per-test results via GET /public-api/v1/test-runs/latest; with =$TEST_RUN_ID, uses that run. Resolves failing file paths and passes them to Playwright. If every test passed, exits successfully with nothing to run. Can’t be combined with --cksm-affected or -g.Reporting: with hostReports enabled (default true when CI=true), results upload to the dashboard.

Run from GitHub Actions

On GitHub, the simplest option is the Checksum GitHub Action. It starts a cloud run from your workflow and calls the REST API for you, so you don’t need Playwright on your runner. Store your API key as the repository secret CHECKSUM_API_KEY:

.github/workflows/checksum.yml

name: Checksum tests
on: pull_request

permissions:
  contents: read
  pull-requests: read

jobs:
  checksum:
    runs-on: ubuntu-latest
    steps:
      - uses: checksum-ai/test-run-action@v2
        with:
          api-key: ${{ secrets.CHECKSUM_API_KEY }}
          grep: 'checkout'
          auto-heal: true

The action can pick tests by name pattern, by what a PR changed, or by suite, test, or collection ID, and it can wait for the result, shard, and override environment variables. All of its options, plus GitLab and other CI recipes, are in CI/CD Integration → GitHub Action.

Reference for AI: GitHub Action summary

Action: checksum-ai/test-run-action@v2. Calls the REST execution endpoints. Required input: api-key: ${{ secrets.CHECKSUM_API_KEY }}. Selection modes: grep, affected, suite-ids, test-ids, collection-id. Other inputs: wait, sharding, env overrides, auto-heal. Full input table: CI/CD Integration → GitHub Action.

Start a cloud run from the REST API

With the REST API, Checksum runs your tests in its own cloud, which works from any CI system or script. Each call starts a run and immediately returns a runId. You then use that ID to check the result. All calls send Authorization: Bearer $CHECKSUM_API_KEY (see Base URL & versions). The v1 execution endpoints run against the project’s configured branch and environment.

{
  "runId": "9f2c7a4e-8b31-4d6a-a2f0-3c5e1b7d9a42",
  "name": "job-name-12345",
  "sharded": false
}

Every way of starting a run also accepts two optional settings: autoHeal, which heals failures automatically when the run ends (Auto-Healing), and shardCount, which splits the run across parallel machines (Sharding).

Update checksumai before shardingAfter all shards finish, Checksum merges their results using the checksumai version installed on the branch you run against. If that version is older than 4.4.0, the shards still run but may never merge, and the run never returns a final verdict. An autoHeal request on that run is then never evaluated. Run npm install checksumai@latest on your tests branch and commit it before your first sharded run.

Run the whole suite

POSThttps://api.checksum.ai/public-api/v1/execution/suite

curl -X POST https://api.checksum.ai/public-api/v1/execution/suite \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"autoHeal": {"autoCreatePR": true, "branch": "main", "repoName": "acme-co/my-tests"}}'

The body is optional. This example also asks Checksum to heal any failures and open a PR in acme-co/my-tests.

Run a collection

POST

curl -X POST https://api.checksum.ai/public-api/v1/execution/collection/$COLLECTION_ID \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json"

Runs every test in one collection, such as “Checkout”.

Run specific tests

POSThttps://api.checksum.ai/public-api/v1/execution/tests

curl -X POST https://api.checksum.ai/public-api/v1/execution/tests \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"testIds": ["id1", "id2"]}'

Pass the Checksum test IDs you want to run. These can come straight from the affected-tests call.

Run tests that match a name or tag

POSThttps://api.checksum.ai/public-api/v2/execution/grepv2

The most flexible option, and the one to use for pull-request checks. Besides a name or tag pattern such as @smoke, it can check out a specific branch of your tests repository and point the run at a preview deployment:

# PR check against a preview deployment
curl -X POST https://api.checksum.ai/public-api/v2/execution/grep \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "grep": "@smoke",
    "branch": "feature/checkout-preview",
    "envOverrides": { "BASE_URL": "https://pr-42.preview.example.com" },
    "shardCount": 8
  }'

Only your own application variables (like BASE_URL) can be overridden. Names reserved by Checksum are rejected. An older v1 version of this endpoint accepts only the pattern. Use v2 for new integrations.

Find the tests a change affects

POSThttps://api.checksum.ai/public-api/v1/affected-tests

Send the list of files a change touched (for example, the output of git diff --name-only) and Checksum returns the tests most likely affected. Then run just those with Run specific tests. The CLI’s --cksm-affected flag does both steps for you.

curl -X POST https://api.checksum.ai/public-api/v1/affected-tests \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "changedFiles": [
      "src/checkout/payment.ts",
      "src/cart/cart.service.ts"
    ]
  }'

{ "affectedTestIds": ["D7iP0", "Ab3xY"] }

Reference for AI: execution and affected-tests endpoints

Base URLs: https://api.checksum.ai/public-api/v1/ (all execution endpoints except grep) and https://api.checksum.ai/public-api/v2/ (grep). Headers on every call: Authorization: Bearer $CHECKSUM_API_KEY, Content-Type: application/json. v1 execution endpoints run against the project’s configured branch and environment.

Execution response (all execution endpoints)

Field	Type	Description
`runId`	string (UUID)	UUID of the run. Use it to poll status, fetch results, and heal it later (as `testRunId`).
`name`	string \| null	Underlying job name for non-sharded runs. `null` for sharded runs. Only used by the legacy status-by-job-name endpoint.
`sharded`	boolean	Whether the run was split across shards

Optional fields accepted by every execution endpoint

Field	Type	Required	Description
`autoHeal`	object	No	Opt in to heal-on-failure. When present, Checksum starts auto-healing if the run ends `failed`. Shape: `{ autoCreatePR, branch, repoName, prNumber, metadata }`. `autoHeal.branch` is the tests-repo branch where the heal PR is opened, not the branch the run checks out. Omit to opt out. Field reference: Where healing runs.
`shardCount`	integer	No	Omit, or set `1`, for a normal run. Set `2`–`40` to run shards in parallel and merge them into one result. Each shard runs one Playwright worker. Requires `checksumai` 4.4.0+ on the branch; older versions may never merge, so no final `verdict` and any `autoHeal` is never evaluated.

`POST https://api.checksum.ai/public-api/v1/execution/suite`

Runs the full test suite in Checksum’s cloud. Body optional: autoHeal, shardCount. Example body:

{
  "autoHeal": {
    "autoCreatePR": true,
    "branch": "main",
    "repoName": "acme-co/my-tests",
    "prNumber": 42
  }
}

Response: execution response, e.g. { "runId": "9f2c7a4e-8b31-4d6a-a2f0-3c5e1b7d9a42", "name": "job-name-12345", "sharded": false }. Next: GET /public-api/v1/execution/status/run/{runId}.

`POST https://api.checksum.ai/public-api/v1/execution/collection/{id}`

Path parameter	Type	Required	Description
`id`	string	Yes	The collection ID

Runs every test in the collection. Body optional: autoHeal, shardCount. Response: execution response. Next: GET /execution/status/run/{runId}.

`POST https://api.checksum.ai/public-api/v1/execution/tests`

Field	Type	Required	Description
`testIds`	string[]	Yes	Checksum test IDs (`checksumTestId`) to run. Accepts `affectedTestIds` from `POST /affected-tests` directly.
`autoHeal`	object	No	Heal-on-failure (see optional fields)
`shardCount`	integer	No	`2`–`40` for a sharded run

Response: execution response. Next: GET /execution/status/run/{runId}.

`POST https://api.checksum.ai/public-api/v2/execution/grep`

Runs every test whose name matches a pattern. The only execution endpoint that can check out a specific branch and inject per-run environment variables; use it for PR checks against preview deployments.

Field	Type	Required	Description
`grep`	string	Yes	Substring or regex matched against test names. Tags such as `@smoke` work too.
`branch`	string	No	Branch of the tests repository to check out for this run
`envOverrides`	object (string → string)	No	Per-run environment variables (v2 grep only). Keys named `CI` or starting with `CHECKSUM_` are rejected with `400`.
`shardCount`	integer	No	`2`–`40` for a sharded run. Each shard runs one Playwright worker.
`autoHeal`	object	No	Heal-on-failure, same shape as the optional fields. Works together with `shardCount`. `autoHeal.branch` is where the heal PR lands, not the run’s checkout branch.

{
  "grep": "checkout",
  "branch": "feature/preview",
  "envOverrides": { "BASE_URL": "https://pr-42.preview.example.com" },
  "shardCount": 8
}

Error	Cause
`400`	`envOverrides` contains a reserved key (`CI`, or any key starting with `CHECKSUM_`). `envOverrides` is for application variables only (e.g. `BASE_URL`).

Legacy: POST https://api.checksum.ai/public-api/v1/execution/grep still exists with a smaller body (grep only). New integrations should use v2.Response: execution response. Next: poll GET /public-api/v1/execution/status/run/{runId}; pass the CI check only when verdict is "pass".

`POST https://api.checksum.ai/public-api/v1/affected-tests`

Returns the Checksum test IDs most likely affected by a set of changed source files.

Field	Type	Required	Description
`changedFiles`	string[]	Yes	Changed source file paths, e.g. the output of `git diff --name-only`. Capped at 1,000 files.

Response field	Type	Description
`affectedTestIds`	string[]	Checksum test IDs likely affected by the changed files

Next: pass affectedTestIds as testIds to POST /public-api/v1/execution/tests. The CLI flag --cksm-affected does both steps.

Check whether a run passed

A cloud run takes a few minutes. To find out how it went, ask for its status using the runId you got back when you started it. Keep asking until isTerminal is true, then look at verdict: "pass" means the run passed.

Get a run’s status and verdict

GET

curl https://api.checksum.ai/public-api/v1/execution/status/run/$RUN_ID \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"

When the run is done, you’ll see something like this:

{
  "status": "passed",
  "isTerminal": true,
  "verdict": "pass",
  "phase": "complete",
  "executedCount": 47,
  "sharded": true,
  "shardTotal": 8,
  "passed": 45,
  "failed": 0,
  "recovered": 2,
  "bug": 0,
  "failureReason": null
}

Gate CI on verdictPass your CI check only when verdict is "pass", not on status or on the counts. Checksum computes verdict only after a sharded run has merged its results, and it treats an empty test selection as a failure. A passed status with executedCount: 0 still returns verdict: "fail".

From here you can read the per-test details (Results API) or heal failures (Auto-Healing).

Older integrations: status by job name

Before run IDs, integrations checked status by the job name returned when a run started. That still works for non-sharded runs, but use the run ID for anything new, and always for sharded runs.

curl https://api.checksum.ai/public-api/v1/execution/status/job-name-12345 \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"

API-triggered runs can’t be cancelledThere’s no cancel endpoint in the public API. Cancelling your CI job only stops your workflow from polling. The Checksum run keeps going until it finishes. Plan concurrency to match: GitHub Actions’ cancel-in-progress, for example, cancels your workflow but not the run on Checksum.

Reference for AI: run status endpoints

`GET https://api.checksum.ai/public-api/v1/execution/status/run/{runId}`

Server-computed status of a run by runId from any execution endpoint. Works for sharded and non-sharded runs. Use for all new integrations. Header: Authorization: Bearer $CHECKSUM_API_KEY.

Path parameter	Type	Description
`runId`	string (UUID)	The `runId` returned when the run was triggered

Response while running:

{
  "status": "running",
  "isTerminal": false,
  "verdict": "pending",
  "phase": "sharding",
  "executedCount": 0,
  "sharded": true,
  "shardTotal": 8,
  "passed": 0,
  "failed": 0,
  "recovered": 0,
  "bug": 0,
  "failureReason": null
}

Field	Type	Description
`status`	string	Current run state (e.g. `queued`, `running`, `passed`, `failed`). Informational. Gate on `verdict`.
`isTerminal`	boolean	`true` once the run has reached a final state
`verdict`	enum	CI-friendly result: `pass`, `fail`, or `pending`
`phase`	enum	`running`, `sharding`, `merging`, `complete`, or `failed`. Informational only.
`executedCount`	number	Number of tests that actually ran. An empty selection returns `verdict: "fail"`.
`passed` / `failed`	number	Count of tests that passed / failed
`recovered`	number	Tests that failed at first, then passed through runtime auto-recovery or a retry. They count as passing.
`bug`	number	Tests Checksum flagged as likely product bugs
`sharded`	boolean	Whether the run used sharding
`shardTotal`	number	Number of shards, or `1` for a non-sharded run
`failureReason`	string \| null	Details when a run fails for infrastructure reasons, when available

Gating rule: poll until isTerminal is true, then pass only when verdict is "pass". Don’t gate on status or on counts. verdict is computed only after sharded results merge; an empty selection is a failure.Next: GET /public-api/v1/test-runs/{id}/results for per-test details, or POST /public-api/v1/auto-heal to heal failures.

`GET https://api.checksum.ai/public-api/v1/execution/status/{jobName}` (legacy)

Also: GET https://api.checksum.ai/public-api/v2/execution/status/{jobName} for v2 grep runs. Non-sharded runs only. Prefer GET /execution/status/run/{runId} for new integrations and all sharded runs. The v2 response adds testRunId (UUID) once terminal; use that value, not jobName, for POST /auto-heal.

Path parameter	Type	Description
`jobName`	string	The `name` value returned when the run was triggered

Response while running:

{ "status": "running" }

Response when complete (v1):

{
  "status": "passed",
  "passed": 45,
  "failed": 2,
  "recovered": 1,
  "bug": 0
}

Response when complete (v2):

{
  "testRunId": "00000000-0000-0000-0000-000000000000",
  "status": "passed",
  "passed": 45,
  "failed": 2,
  "recovered": 1,
  "bug": 0
}

`status` value	Meaning
`queued`	The run is waiting to start
`running`	Tests are executing
`passed`	All tests passed (or were recovered)
`healed`	Tests passed after healing was applied
`failed`	One or more tests failed
`process-error`	The run hit an infrastructure error
`cancelled`	The run was cancelled

Cancellation: there is no cancel endpoint. Cancelling a CI job stops polling only; the Checksum run continues. GitHub Actions cancel-in-progress cancels the workflow, not the Checksum run.

Example: run the suite, wait for the verdict, heal failures

This script puts the pieces together. It starts a full suite run with auto-heal on failure, then waits for it to finish. If the run fails, healing starts automatically, with no separate heal call. It needs curl, jq, and CHECKSUM_API_KEY in the environment. Set TESTS_REPO and TESTS_BRANCH to your tests repository and its integration branch.

Example: the full script

#!/usr/bin/env bash
set -euo pipefail

API_KEY="$CHECKSUM_API_KEY"
BASE="https://api.checksum.ai/public-api/v1"
TESTS_REPO="acme-co/my-tests"
TESTS_BRANCH="main"

# 1. Start a full suite run (opt into heal-on-failure)
RUN_ID=$(curl -s -X POST "$BASE/execution/suite" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d "$(jq -n \
    --arg repo "$TESTS_REPO" \
    --arg branch "$TESTS_BRANCH" \
    '{autoHeal: {autoCreatePR: true, repoName: $repo, branch: $branch}}')" \
  | jq -r '.runId')

echo "Started run: $RUN_ID"

# 2. Poll until the run finishes
while true; do
  RESULT=$(curl -s "$BASE/execution/status/run/$RUN_ID" \
    -H "Authorization: Bearer $API_KEY")
  STATUS=$(echo "$RESULT" | jq -r '.status')
  TERMINAL=$(echo "$RESULT" | jq -r '.isTerminal')

  echo "Status: $STATUS"
  if [[ "$TERMINAL" == "true" ]]; then
    break
  fi
  sleep 10
done

# 3. Gate on the verdict
VERDICT=$(echo "$RESULT" | jq -r '.verdict')
if [[ "$VERDICT" == "pass" ]]; then
  echo "All tests passed."
else
  echo "Run did not pass (verdict: $VERDICT) — auto-heal was dispatched with the run."
fi

To heal a run that has already finished without autoHeal, call POST /public-api/v1/auto-heal with testRunId set to the run’s UUID (the runId, or the id from GET /test-runs/latest). Don’t use the dispatch name.

Pick which tests to run

Every interface can narrow a run down. Here’s how the same choice looks in each:

Select by	CLI	REST API	GitHub Action input
Everything	`npx checksumai test`	`POST /v1/execution/suite`	`suite-ids`
Name / tag pattern	`-g "pattern"`	`POST /v2/execution/grep`	`grep`
A collection	—	`POST /v1/execution/collection/{id}`	`collection-id`
Specific tests	—	`POST /v1/execution/tests`	`test-ids`
Affected by a change	`--cksm-affected[=<ref>]`	`POST /v1/affected-tests` → `/execution/tests`	`affected`
Previously failing	`--cksm-rerun-failed[=<id>]`	Compose it yourself: filter results by `status=failed`, then pass the IDs to `/execution/tests`	—

Environment overrides per run

Runs use the environment settings from your project (Environments → Environment variables). You can override any variable for a single run:

Where	How
CLI, locally	`BASE_URL=http://localhost:3000 npx checksumai test`
CLI, in CI	Set CI environment variables or secrets (`BASE_URL`, `USERNAME`, `PASSWORD`, `LOGIN_URL`…)
REST API	`envOverrides` on `POST /public-api/v2/execution/grep` only
GitHub Action	`env-overrides` (grep mode only)

Variables you set explicitly always take precedence over the downloaded .env file.

Run modes and runtime recovery (`runMode`)

runMode in checksum.config.ts controls how the CLI reacts to a failing step:

Mode	Behavior
`RunMode.Normal` default	Standard execution. Tests run and fail on assertion errors, and results are reported.
`RunMode.Heal`	A failing step first tries auto-recovery in real time (finding the element another way, waiting for the right condition, adapting to UI changes). Only then is it reported as a failure. Tests that succeed this way are counted as recovered.

Two related options fine-tune recovery: useChecksumSelectors (smart selector recovery, default true) and useChecksumAI ({ actions: true, assertions: false } by default, so assertion failures aren’t auto-recovered). See Auto-Maintenance & Recovery and checksum.config.ts.

What gets reported (`hostReports`)

When hostReports is enabled (default true when CI=true, which most CI providers set automatically), the CLI uploads the following after each run. Local runs keep reports on your machine unless you turn hostReports on.

Test results: passed / failed / recovered / bug status per test
Videos of each test’s browser session
Screenshots at key points and on failure
HAR files of network traffic
Playwright traces for step-by-step debugging

View them in Test Results in the web app, or fetch them over the API. See Results, Reports & Traces.

How it works

Where tests execute, scheduling, and troubleshooting

Ways to run

Option	Where tests execute	Starts with	Auto-heal on failure	Best for
CLI Local	Your machine	`npx checksumai test`	`--cksm-auto-heal`	Debugging, reviewing a generated PR
CLI In your CI	Your CI runner	`npx checksumai test` in a pipeline	`--cksm-auto-heal`, or app-wide	Full control over the runner, network, and secrets
GitHub Action	Checksum’s cloud	`checksum-ai/test-run-action@v2`	`auto-heal: true`	PR checks without installing Playwright on your runner
API REST	Checksum’s cloud	`POST /execution/…`	`autoHeal` block	Any CI system, orchestration tools, internal scripts
Checksum-enabled Scheduled	Checksum’s cloud	Set up by your Checksum team	Project-level setting	Nightly or regular health checks (see Scheduled runs)

For pipeline recipes (GitHub Actions, GitLab CI, cross-repo checkout, PR gating), see CI/CD Integration.

Scheduled runs

Periodic runs execute your tests on a recurring, cron-based schedule. You don’t need to trigger runs by hand or rely only on CI events.

Coming soon: self-serve schedulingScheduling recurring runs in the web app is internal only for now. Until it’s generally available, you can:

Schedule runs in your own CI (schedule: cron in GitHub Actions, $CI_PIPELINE_SOURCE == "schedule" in GitLab). See CI/CD Integration.
Ask your Checksum team to set up scheduled runs, and to turn on auto-healing after scheduled runs (a project-level setting).

Planned capabilities for self-serve periodic runs:

Create scheduled runs with cron patterns (e.g. daily at midnight, every 6 hours)
Branch selection: choose which branch to run against
Test filtering with grep patterns
Activate / deactivate schedules without deleting them
Run now: trigger an immediate run outside the schedule
View results from scheduled runs alongside manual runs
Auto-heal on failure (project-level setting)

Troubleshooting

The status is passed but verdict is fail

Check executedCount. An empty selection (e.g. a grep that matches nothing) is treated as a failure. Fix the pattern or IDs.

A sharded run never reaches a final verdict

The shards finished but couldn’t be merged, usually because the tests branch has checksumai older than 4.4.0. Update it, commit, and re-run. Always set a timeout on your polling loop.

I cancelled the CI job but the run kept going

That’s expected. API-triggered runs can’t be cancelled through the public API.

400 on envOverrides

Remove any key named CI or starting with CHECKSUM_. Also, envOverrides is only accepted on v2 grep.

--cksm-affected fails in CI

Your checkout is probably shallow. Increase the fetch depth (e.g. fetch-depth: 0 in actions/checkout) so the merge base with the target branch is available.

The example test fails locally

The example test checks your login. Re-download .env, confirm the environment URL and test user in Environments, and make sure your network can reach the environment.

CI/CD Integration

GitHub Actions, GitLab, and PR gating recipes.

Sharding

Parallel runs, and how to prepare your suite.

Results, Reports & Traces

Dashboard and results API.

Auto-Healing

What happens when a run fails.

​At a glance

​Before you run the CLI

​Run tests with the CLI

​Everyday commands

​Useful flags

​Run only the tests a change affects

​Re-run only what failed

​Update the CLI

​Other CLI commands

​Commands

​npx checksumai test flags

​--cksm-affected behavior and constraints

​--cksm-rerun-failed behavior

​Run from GitHub Actions

​Start a cloud run from the REST API

​Run the whole suite

​Run a collection

​Run specific tests

​Run tests that match a name or tag

​Find the tests a change affects

​Execution response (all execution endpoints)

​Optional fields accepted by every execution endpoint

​POST https://api.checksum.ai/public-api/v1/execution/suite

​POST https://api.checksum.ai/public-api/v1/execution/collection/{id}

​POST https://api.checksum.ai/public-api/v1/execution/tests

​POST https://api.checksum.ai/public-api/v2/execution/grep

​POST https://api.checksum.ai/public-api/v1/affected-tests

​Check whether a run passed

​Get a run’s status and verdict

​Older integrations: status by job name

​GET https://api.checksum.ai/public-api/v1/execution/status/run/{runId}

​GET https://api.checksum.ai/public-api/v1/execution/status/{jobName} (legacy)

​Example: run the suite, wait for the verdict, heal failures

​Pick which tests to run

​Environment overrides per run

​Run modes and runtime recovery (runMode)

​What gets reported (hostReports)

​Ways to run

​Scheduled runs

​Troubleshooting

​Related

CI/CD Integration

Sharding

Results, Reports & Traces

Auto-Healing

At a glance

Before you run the CLI

Run tests with the CLI

Everyday commands

Useful flags

Run only the tests a change affects

Re-run only what failed

Update the CLI

Other CLI commands

Commands

`npx checksumai test` flags

`--cksm-affected` behavior and constraints

`--cksm-rerun-failed` behavior

Run from GitHub Actions

Start a cloud run from the REST API

Run the whole suite

Run a collection

Run specific tests

Run tests that match a name or tag

Find the tests a change affects

Execution response (all execution endpoints)

Optional fields accepted by every execution endpoint

`POST https://api.checksum.ai/public-api/v1/execution/suite`

`POST https://api.checksum.ai/public-api/v1/execution/collection/{id}`

`POST https://api.checksum.ai/public-api/v1/execution/tests`

`POST https://api.checksum.ai/public-api/v2/execution/grep`

`POST https://api.checksum.ai/public-api/v1/affected-tests`

Check whether a run passed

Get a run’s status and verdict

Older integrations: status by job name

`GET https://api.checksum.ai/public-api/v1/execution/status/run/{runId}`

`GET https://api.checksum.ai/public-api/v1/execution/status/{jobName}` (legacy)

Example: run the suite, wait for the verdict, heal failures

Pick which tests to run

Environment overrides per run

Run modes and runtime recovery (`runMode`)

What gets reported (`hostReports`)

Ways to run

Scheduled runs

Troubleshooting

Related