> ## Documentation Index
> Fetch the complete documentation index at: https://checksum.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Running Tests

> Your Checksum tests are Playwright tests, so they run anywhere. Run them on your own machine or CI runner with the Checksum CLI (npx checksumai test), or have Checksum run them in its cloud with the REST API or the GitHub Action. Every option reports to the same dashboard and can hand failures straight to auto-healing.

## At a glance

| You want to | Use | Section |
| - | - | - |
| Run tests on your machine or your own CI runner | `npx checksumai test` | [Run tests with the CLI](#run-tests-with-the-cli) |
| Start a cloud run from a GitHub workflow | The Checksum GitHub Action | [GitHub Actions](#run-from-github-actions) |
| Start a cloud run from any tool or CI system | The REST API | [Start a cloud run](#start-a-cloud-run-from-the-rest-api) |
| Start a run from your coding agent | The MCP `checksum_test_run` tool | [Coding Agents & MCP](/docs/coding-agents#typical-flows) |
| Find out whether a run passed | The run status endpoint | [Check whether a run passed](#check-whether-a-run-passed) |
| Run fewer tests (affected, failed, by name) | CLI flags or API options | [Pick which tests to run](#pick-which-tests-to-run) |

<div className="ai-ref">
  <Accordion title="Reference for AI: running tests at a glance" icon="robot">
    | Interface | Command / endpoint | What it does | Key constraint |
    | - | - | - | - |
    | CLI | `npx checksumai test` | Runs tests on your machine or CI runner | Run `npx checksumai dotenv --download --api-key=$CHECKSUM_API_KEY` first |
    | CLI | `--cksm-auto-heal`, `--cksm-affected`, `--cksm-rerun-failed` | Heal failures, run affected tests only, re-run failures | `--cksm-affected` and `--cksm-rerun-failed` can't be combined with each other or with `-g` |
    | GitHub Action | `checksum-ai/test-run-action@v2` | Starts a cloud run from a workflow | Wraps the REST endpoints |
    | REST API | `POST https://api.checksum.ai/public-api/v1/execution/suite` | Runs the full suite in Checksum's cloud | Returns `runId` |
    | REST API | `POST https://api.checksum.ai/public-api/v1/execution/collection/{id}` | Runs one collection | — |
    | REST API | `POST https://api.checksum.ai/public-api/v1/execution/tests` | Runs specific test IDs | Body: `testIds` |
    | REST API | `POST https://api.checksum.ai/public-api/v2/execution/grep` | Runs tests matching a name/tag pattern | Only endpoint with `branch` + `envOverrides` |
    | REST API | `POST https://api.checksum.ai/public-api/v1/affected-tests` | Returns test IDs affected by changed files | Max 1,000 files |
    | REST API | `GET https://api.checksum.ai/public-api/v1/execution/status/run/{runId}` | Run status + CI verdict | Gate on `verdict`, not `status` |
    | MCP | `checksum_test_run` (`mode`: `suite`, `collection`, `tests`, `grep`) | Starts a cloud run from a coding agent; returns `testRunId` | Optional `envOverrides` (grep), environment overrides (collection/tests), `shardCount`, `autoHeal` |

    * Base URL: `https://api.checksum.ai/public-api/v1/`. Only grep (and its job-name status) uses `https://api.checksum.ai/public-api/v2/`.
    * Auth: every REST call sends `Authorization: Bearer $CHECKSUM_API_KEY`. Key location: **Settings → Project Settings**.
    * Gating CI: poll `GET /execution/status/run/{runId}` until `isTerminal` is `true`, then pass only when `verdict` is `"pass"`.
    * Heal on failure: add an `autoHeal` object to any execution request, or pass `--cksm-auto-heal` to the CLI.
    * No cancel: API-triggered runs can't be cancelled through the public API.
    * Sharding: `shardCount` `2`–`40`. Requires `checksumai` 4.4.0+ on the tests branch.
  </Accordion>
</div>

<div className="part dev"><span className="part-icon">{"</>"}</span><div><div className="part-title">Developer guide</div><div className="part-sub">The CLI, GitHub Actions, and the REST API</div></div></div>

## Before you run the CLI

Checksum sets up your tests repository during [onboarding](/docs/onboarding). To run it with the CLI, clone the tests repo and prepare it once:

```bash theme={null}
npm install
npx playwright install --with-deps
npx checksumai dotenv --download --api-key=$CHECKSUM_API_KEY
```

`dotenv --download` writes a `.env` file with your project's environment settings: environment URL, login URL, credentials, and custom variables. Your API key is in **Settings → Project Settings** (see [API Keys](/docs/authentication#your-api-key)). Checksum maintains the file's values, so to change them, contact your Checksum team and then download it again. Any automation that starts a Checksum test run or AI generation should download the latest `.env` first (see [Environments](/docs/environments#download-your-settings-with-the-cli)).

To check your setup, run the built-in example test. It's a quick check that your login works:

```bash theme={null}
npx checksumai test -g "example"
```

## Run tests with the CLI

The Checksum CLI (`checksumai`) runs your generated Playwright tests. It downloads your environment configuration, runs the tests, uploads the results to the dashboard, and can hand failures to auto-healing. It ships in the `@checksum-ai/runtime` npm package, and every command runs with `npx checksumai`. Because it wraps Playwright, your tests also run under plain Playwright.

### Everyday commands

```bash theme={null}
# Run all tests
npx checksumai test

# Run tests whose name matches a pattern
npx checksumai test -g "User Can Create Opportunity"

# Run only tests affected by your changes vs. main
npx checksumai test --cksm-affected=origin/main

# Re-run only what failed last time
npx checksumai test --cksm-rerun-failed

# Run, then auto-heal any failures (repo/branch/PR auto-detected in CI)
npx checksumai test --cksm-auto-heal

# Heal failures but don't open a PR
npx checksumai test --cksm-auto-heal --cksm-auto-heal-create-pr=false

# Point at a local build instead of the downloaded environment URL
BASE_URL=http://localhost:3000 npx checksumai test
```

In CI (where `CI=true`), results upload to the dashboard automatically. See [What gets reported](#what-gets-reported-hostreports).

### Useful flags

Most runs need no flags at all. The ones you'll reach for:

* `-g "pattern"` runs only the tests whose name matches.
* `--cksm-auto-heal` sends failures to [auto-healing](/docs/auto-healing#choose-how-to-start-healing) when the run ends. In GitHub Actions and GitLab CI, Checksum detects the repository, branch, and pull request on its own, so this flag alone is usually enough.
* `--cksm-affected` runs only the tests a change affects ([below](#run-only-the-tests-a-change-affects)).
* `--cksm-rerun-failed` re-runs only what didn't pass last time ([below](#re-run-only-what-failed)).

The full flag list, with defaults, is in the Reference for AI block at the end of this CLI section.

### Run only the tests a change affects

For pull-request checks, you usually don't need the whole suite. `--cksm-affected` looks at which files changed since a base branch, asks Checksum which tests those files affect, and runs just those:

```bash theme={null}
# Compare against origin/main
npx checksumai test --cksm-affected=origin/main

# In GitHub Actions / GitLab MRs, the base ref is detected automatically:
#   GitHub Actions reads GITHUB_BASE_REF
#   GitLab merge requests read CI_MERGE_REQUEST_TARGET_BRANCH_NAME
npx checksumai test --cksm-affected

# Print the changed files and affected test IDs without running anything
npx checksumai test --cksm-affected-dry-run
```

It needs enough git history to find where your branch split off, so use a full checkout in CI (for example `fetch-depth: 0` in `actions/checkout`). It can't be combined with `-g` or `--cksm-rerun-failed`.

<Tip>
  **Try it with a dry run first**

  Run `--cksm-affected-dry-run` in CI for a while to see which tests *would* run, before you turn the flag on for PR workflows.
</Tip>

### Re-run only what failed

After a run with failures, you can retry just the test files that didn't pass (failed or recovered), without running everything again:

```bash theme={null}
# From the latest completed non-manual run
npx checksumai test --cksm-rerun-failed

# From a specific run (UUID)
npx checksumai test --cksm-rerun-failed=$TEST_RUN_ID

# After a CI run failed, retry only the failing specs locally
npx checksumai test --cksm-rerun-failed
```

If every test in that run passed, the command finishes successfully with nothing to run.

### Update the CLI

Installing your tests repository's dependencies installs the CLI (see [Before you run the CLI](#before-you-run-the-cli)). To update to the latest release, which you need before using [sharding](/docs/sharding) (minimum `4.4.0`), run this and commit the result:

```bash theme={null}
npm install checksumai@latest
# commit the updated package.json / lockfile to your tests branch
```

### Other CLI commands

| Command | What it does |
| - | - |
| `npx checksumai dotenv` | Downloads your environment variables into `.env` ([Environment variables](/docs/environments#work-with-environment-variables)) |
| `npx checksumai show-report` | Opens the HTML report from your most recent local run in your browser ([Results, Reports & Traces](/docs/results-and-reports)) |
| `npx checksumai init`, `tsconfig`, `eslint`, `postinstall` | Set up or refresh the `checksum/` folder. Checksum runs `init` for you during onboarding ([Repository tooling](/docs/test-repository#repository-tooling)). |

<div className="ai-ref">
  <Accordion title="Reference for AI: Checksum CLI (npx checksumai)" icon="robot">
    Package: `checksumai` (npm dependency of the tests repository). Invocation: `npx checksumai <command>`. Wraps Playwright; generated tests also run under plain Playwright. Update: `npm install checksumai@latest`, then commit `package.json` / lockfile to the tests branch; sharding requires `4.4.0+`.

    Setup per checkout: `npm install`, `npx playwright install --with-deps`, `npx checksumai dotenv --download --api-key=$CHECKSUM_API_KEY`. Verify with `npx checksumai test -g "example"` (checks login). Re-download after Checksum updates the variables, and in any automation that starts a Checksum test run or AI generation.

    #### Commands

    | Command | Purpose | Details |
    | - | - | - |
    | `npx checksumai test` | Run tests, optionally healing failures, running affected tests only, or re-running failures | Flags below |
    | `npx checksumai dotenv` | Download environment variables into `.env` (`--download`, `--api-key=<KEY>`) | [Environment variables](/docs/environments#work-with-environment-variables) |
    | `npx checksumai show-report` | Open the HTML report from the most recent local run in the default browser | [Results, Reports & Traces](/docs/results-and-reports) |
    | `npx checksumai init` | Create the `checksum/` folder. Checksum runs this during onboarding. | [Repository tooling](/docs/test-repository#repository-tooling) |
    | `npx checksumai tsconfig` | Add or refresh `checksum/tsconfig.json` | [Repository tooling](/docs/test-repository#repository-tooling) |
    | `npx checksumai eslint` | Add an ESLint configuration to `checksum/` | [Repository tooling](/docs/test-repository#repository-tooling) |
    | `npx checksumai postinstall` | Post-install setup. Runs automatically after `npm install`. | [Repository tooling](/docs/test-repository#repository-tooling) |

    #### `npx checksumai test` flags

    | Flag | Default | Description |
    | - | - | - |
    | `-g "pattern"` | All tests | Run only tests whose name matches the pattern |
    | `--cksm-auto-heal` | Off | After the run finishes, heal any failing tests automatically. If the run has failures, Checksum analyzes them and (by default) opens a PR with the fixes. |
    | `--cksm-auto-heal-create-pr[=false]` | `true` when `--cksm-auto-heal` is set | Whether healing opens a pull request with the fixes. Pass `=false` to start healing without creating a PR. |
    | `--cksm-auto-heal-repo-name=<owner>/<repo>` | Auto-detected in CI | Repository that hosts the source PR for healing progress comments (usually the app-code repo). Auto-detected from the CI environment when omitted. |
    | `--cksm-auto-heal-branch=<branch>` | Auto-detected in CI | Branch in the **tests repository** that the healed PR targets. Auto-detected from the CI environment when omitted. |
    | `--cksm-auto-heal-pr-number=<number>` | Auto-detected for PR events | Existing pull request to post healing progress on. Auto-detected for pull-request CI events when omitted. |
    | `--cksm-affected[=<ref>]` | PR/MR target branch in CI | Run only tests affected by files changed since a git ref (branch, tag, or SHA). In GitHub Actions / GitLab CI, defaults to the PR/MR target branch when no ref is passed (`GITHUB_BASE_REF` / `CI_MERGE_REQUEST_TARGET_BRANCH_NAME`). |
    | `--cksm-affected-dry-run` | Off | Print the resolved changed files and affected test IDs, then exit without running Playwright. Works without `--cksm-affected`. |
    | `--cksm-rerun-failed[=<id>]` | Latest completed non-manual run | Re-run the non-passing test files from the latest completed non-manual run, or from the given test run UUID. |

    CI auto-detection: in GitHub Actions and GitLab CI, repository, branch, and pull-request number are read from the CI environment, so `--cksm-auto-heal` alone is usually enough.

    #### `--cksm-affected` behavior and constraints

    Three steps: (1) compute changed files between the current checkout and the base ref; (2) call `POST /public-api/v1/affected-tests`; (3) run Playwright with an internal `--grep` over the returned test IDs.

    | Constraint | Details |
    | - | - |
    | Can't be combined with `-g` / `--grep` | Affected mode sets grep internally |
    | Can't be combined with `--cksm-rerun-failed` | Pick one selection mode per run |
    | Needs git history | Shallow clones without enough history may fail. The fetch depth must include the merge base with the target ref (e.g. `fetch-depth: 0` in `actions/checkout`). |

    #### `--cksm-rerun-failed` behavior

    Re-runs only test **files** that didn't pass (failed or recovered). Without an ID, fetches per-test results via `GET /public-api/v1/test-runs/latest`; with `=$TEST_RUN_ID`, uses that run. Resolves failing file paths and passes them to Playwright. If every test passed, exits successfully with nothing to run. Can't be combined with `--cksm-affected` or `-g`.

    Reporting: with `hostReports` enabled (default `true` when `CI=true`), results upload to the dashboard.
  </Accordion>
</div>

## Run from GitHub Actions

On GitHub, the simplest option is the Checksum GitHub Action. It starts a cloud run from your workflow and calls the REST API for you, so you don't need Playwright on your runner. Store your API key as the repository secret `CHECKSUM_API_KEY`:

```yaml .github/workflows/checksum.yml theme={null}
name: Checksum tests
on: pull_request

permissions:
  contents: read
  pull-requests: read

jobs:
  checksum:
    runs-on: ubuntu-latest
    steps:
      - uses: checksum-ai/test-run-action@v2
        with:
          api-key: ${{ secrets.CHECKSUM_API_KEY }}
          grep: 'checkout'
          auto-heal: true
```

The action can pick tests by name pattern, by what a PR changed, or by suite, test, or collection ID, and it can wait for the result, shard, and override environment variables. All of its options, plus GitLab and other CI recipes, are in [CI/CD Integration → GitHub Action](/docs/ci-integration#run-checksum-with-the-github-action).

<div className="ai-ref">
  <Accordion title="Reference for AI: GitHub Action summary" icon="robot">
    Action: `checksum-ai/test-run-action@v2`. Calls the REST execution endpoints. Required input: `api-key: ${{ secrets.CHECKSUM_API_KEY }}`. Selection modes: `grep`, `affected`, `suite-ids`, `test-ids`, `collection-id`. Other inputs: `wait`, sharding, env overrides, `auto-heal`. Full input table: [CI/CD Integration → GitHub Action](/docs/ci-integration#run-checksum-with-the-github-action).
  </Accordion>
</div>

## Start a cloud run from the REST API

With the REST API, Checksum runs your tests in its own cloud, which works from any CI system or script. Each call starts a run and immediately returns a `runId`. You then use that ID to check the result. All calls send `Authorization: Bearer $CHECKSUM_API_KEY` (see [Base URL & versions](/docs/authentication#base-url-and-versions)). The v1 execution endpoints run against the project's configured branch and environment.

```json theme={null}
{
  "runId": "9f2c7a4e-8b31-4d6a-a2f0-3c5e1b7d9a42",
  "name": "job-name-12345",
  "sharded": false
}
```

Every way of starting a run also accepts two optional settings: `autoHeal`, which heals failures automatically when the run ends ([Auto-Healing](/docs/auto-healing#choose-how-to-start-healing)), and `shardCount`, which splits the run across parallel machines ([Sharding](/docs/sharding)).

<Warning>
  **Update `checksumai` before sharding**

  After all shards finish, Checksum merges their results using the `checksumai` version installed on the branch you run against. If that version is older than **4.4.0**, the shards still run but may never merge, and the run never returns a final `verdict`. An `autoHeal` request on that run is then never evaluated. Run `npm install checksumai@latest` on your tests branch and commit it before your first sharded run.
</Warning>

### Run the whole suite

<div className="endpoint"><span className="method post">POST</span><code>[https://api.checksum.ai/public-api/v1/execution/suite](https://api.checksum.ai/public-api/v1/execution/suite)</code></div>

```bash theme={null}
curl -X POST https://api.checksum.ai/public-api/v1/execution/suite \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"autoHeal": {"autoCreatePR": true, "branch": "main", "repoName": "acme-co/my-tests"}}'
```

The body is optional. This example also asks Checksum to heal any failures and open a PR in `acme-co/my-tests`.

### Run a collection

<div className="endpoint"><span className="method post">POST</span><code>{"https://api.checksum.ai/public-api/v1/execution/collection/{id}"}</code></div>

```bash theme={null}
curl -X POST https://api.checksum.ai/public-api/v1/execution/collection/$COLLECTION_ID \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json"
```

Runs every test in one collection, such as "Checkout".

### Run specific tests

<div className="endpoint"><span className="method post">POST</span><code>[https://api.checksum.ai/public-api/v1/execution/tests](https://api.checksum.ai/public-api/v1/execution/tests)</code></div>

```bash theme={null}
curl -X POST https://api.checksum.ai/public-api/v1/execution/tests \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"testIds": ["id1", "id2"]}'
```

Pass the Checksum test IDs you want to run. These can come straight from [the affected-tests call](#find-the-tests-a-change-affects).

### Run tests that match a name or tag

<div className="endpoint"><span className="method post">POST</span><code>[https://api.checksum.ai/public-api/v2/execution/grep](https://api.checksum.ai/public-api/v2/execution/grep)</code><span className="ver">v2</span></div>

The most flexible option, and the one to use for pull-request checks. Besides a name or tag pattern such as `@smoke`, it can check out a specific branch of your tests repository and point the run at a preview deployment:

```bash theme={null}
# PR check against a preview deployment
curl -X POST https://api.checksum.ai/public-api/v2/execution/grep \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "grep": "@smoke",
    "branch": "feature/checkout-preview",
    "envOverrides": { "BASE_URL": "https://pr-42.preview.example.com" },
    "shardCount": 8
  }'
```

Only your own application variables (like `BASE_URL`) can be overridden. Names reserved by Checksum are rejected. An older `v1` version of this endpoint accepts only the pattern. Use v2 for new integrations.

### Find the tests a change affects

<div className="endpoint"><span className="method post">POST</span><code>[https://api.checksum.ai/public-api/v1/affected-tests](https://api.checksum.ai/public-api/v1/affected-tests)</code></div>

Send the list of files a change touched (for example, the output of `git diff --name-only`) and Checksum returns the tests most likely affected. Then run just those with [Run specific tests](#run-specific-tests). The CLI's [`--cksm-affected`](#run-only-the-tests-a-change-affects) flag does both steps for you.

```bash theme={null}
curl -X POST https://api.checksum.ai/public-api/v1/affected-tests \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "changedFiles": [
      "src/checkout/payment.ts",
      "src/cart/cart.service.ts"
    ]
  }'
```

```json theme={null}
{ "affectedTestIds": ["D7iP0", "Ab3xY"] }
```

<div className="ai-ref">
  <Accordion title="Reference for AI: execution and affected-tests endpoints" icon="robot">
    Base URLs: `https://api.checksum.ai/public-api/v1/` (all execution endpoints except grep) and `https://api.checksum.ai/public-api/v2/` (grep). Headers on every call: `Authorization: Bearer $CHECKSUM_API_KEY`, `Content-Type: application/json`. v1 execution endpoints run against the project's configured branch and environment.

    #### Execution response (all execution endpoints)

    | Field | Type | Description |
    | - | - | - |
    | `runId` | string (UUID) | UUID of the run. Use it to poll status, fetch [results](/docs/results-and-reports#get-results-with-the-rest-api), and [heal](/docs/auto-healing#choose-how-to-start-healing) it later (as `testRunId`). |
    | `name` | string \| null | Underlying job name for non-sharded runs. `null` for sharded runs. Only used by the legacy status-by-job-name endpoint. |
    | `sharded` | boolean | Whether the run was split across shards |

    #### Optional fields accepted by every execution endpoint

    | Field | Type | Required | Description |
    | - | - | - | - |
    | `autoHeal` | object | No | Opt in to heal-on-failure. When present, Checksum starts auto-healing if the run ends `failed`. Shape: `{ autoCreatePR, branch, repoName, prNumber, metadata }`. `autoHeal.branch` is the **tests-repo** branch where the heal PR is opened, not the branch the run checks out. Omit to opt out. Field reference: [Where healing runs](/docs/auto-healing#choose-where-the-fix-pr-lands). |
    | `shardCount` | integer | No | Omit, or set `1`, for a normal run. Set `2`–`40` to run shards in parallel and merge them into one result. Each shard runs one Playwright worker. Requires `checksumai` **4.4.0+** on the branch; older versions may never merge, so no final `verdict` and any `autoHeal` is never evaluated. |

    #### `POST https://api.checksum.ai/public-api/v1/execution/suite`

    Runs the full test suite in Checksum's cloud. Body optional: `autoHeal`, `shardCount`. Example body:

    ```json theme={null}
    {
      "autoHeal": {
        "autoCreatePR": true,
        "branch": "main",
        "repoName": "acme-co/my-tests",
        "prNumber": 42
      }
    }
    ```

    Response: execution response, e.g. `{ "runId": "9f2c7a4e-8b31-4d6a-a2f0-3c5e1b7d9a42", "name": "job-name-12345", "sharded": false }`. Next: `GET /public-api/v1/execution/status/run/{runId}`.

    #### `POST https://api.checksum.ai/public-api/v1/execution/collection/{id}`

    | Path parameter | Type | Required | Description |
    | - | - | - | - |
    | `id` | string | Yes | The collection ID |

    Runs every test in the collection. Body optional: `autoHeal`, `shardCount`. Response: execution response. Next: `GET /execution/status/run/{runId}`.

    #### `POST https://api.checksum.ai/public-api/v1/execution/tests`

    | Field | Type | Required | Description |
    | - | - | - | - |
    | `testIds` | string\[] | Yes | Checksum test IDs (`checksumTestId`) to run. Accepts `affectedTestIds` from `POST /affected-tests` directly. |
    | `autoHeal` | object | No | Heal-on-failure (see optional fields) |
    | `shardCount` | integer | No | `2`–`40` for a sharded run |

    Response: execution response. Next: `GET /execution/status/run/{runId}`.

    #### `POST https://api.checksum.ai/public-api/v2/execution/grep`

    Runs every test whose name matches a pattern. The only execution endpoint that can check out a specific branch and inject per-run environment variables; use it for PR checks against preview deployments.

    | Field | Type | Required | Description |
    | - | - | - | - |
    | `grep` | string | Yes | Substring or regex matched against test names. Tags such as `@smoke` work too. |
    | `branch` | string | No | Branch of the tests repository to check out **for this run** |
    | `envOverrides` | object (string → string) | No | Per-run environment variables (v2 grep only). Keys named `CI` or starting with `CHECKSUM_` are rejected with `400`. |
    | `shardCount` | integer | No | `2`–`40` for a sharded run. Each shard runs one Playwright worker. |
    | `autoHeal` | object | No | Heal-on-failure, same shape as the optional fields. Works together with `shardCount`. `autoHeal.branch` is where the heal PR lands, not the run's checkout branch. |

    ```json theme={null}
    {
      "grep": "checkout",
      "branch": "feature/preview",
      "envOverrides": { "BASE_URL": "https://pr-42.preview.example.com" },
      "shardCount": 8
    }
    ```

    | Error | Cause |
    | - | - |
    | `400` | `envOverrides` contains a reserved key (`CI`, or any key starting with `CHECKSUM_`). `envOverrides` is for application variables only (e.g. `BASE_URL`). |

    Legacy: `POST https://api.checksum.ai/public-api/v1/execution/grep` still exists with a smaller body (`grep` only). New integrations should use v2.

    Response: execution response. Next: poll `GET /public-api/v1/execution/status/run/{runId}`; pass the CI check only when `verdict` is `"pass"`.

    #### `POST https://api.checksum.ai/public-api/v1/affected-tests`

    Returns the Checksum test IDs most likely affected by a set of changed source files.

    | Field | Type | Required | Description |
    | - | - | - | - |
    | `changedFiles` | string\[] | Yes | Changed source file paths, e.g. the output of `git diff --name-only`. Capped at 1,000 files. |

    | Response field | Type | Description |
    | - | - | - |
    | `affectedTestIds` | string\[] | Checksum test IDs likely affected by the changed files |

    Next: pass `affectedTestIds` as `testIds` to `POST /public-api/v1/execution/tests`. The CLI flag `--cksm-affected` does both steps.
  </Accordion>
</div>

## Check whether a run passed

A cloud run takes a few minutes. To find out how it went, ask for its status using the `runId` you got back when you started it. Keep asking until `isTerminal` is `true`, then look at `verdict`: `"pass"` means the run passed.

### Get a run's status and verdict

<div className="endpoint"><span className="method get">GET</span><code>{"https://api.checksum.ai/public-api/v1/execution/status/run/{runId}"}</code></div>

```bash theme={null}
curl https://api.checksum.ai/public-api/v1/execution/status/run/$RUN_ID \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"
```

When the run is done, you'll see something like this:

```json theme={null}
{
  "status": "passed",
  "isTerminal": true,
  "verdict": "pass",
  "phase": "complete",
  "executedCount": 47,
  "sharded": true,
  "shardTotal": 8,
  "passed": 45,
  "failed": 0,
  "recovered": 2,
  "bug": 0,
  "failureReason": null
}
```

<Tip>
  **Gate CI on `verdict`**

  Pass your CI check **only when `verdict` is `"pass"`**, not on `status` or on the counts. Checksum computes `verdict` only after a sharded run has merged its results, and it treats an empty test selection as a failure. A `passed` status with `executedCount: 0` still returns `verdict: "fail"`.
</Tip>

From here you can read the per-test details ([Results API](/docs/results-and-reports#get-results-with-the-rest-api)) or heal failures ([Auto-Healing](/docs/auto-healing#choose-how-to-start-healing)).

### Older integrations: status by job name

Before run IDs, integrations checked status by the job `name` returned when a run started. That still works for non-sharded runs, but use the run ID for anything new, and always for sharded runs.

```bash theme={null}
curl https://api.checksum.ai/public-api/v1/execution/status/job-name-12345 \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"
```

<Warning>
  **API-triggered runs can't be cancelled**

  There's no cancel endpoint in the public API. Cancelling your CI job only stops your workflow from polling. The Checksum run keeps going until it finishes. Plan concurrency to match: GitHub Actions' `cancel-in-progress`, for example, cancels your workflow but not the run on Checksum.
</Warning>

<div className="ai-ref">
  <Accordion title="Reference for AI: run status endpoints" icon="robot">
    #### `GET https://api.checksum.ai/public-api/v1/execution/status/run/{runId}`

    Server-computed status of a run by `runId` from any execution endpoint. Works for sharded and non-sharded runs. Use for all new integrations. Header: `Authorization: Bearer $CHECKSUM_API_KEY`.

    | Path parameter | Type | Description |
    | - | - | - |
    | `runId` | string (UUID) | The `runId` returned when the run was triggered |

    Response while running:

    ```json theme={null}
    {
      "status": "running",
      "isTerminal": false,
      "verdict": "pending",
      "phase": "sharding",
      "executedCount": 0,
      "sharded": true,
      "shardTotal": 8,
      "passed": 0,
      "failed": 0,
      "recovered": 0,
      "bug": 0,
      "failureReason": null
    }
    ```

    | Field | Type | Description |
    | - | - | - |
    | `status` | string | Current run state (e.g. `queued`, `running`, `passed`, `failed`). Informational. Gate on `verdict`. |
    | `isTerminal` | boolean | `true` once the run has reached a final state |
    | `verdict` | enum | CI-friendly result: `pass`, `fail`, or `pending` |
    | `phase` | enum | `running`, `sharding`, `merging`, `complete`, or `failed`. Informational only. |
    | `executedCount` | number | Number of tests that actually ran. An empty selection returns `verdict: "fail"`. |
    | `passed` / `failed` | number | Count of tests that passed / failed |
    | `recovered` | number | Tests that failed at first, then passed through runtime [auto-recovery](/docs/auto-maintenance) or a retry. They count as passing. |
    | `bug` | number | Tests Checksum flagged as likely product bugs |
    | `sharded` | boolean | Whether the run used sharding |
    | `shardTotal` | number | Number of shards, or `1` for a non-sharded run |
    | `failureReason` | string \| null | Details when a run fails for infrastructure reasons, when available |

    Gating rule: poll until `isTerminal` is `true`, then pass only when `verdict` is `"pass"`. Don't gate on `status` or on counts. `verdict` is computed only after sharded results merge; an empty selection is a failure.

    Next: `GET /public-api/v1/test-runs/{id}/results` for per-test details, or `POST /public-api/v1/auto-heal` to heal failures.

    #### `GET https://api.checksum.ai/public-api/v1/execution/status/{jobName}` (legacy)

    Also: `GET https://api.checksum.ai/public-api/v2/execution/status/{jobName}` for v2 grep runs. Non-sharded runs only. Prefer `GET /execution/status/run/{runId}` for new integrations and all sharded runs. The v2 response adds `testRunId` (UUID) once terminal; use that value, not `jobName`, for `POST /auto-heal`.

    | Path parameter | Type | Description |
    | - | - | - |
    | `jobName` | string | The `name` value returned when the run was triggered |

    Response while running:

    ```json theme={null}
    { "status": "running" }
    ```

    Response when complete (v1):

    ```json theme={null}
    {
      "status": "passed",
      "passed": 45,
      "failed": 2,
      "recovered": 1,
      "bug": 0
    }
    ```

    Response when complete (v2):

    ```json theme={null}
    {
      "testRunId": "00000000-0000-0000-0000-000000000000",
      "status": "passed",
      "passed": 45,
      "failed": 2,
      "recovered": 1,
      "bug": 0
    }
    ```

    | `status` value | Meaning |
    | - | - |
    | `queued` | The run is waiting to start |
    | `running` | Tests are executing |
    | `passed` | All tests passed (or were recovered) |
    | `healed` | Tests passed after healing was applied |
    | `failed` | One or more tests failed |
    | `process-error` | The run hit an infrastructure error |
    | `cancelled` | The run was cancelled |

    Cancellation: there is no cancel endpoint. Cancelling a CI job stops polling only; the Checksum run continues. GitHub Actions `cancel-in-progress` cancels the workflow, not the Checksum run.
  </Accordion>
</div>

## Example: run the suite, wait for the verdict, heal failures

This script puts the pieces together. It starts a full suite run with **auto-heal on failure**, then waits for it to finish. If the run fails, healing starts automatically, with no separate heal call. It needs `curl`, `jq`, and `CHECKSUM_API_KEY` in the environment. Set `TESTS_REPO` and `TESTS_BRANCH` to your tests repository and its integration branch.

<Accordion title="Example: the full script">
  ```bash theme={null}
  #!/usr/bin/env bash
  set -euo pipefail

  API_KEY="$CHECKSUM_API_KEY"
  BASE="https://api.checksum.ai/public-api/v1"
  TESTS_REPO="acme-co/my-tests"
  TESTS_BRANCH="main"

  # 1. Start a full suite run (opt into heal-on-failure)
  RUN_ID=$(curl -s -X POST "$BASE/execution/suite" \
    -H "Authorization: Bearer $API_KEY" \
    -H "Content-Type: application/json" \
    -d "$(jq -n \
      --arg repo "$TESTS_REPO" \
      --arg branch "$TESTS_BRANCH" \
      '{autoHeal: {autoCreatePR: true, repoName: $repo, branch: $branch}}')" \
    | jq -r '.runId')

  echo "Started run: $RUN_ID"

  # 2. Poll until the run finishes
  while true; do
    RESULT=$(curl -s "$BASE/execution/status/run/$RUN_ID" \
      -H "Authorization: Bearer $API_KEY")
    STATUS=$(echo "$RESULT" | jq -r '.status')
    TERMINAL=$(echo "$RESULT" | jq -r '.isTerminal')

    echo "Status: $STATUS"
    if [[ "$TERMINAL" == "true" ]]; then
      break
    fi
    sleep 10
  done

  # 3. Gate on the verdict
  VERDICT=$(echo "$RESULT" | jq -r '.verdict')
  if [[ "$VERDICT" == "pass" ]]; then
    echo "All tests passed."
  else
    echo "Run did not pass (verdict: $VERDICT) — auto-heal was dispatched with the run."
  fi
  ```
</Accordion>

To heal a run that has **already finished** without `autoHeal`, call [`POST /public-api/v1/auto-heal`](/docs/auto-healing#choose-how-to-start-healing) with `testRunId` set to the run's UUID (the `runId`, or the `id` from `GET /test-runs/latest`). Don't use the dispatch `name`.

## Pick which tests to run

Every interface can narrow a run down. Here's how the same choice looks in each:

| Select by | CLI | REST API | GitHub Action input |
| - | - | - | - |
| Everything | `npx checksumai test` | [`POST /v1/execution/suite`](#run-the-whole-suite) | `suite-ids` |
| Name / tag pattern | `-g "pattern"` | [`POST /v2/execution/grep`](#run-tests-that-match-a-name-or-tag) | `grep` |
| A collection | — | [`POST /v1/execution/collection/{id}`](#run-a-collection) | `collection-id` |
| Specific tests | — | [`POST /v1/execution/tests`](#run-specific-tests) | `test-ids` |
| Affected by a change | [`--cksm-affected[=<ref>]`](#run-only-the-tests-a-change-affects) | [`POST /v1/affected-tests`](#find-the-tests-a-change-affects) → `/execution/tests` | `affected` |
| Previously failing | [`--cksm-rerun-failed[=<id>]`](#re-run-only-what-failed) | Compose it yourself: filter [results](/docs/results-and-reports#get-results-with-the-rest-api) by `status=failed`, then pass the IDs to `/execution/tests` | — |

## Environment overrides per run

Runs use the environment settings from your project ([Environments → Environment variables](/docs/environments#work-with-environment-variables)). You can override any variable for a single run:

| Where | How |
| - | - |
| CLI, locally | `BASE_URL=http://localhost:3000 npx checksumai test` |
| CLI, in CI | Set CI environment variables or secrets (`BASE_URL`, `USERNAME`, `PASSWORD`, `LOGIN_URL`…) |
| REST API | `envOverrides` on [`POST /public-api/v2/execution/grep`](#run-tests-that-match-a-name-or-tag) only |
| GitHub Action | `env-overrides` (grep mode only) |

Variables you set explicitly always take precedence over the downloaded `.env` file.

## Run modes and runtime recovery (`runMode`)

`runMode` in `checksum.config.ts` controls how the CLI reacts to a failing step:

| Mode | Behavior |
| - | - |
| `RunMode.Normal` <span className="pill">default</span> | Standard execution. Tests run and fail on assertion errors, and results are reported. |
| `RunMode.Heal` | A failing step first tries **auto-recovery** in real time (finding the element another way, waiting for the right condition, adapting to UI changes). Only then is it reported as a failure. Tests that succeed this way are counted as **recovered**. |

Two related options fine-tune recovery: `useChecksumSelectors` (smart selector recovery, default `true`) and `useChecksumAI` (`{ actions: true, assertions: false }` by default, so assertion failures aren't auto-recovered). See [Auto-Maintenance & Recovery](/docs/auto-maintenance) and [checksum.config.ts](/docs/test-repository#configure-checksum-config-ts).

## What gets reported (`hostReports`)

When `hostReports` is enabled (default `true` when `CI=true`, which most CI providers set automatically), the CLI uploads the following after each run. Local runs keep reports on your machine unless you turn `hostReports` on.

* **Test results**: passed / failed / recovered / bug status per test
* **Videos** of each test's browser session
* **Screenshots** at key points and on failure
* **HAR files** of network traffic
* **Playwright traces** for step-by-step debugging

View them in **Test Results** in the web app, or fetch them over the API. See [Results, Reports & Traces](/docs/results-and-reports).

<div className="part bg"><span className="part-icon">i</span><div><div className="part-title">How it works</div><div className="part-sub">Where tests execute, scheduling, and troubleshooting</div></div></div>

## Ways to run

| Option | Where tests execute | Starts with | Auto-heal on failure | Best for |
| - | - | - | - | - |
| <span className="pill">CLI</span> Local | Your machine | `npx checksumai test` | `--cksm-auto-heal` | Debugging, reviewing a generated PR |
| <span className="pill">CLI</span> In your CI | Your CI runner | `npx checksumai test` in a pipeline | `--cksm-auto-heal`, or app-wide | Full control over the runner, network, and secrets |
| <span className="pill git">GitHub</span> Action | Checksum's cloud | `checksum-ai/test-run-action@v2` | `auto-heal: true` | PR checks without installing Playwright on your runner |
| <span className="pill api">API</span> REST | Checksum's cloud | `POST /execution/…` | `autoHeal` block | Any CI system, orchestration tools, internal scripts |
| <span className="pill cs">Checksum-enabled</span> Scheduled | Checksum's cloud | Set up by your Checksum team | Project-level setting | Nightly or regular health checks (see [Scheduled runs](#scheduled-runs)) |

For pipeline recipes (GitHub Actions, GitLab CI, cross-repo checkout, PR gating), see [CI/CD Integration](/docs/ci-integration).

## Scheduled runs

Periodic runs execute your tests on a recurring, cron-based schedule. You don't need to trigger runs by hand or rely only on CI events.

<Info>
  **Coming soon: self-serve scheduling**

  Scheduling recurring runs in the web app is **internal only** for now. Until it's generally available, you can:

  * Schedule runs in your own CI (`schedule: cron` in GitHub Actions, `$CI_PIPELINE_SOURCE == "schedule"` in GitLab). See [CI/CD Integration](/docs/ci-integration).
  * Ask your Checksum team to set up scheduled runs, and to turn on **auto-healing after scheduled runs** (a project-level setting).
</Info>

Planned capabilities for self-serve periodic runs:

* **Create scheduled runs** with cron patterns (e.g. daily at midnight, every 6 hours)
* **Branch selection**: choose which branch to run against
* **Test filtering** with grep patterns
* **Activate / deactivate** schedules without deleting them
* **Run now**: trigger an immediate run outside the schedule
* **View results** from scheduled runs alongside manual runs
* **Auto-heal on failure** (project-level setting)

## Troubleshooting

<AccordionGroup>
  <Accordion title="The status is passed but verdict is fail">
    Check `executedCount`. An empty selection (e.g. a grep that matches nothing) is treated as a failure. Fix the pattern or IDs.
  </Accordion>

  <Accordion title="A sharded run never reaches a final verdict">
    The shards finished but couldn't be merged, usually because the tests branch has `checksumai` older than 4.4.0. Update it, commit, and re-run. Always set a timeout on your polling loop.
  </Accordion>

  <Accordion title="I cancelled the CI job but the run kept going">
    That's expected. API-triggered runs can't be cancelled through the public API.
  </Accordion>

  <Accordion title="400 on envOverrides">
    Remove any key named `CI` or starting with `CHECKSUM_`. Also, `envOverrides` is only accepted on v2 grep.
  </Accordion>

  <Accordion title="--cksm-affected fails in CI">
    Your checkout is probably shallow. Increase the fetch depth (e.g. `fetch-depth: 0` in `actions/checkout`) so the merge base with the target branch is available.
  </Accordion>

  <Accordion title="The example test fails locally">
    The example test checks your login. Re-download `.env`, confirm the environment URL and test user in [Environments](/docs/environments#test-users), and make sure your network can reach the environment.
  </Accordion>
</AccordionGroup>

## Related

<CardGroup cols={2}>
  <Card title="CI/CD Integration" icon="code-branch" href="/docs/ci-integration">
    GitHub Actions, GitLab, and PR gating recipes.
  </Card>

  <Card title="Sharding" icon="layer-group" href="/docs/sharding">
    Parallel runs, and how to prepare your suite.
  </Card>

  <Card title="Results, Reports & Traces" icon="chart-line" href="/docs/results-and-reports">
    Dashboard and results API.
  </Card>

  <Card title="Auto-Healing" icon="wand-magic-sparkles" href="/docs/auto-healing">
    What happens when a run fails.
  </Card>
</CardGroup>