CI Integration

Overview

The typical CI setup:

Get your API key from the Checksum web app and store it as a CI secret
Choose a running strategy (on PR merge, on schedule, manual trigger)
If tests are in a separate repo, create a Personal Access Token (PAT) with repo access
Add the CI configuration below

GitHub Actions

Basic Setup

Create .github/workflows/checksum-tests.yml:

name: Run Checksum Tests

on:
  # Choose your triggers:
  # schedule:
  #   - cron: '0 0 * * *'  # Runs every day at 00:00 UTC
  workflow_dispatch:        # Allows manual triggering

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install npm dependencies
        run: npm install

      - name: Install Playwright with dependencies
        run: npx playwright install --with-deps

      - name: Download .env from Checksum
        run: npx checksumai dotenv --download --api-key="${{ secrets.CHECKSUM_API_KEY }}"

      - name: Run Checksum tests
        run: npx checksumai test
        env:
          CHECKSUM_API_KEY: ${{ secrets.CHECKSUM_API_KEY }}
          USERNAME: ${{ secrets.USERNAME }}
          PASSWORD: ${{ secrets.PASSWORD }}
          LOGIN_URL: ${{ secrets.LOGIN_URL }}
          BASE_URL: ${{ secrets.BASE_URL }}
          CI: true

Cross-Repo Setup

If your tests live in a different repository than the one running the action, checkout the test repo using a Personal Access Token:

steps:
  - uses: actions/checkout@v4

  - name: Checkout test repo
    uses: actions/checkout@v4
    with:
      repository: org/your-test-repo
      ref: main
      token: ${{ secrets.TEST_REPO_PAT }}
      path: .

  # ... rest of the steps

Setting Up Secrets

Go to your GitHub repository → Settings → Secrets and variables → Actions
Add the following secrets:
- CHECKSUM_API_KEY — your Checksum API key
- USERNAME — test user credentials
- PASSWORD — test user password
- BASE_URL — your application URL
- LOGIN_URL — your login page URL
- TEST_REPO_PAT (if cross-repo) — Personal Access Token with repo permissions

Testing the Action

Commit the workflow file to your main branch
Go to Actions → select your workflow → Run Workflow
Monitor the run to confirm tests execute successfully

GitHub Action (no CLI install)

If you don’t need to run tests on your own runner — e.g., your CI just needs to trigger a Checksum run and let Checksum execute it — use checksum-ai/test-run-action. It wraps the public-API execution endpoints in a single step, with optional auto-heal-on-failure built in.

Quick start

name: Checksum tests
on: pull_request

permissions:
  contents: read
  pull-requests: read

jobs:
  checksum:
    runs-on: ubuntu-latest
    steps:
      - uses: checksum-ai/test-run-action@v2
        with:
          api-key: ${{ secrets.CHECKSUM_API_KEY }}
          grep: 'checkout'
          auto-heal: true

That’s the whole step. The action auto-detects the source PR (from the event payload on pull_request, or via the GitHub API on push) and threads it through to auto-heal so progress comments land on the right PR.

@v2 is a breaking release. An empty test selection (e.g. a grep that matches nothing) now fails the step instead of passing. See the v2.0.0 release notes for the full list of behavior changes before upgrading from @v1.

Execution modes

Set exactly one of the following inputs to pick which tests to run:

Input	Endpoint	Use when
`grep`	`POST /public-api/v2/execution/grep`	You want to filter tests by name pattern (substring or regex).
`affected`	`POST /public-api/v1/affected-tests`, then grep dispatch	You want to run only the tests affected by the PR’s changed files.
`suite-ids`	`POST /public-api/v1/execution/suite`	You want to run one or more specific test suites. Comma-separated UUIDs.
`test-ids`	`POST /public-api/v1/execution/tests`	You want to run a specific list of tests by UUID.
`collection-id`	`POST /public-api/v1/execution/collection/:id`	You want to run a saved collection.

Auto-heal on failure

Set auto-heal: true to run Auto-Healing when the test run fails. Healing sessions start automatically and (by default) push fixes as a PR. Healing progress is reported as a comment on the originating PR — no extra wiring needed when running on pull_request events. To dispatch heal sessions without auto-creating a PR:

- uses: checksum-ai/test-run-action@v2
  with:
    api-key: ${{ secrets.CHECKSUM_API_KEY }}
    grep: 'checkout'
    auto-heal: true
    auto-create-pr: false

Wait for completion (gate the workflow)

By default the action exits as soon as the dispatch is accepted (~15s) — you get notified about results via the standard PR comment. Set wait: true if you want the workflow check itself to gate on the test outcome. The exit code gates on the run’s verdict (pass/fail), which correctly reflects sharded runs, empty selections, and infra failures:

- uses: checksum-ai/test-run-action@v2
  with:
    api-key: ${{ secrets.CHECKSUM_API_KEY }}
    grep: 'checkout'
    auto-heal: true
    wait: true

Outcome	Step exit
`verdict: pass`	success
`verdict: fail` (includes a failed run, an empty selection, a cancelled run, or an infra/process error)	failure
timeout (`verdict: pending`, run never reached terminal status)	failure

wait: true keeps a runner allocated for the full test-run duration (typically 5–25 minutes). When runner-minute cost matters, prefer wait: false and use the standard PR-comment notification.

Sharding

Set shard-count (2–40, honored in grep and affected modes) to fan the run out across parallel shards and merge the results into one verdict — the same sharding used in the Sharded PR Run curl example, wrapped in a single step:

- uses: checksum-ai/test-run-action@v2
  with:
    api-key: ${{ secrets.CHECKSUM_API_KEY }}
    grep: '@smoke'
    shard-count: 8
    wait: true
    wait-timeout-seconds: 1800

Before your first sharded run, update checksumai on your tests branch to 4.4.0 or later (the minimum production-supported version for sharded runs) and commit it — npm install checksumai@latest is the safe default. Older versions can hit report-merge failures, delaying the run’s terminal verdict well past a normal run’s duration; combined with wait: true and no wait-timeout-seconds, the step then rides the workflow job’s own timeout-minutes. Always set wait-timeout-seconds (or a job-level timeout-minutes) alongside wait: true so the action bounds runner time if a stale runtime delays report merging.

shard-count composes with auto-heal from action v2.1.0 onward: once the shards are merged, a merged run that ends failed is healed exactly as a non-sharded run would be. If the shards never merge (stale checksumai, see above), the heal is never evaluated. Older action versions reject the combination client-side, so update your pin (@v2 already resolves to the latest 2.x).

Sharding works best on a suite designed for parallel execution — isolated test data, no shared login/session state across tests. See Sharding Pitfalls before rolling out a large shard count.

Per-PR preview environments

Pass env-overrides (grep mode only) to inject per-run environment variables — handy when each PR is deployed to its own preview URL:

- uses: checksum-ai/test-run-action@v2
  with:
    api-key: ${{ secrets.CHECKSUM_API_KEY }}
    grep: 'checkout'
    auto-heal: true
    env-overrides: |
      {"BASE_URL": "https://pr-${{ github.event.pull_request.number }}.preview.example.com"}

Full reference

The complete inputs/outputs reference and changelog live in the action’s README. Use @v2 for compatible updates, @v2.0.0 for a release-specific tag, or a full commit SHA for immutability — tags (including @v2.0.0) can technically be moved by the repo owner, so a SHA pin is the only guarantee (see GitHub’s guidance on pinning third-party actions). @v1 remains available for the pre-sharding behavior but is no longer updated — see the v2.0.0 release notes for what changed.

GitLab CI/CD

Basic Setup

Add to your .gitlab-ci.yml:

image: node:20-bookworm

stages:
  - test

run-checksum-tests:
  stage: test
  rules:
    # - if: '$CI_PIPELINE_SOURCE == "schedule"'  # Runs for scheduled pipelines
    - if: '$CI_PIPELINE_SOURCE == "web"'          # Allows manual triggering
      when: manual

  before_script:
    - npm ci
    - npx playwright install --with-deps
    - npx checksumai dotenv --download --api-key="${CHECKSUM_API_KEY}"

  script:
    - npx checksumai test

  variables:
    CHECKSUM_API_KEY: $CHECKSUM_API_KEY
    USERNAME: $USERNAME
    PASSWORD: $PASSWORD
    LOGIN_URL: $LOGIN_URL
    BASE_URL: $BASE_URL
    CI: "true"

  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/

Cross-Repo Setup

If tests are in a separate repository:

  before_script:
    - git clone "https://gitlab-ci-token:${TEST_REPO_PAT}@gitlab.com/org/your-test-repo.git" tests
    - cd tests
    - npm ci
    - npx playwright install --with-deps
    - npx checksumai dotenv --download --api-key="${CHECKSUM_API_KEY}"

Setting Up CI/CD Variables

Go to your GitLab project → Settings → CI/CD → Variables
Add the same variables as listed in the GitHub section above

Programmatic CI Integration (API)

Instead of running the CLI directly, you can trigger and manage test runs programmatically using the Checksum API. This is useful for custom CI pipelines, orchestration tools, or when you need to chain test runs with other operations.

If you’re on GitHub Actions, the Checksum AI Test Run action wraps these endpoints in a single step — including the auto-heal trigger — and is usually a cleaner starting point than the raw API.

Run Tests

# Run all tests
curl -X POST https://api.checksum.ai/public-api/v1/execution/suite \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"shardCount": 4}'
# Response: { "runId": "9f2c7a4e-8b31-4d6a-a2f0-3c5e1b7d9a42", "name": null, "sharded": true }

shardCount is optional. Omit it, or set it to 1, for a non-sharded run. Set it to 2 through 40 to split the selected tests across parallel shards. Each shard runs one Playwright worker, so parallelism is controlled by shardCount alone.

Sharding composes with autoHeal: once the shards are merged, a merged run that ends failed is healed exactly as a non-sharded run would be. If the shards never merge (stale checksumai, see above), the heal is never evaluated.

Poll for Results by Run ID

# Check status
curl https://api.checksum.ai/public-api/v1/execution/status/run/9f2c7a4e-8b31-4d6a-a2f0-3c5e1b7d9a42 \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"
# Response: { "isTerminal": true, "verdict": "pass", "status": "passed", "passed": 45, "failed": 0, "recovered": 1, "bug": 0 }

For CI checks, poll until isTerminal is true, then pass the workflow only when verdict is "pass".

API-triggered runs can’t currently be cancelled through the public API. Cancelling the workflow (for example via concurrency: cancel-in-progress) stops your status polling, but the Checksum run continues until it finishes.

Sharded PR Run with `curl`

For pull-request checks that need a PR branch and preview URL, use the v2 grep endpoint. It accepts branch, envOverrides, and shardCount in one request.

Update checksumai before enabling sharding. After all shards finish, Checksum combines their results using the checksumai version installed on the branch you run against. If that version is older than 4.4.0 (the minimum production-supported version for sharded runs), the merge can fail to complete, delaying the run’s terminal verdict well past a normal run’s duration. Update checksumai on your tests branch to 4.4.0 or later (npm install checksumai@latest is the safe default), commit it, and re-run.

envOverrides is for your own application variables only. Names reserved by Checksum, and any key starting with CHECKSUM_, are rejected with 400.

name: Checksum sharded PR tests

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  checksum:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Trigger sharded Checksum run
        id: trigger
        run: |
          RESPONSE=$(curl -sf -X POST https://api.checksum.ai/public-api/v2/execution/grep \
            -H "Authorization: Bearer ${{ secrets.CHECKSUM_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d "$(jq -n \
              --arg grep "@smoke" \
              --arg branch "${{ github.head_ref }}" \
              --arg baseUrl "https://pr-${{ github.event.pull_request.number }}.preview.example.com" \
              '{
                grep: $grep,
                branch: $branch,
                envOverrides: { BASE_URL: $baseUrl },
                shardCount: 8
              }')")
          echo "run_id=$(echo "$RESPONSE" | jq -r .runId)" >> "$GITHUB_OUTPUT"

      - name: Poll Checksum verdict
        run: |
          RUN_ID="${{ steps.trigger.outputs.run_id }}"
          while true; do
            STATUS=$(curl -sf "https://api.checksum.ai/public-api/v1/execution/status/run/$RUN_ID" \
              -H "Authorization: Bearer ${{ secrets.CHECKSUM_API_KEY }}")
            echo "$STATUS" | jq '{status, phase, isTerminal, verdict, passed, failed, recovered, bug}'

            if [[ "$(echo "$STATUS" | jq -r .isTerminal)" == "true" ]]; then
              break
            fi

            sleep 15
          done

          [[ "$(echo "$STATUS" | jq -r .verdict)" == "pass" ]]

Use the v2 grep endpoint for PR-scoped sharded runs. The v1 execution endpoints can also shard, but they run against the project’s configured branch and environment.

Get Detailed Results

curl https://api.checksum.ai/public-api/v1/test-runs/<RUN_ID>/results \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"

Trigger Auto-Healing

curl -X POST https://api.checksum.ai/public-api/v1/auto-heal \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "testRunId": "<RUN_ID>",
    "autoCreatePR": true,
    "branch": "main",
    "prNumber": 42,
    "repoName": "<OWNER>/<REPO>"
  }'
# Response: { "batchId": "...", "sessionIds": [...], "failureCount": 3, "testIds": [...] }

Poll Healing Progress

curl https://api.checksum.ai/public-api/v1/auto-heal/batch/<BATCH_ID> \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"

For the complete API documentation, see API Reference.

Auto-healing failed CI runs

Checksum can fix failing tests after a CI run and open a PR with the fixes. Enable auto-heal per run or for your whole app:

Per-run, from the test command

Add the auto-heal flag to your test command. This works the same on GitHub Actions and GitLab CI — the repository, branch, and pull-request number are detected automatically from the CI environment:

- name: Run Checksum tests
  run: npx checksumai test --cksm-auto-heal
  env:
    CHECKSUM_API_KEY: ${{ secrets.CHECKSUM_API_KEY }}
    BASE_URL: ${{ secrets.BASE_URL }}
    CI: true

When the run finishes with failures, healing starts automatically and (by default) opens a PR with the fixes. See the CLI Reference for the full --cksm-auto-heal* flag set, and Auto-Healing for how healing works.

App-wide

Checksum can enable automatic healing for your whole app, so every failing CI run is healed without adding flags to each pipeline. Contact your Checksum representative to turn this on for your app.

When app-wide healing is enabled, tell Checksum which runs to treat as CI runs by setting this environment variable in your pipeline:

env:
  CHECKSUM_RUNTIME_REQUEST_REASON: cicd

With that set, every failing run from this pipeline is healed automatically — no per-command flags needed.

Running Strategies

Strategy	When to Use	Configuration
Manual trigger	Testing and debugging	`workflow_dispatch` (GitHub) / `when: manual` (GitLab)
On schedule	Regular health checks	`schedule: cron` (GitHub) / `$CI_PIPELINE_SOURCE == "schedule"` (GitLab)
On PR merge	Verify after deployments	`push: branches: [main]` (GitHub) / `$CI_PIPELINE_SOURCE == "push"` (GitLab)

Start with manual triggers to verify your setup, then add scheduled or merge-triggered runs.

Getting Started

Test Creation

CLI & CI

Maintenance & Monitoring

Integrations

Security & SSO

Overview

GitHub Actions

Basic Setup

Cross-Repo Setup

Setting Up Secrets

Testing the Action

GitHub Action (no CLI install)

Quick start

Execution modes

Auto-heal on failure

Wait for completion (gate the workflow)

Sharding

Per-PR preview environments

Full reference

GitLab CI/CD

Basic Setup

Cross-Repo Setup

Setting Up CI/CD Variables

Programmatic CI Integration (API)

Run Tests

Poll for Results by Run ID

Sharded PR Run with `curl`

Get Detailed Results

Trigger Auto-Healing

Poll Healing Progress

Auto-healing failed CI runs

Per-run, from the test command

App-wide

Running Strategies

​Overview

​GitHub Actions

​Basic Setup

​Cross-Repo Setup

​Setting Up Secrets

​Testing the Action

​GitHub Action (no CLI install)

​Quick start

​Execution modes

​Auto-heal on failure

​Wait for completion (gate the workflow)

​Sharding

​Per-PR preview environments

​Full reference

​GitLab CI/CD

​Basic Setup

​Cross-Repo Setup

​Setting Up CI/CD Variables

​Programmatic CI Integration (API)

​Run Tests

​Poll for Results by Run ID

​Sharded PR Run with curl

​Get Detailed Results

​Trigger Auto-Healing

​Poll Healing Progress

​Auto-healing failed CI runs

​Per-run, from the test command

​App-wide

​Running Strategies

Overview

GitHub Actions

Basic Setup

Cross-Repo Setup

Setting Up Secrets

Testing the Action

GitHub Action (no CLI install)

Quick start

Execution modes

Auto-heal on failure

Wait for completion (gate the workflow)

Sharding

Per-PR preview environments

Full reference

GitLab CI/CD

Basic Setup

Cross-Repo Setup

Setting Up CI/CD Variables

Programmatic CI Integration (API)

Run Tests

Poll for Results by Run ID

Sharded PR Run with `curl`

Get Detailed Results

Trigger Auto-Healing

Poll Healing Progress

Auto-healing failed CI runs

Per-run, from the test command

App-wide

Running Strategies