Report results from CI — qarunbook API docs

Your automated tests already know whether sign-in works. Let them say so in the runbook, so the grid shows everything that has been checked — by people and by machines — in one place.

The shape of it

Each automated test that covers a check carries that check's ref in its name. After the run, a short script reads the test report and records pass or fail against each ref on the platform the run covered. Tests without a ref are ignored, so you can adopt this one check at a time.

Make a key for it

Create a tester key — CI · nightly, say — scoped to the app the suite tests, and add it to your CI secrets as QARUNBOOK_KEY. Results will read CI · nightly via API in the grid, so nobody mistakes a machine's pass for a person's.

The script

This one reads Playwright's JSON reporter. Any runner that can write a report works the same way.

scripts/report-to-qarunbook.mjs

// scripts/report-to-qarunbook.mjs
// Reads Playwright's JSON report and records one result per tagged test.
import { readFileSync } from "node:fs";

const report = JSON.parse(readFileSync("results.json", "utf8"));
const platform = process.env.QA_PLATFORM ?? "WL";

// A test opts in by carrying its check ref in the title: "AUTH-01 signs in".
const REF = /\b([A-Z]{2,8}-\d{2,})\b/;

for (const suite of report.suites) {
  for (const spec of suite.specs ?? []) {
    const ref = spec.title.match(REF)?.[1];
    if (!ref) continue;
    const res = await fetch("https://qarunbook.com/api/v1/results", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.QARUNBOOK_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        app: "Checkout",
        check: ref,
        platform,
        result: spec.ok ? "pass" : "fail",
      }),
    });
    const { data, error } = await res.json();
    console.log(ref, platform, error ? `error: ${error.message}` : data.status);
  }
}

In GitHub Actions

.github/workflows/e2e.yml

# .github/workflows/e2e.yml
name: e2e
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci && npx playwright install --with-deps
      - run: npx playwright test --reporter=json > results.json
        continue-on-error: true
      - run: node scripts/report-to-qarunbook.mjs
        if: github.ref == 'refs/heads/main'
        env:
          QARUNBOOK_KEY: ${{ secrets.QARUNBOOK_KEY }}
          QA_PLATFORM: WL

Report from your main branch only. A red run on somebody's half-finished branch is not news about the release, and would fail checks your testers passed an hour ago.

Clearing retests after a deploy

When a developer marks an issue fixed, its check reads retest until somebody confirms the fix. If a test covers that check, the first run after the deploy is the confirmation — its result clears the retest on its own. Checks no test covers stay in the queue for a person, which is exactly where they belong.

Staying under the limit

A key may make 120 requests a minute. A suite with more tagged tests than that should pause when X-RateLimit-Remaining reaches zero, or retry on 429 after Retry-After seconds — see Rate limits & errors.