Skip to content

Report results from CI

Your automated tests already know whether sign-in works. Let them say so in the runbook, so the grid shows everything that has been checked — by people and by machines — in one place.

The shape of it

Each automated test that covers a check carries that check's ref in its name. After the run, a short script reads the test report and records pass or fail against each ref on the platform the run covered. Tests without a ref are ignored, so you can adopt this one check at a time.

Make a key for it

Create a tester key — CI · nightly, say — scoped to the app the suite tests, and add it to your CI secrets as QARUNBOOK_KEY. Results will read CI · nightly via API in the grid, so nobody mistakes a machine's pass for a person's.

Playwright

Add qarunbook's reporter. It comes inside Download automated tests (App configuration, Checks) as qarunbook-reporter.ts; copy it into your own project to use it there. Each Playwright project is a platform, named by its code.

playwright.config.ts
// playwright.config.ts — one project per platform (WL, WM…); results go onto the checks
export default defineConfig({
  reporter: [['list'], ['./qarunbook-reporter.ts']],
  projects: [{ name: 'WL', use: { baseURL: 'https://staging.example.com' } }],
});

// then, in CI:
// QARUNBOOK_API_KEY=$QARUNBOOK_KEY QARUNBOOK_APP="My app" npx playwright test

Cypress

Add qarunbook-cypress.js (from the same download) to your project and hook it in. Set QARUNBOOK_PLATFORM to the platform the run covers.

cypress.config.js
// cypress.config.js
const qarunbook = require('./qarunbook-cypress');
module.exports = defineConfig({
  e2e: { setupNodeEvents(on, config) { qarunbook(on, config); } },
});

// then, in CI:
// QARUNBOOK_API_KEY=$QARUNBOOK_KEY QARUNBOOK_APP="My app" QARUNBOOK_PLATFORM=WL npx cypress run

Any other framework: JUnit XML

Almost every test runner can write a JUnit XML report. Send it as it is; each test is matched to a check by the ref in its name.

after the tests
# Any test framework that writes JUnit XML (Jest with jest-junit, pytest --junitxml,
# JUnit/TestNG, Cypress, Playwright, Jenkins, Bitbucket…):
curl -X POST "https://qarunbook.com/api/v1/results/batch?app=My%20app&platform=WL" \
  -H "Authorization: Bearer $QARUNBOOK_KEY" \
  -H "Content-Type: application/xml" \
  --data-binary @results.xml
Jenkins and Bitbucket
// Jenkinsfile — after the tests, send the JUnit report
post {
  always {
    withCredentials([string(credentialsId: 'qarunbook-key', variable: 'QARUNBOOK_KEY')]) {
      sh 'curl -sf -X POST "https://qarunbook.com/api/v1/results/batch?app=My%20app&platform=WL" -H "Authorization: Bearer $QARUNBOOK_KEY" -H "Content-Type: application/xml" --data-binary @results.xml'
    }
  }
}

# bitbucket-pipelines.yml — the same curl as a step's last script line, with QARUNBOOK_KEY a secured variable

Your own script

Or read any report yourself and send the results. This one reads Playwright's JSON reporter.

scripts/report-to-qarunbook.mjs
// scripts/report-to-qarunbook.mjs
// Reads Playwright's JSON report and records one result per tagged test.
import { readFileSync } from "node:fs";

const report = JSON.parse(readFileSync("results.json", "utf8"));
const platform = process.env.QA_PLATFORM ?? "WL";

// A test opts in by carrying its check ref in the title: "AUTH-01 signs in".
const REF = /\b([A-Z]{2,8}-\d{2,})\b/;

for (const suite of report.suites) {
  for (const spec of suite.specs ?? []) {
    const ref = spec.title.match(REF)?.[1];
    if (!ref) continue;
    const res = await fetch("https://qarunbook.com/api/v1/results", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.QARUNBOOK_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        app: "Checkout",
        check: ref,
        platform,
        result: spec.ok ? "pass" : "fail",
      }),
    });
    const { data, error } = await res.json();
    console.log(ref, platform, error ? `error: ${error.message}` : data.status);
  }
}

In GitHub Actions

.github/workflows/e2e.yml
# .github/workflows/e2e.yml
name: e2e
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci && npx playwright install --with-deps
      - run: npx playwright test --reporter=json > results.json
        continue-on-error: true
      - run: node scripts/report-to-qarunbook.mjs
        if: github.ref == 'refs/heads/main'
        env:
          QARUNBOOK_KEY: ${{ secrets.QARUNBOOK_KEY }}
          QA_PLATFORM: WL

Report from your main branch only. A red run on somebody's half-finished branch is not news about the release, and would fail checks your testers passed an hour ago.

Clearing retests after a deploy

When a developer marks an issue fixed, its check reads retest until somebody confirms the fix. If a test covers that check, the first run after the deploy is the confirmation — its result clears the retest on its own. Checks no test covers stay in the queue for a person, which is exactly where they belong.

Staying under the limit

A key may make 120 requests a minute. A suite with more tagged tests than that should pause when X-RateLimit-Remaining reaches zero, or retry on 429 after Retry-After seconds — see Rate limits & errors.

Something unclear or missing? Write to [email protected].