The shape of it
Each automated test that covers a check carries that check's ref in its name. After the run, a short script reads the test report and records pass or fail against each ref on the platform the run covered. Tests without a ref are ignored, so you can adopt this one check at a time.
Make a key for it
Create a tester key — CI · nightly, say — scoped to the app the suite tests, and add it to your CI secrets as QARUNBOOK_KEY. Results will read CI · nightly via API in the grid, so nobody mistakes a machine's pass for a person's.
Playwright
Add qarunbook's reporter. It comes inside Download automated tests (App configuration, Checks) as qarunbook-reporter.ts; copy it into your own project to use it there. Each Playwright project is a platform, named by its code.
// playwright.config.ts — one project per platform (WL, WM…); results go onto the checks
export default defineConfig({
reporter: [['list'], ['./qarunbook-reporter.ts']],
projects: [{ name: 'WL', use: { baseURL: 'https://staging.example.com' } }],
});
// then, in CI:
// QARUNBOOK_API_KEY=$QARUNBOOK_KEY QARUNBOOK_APP="My app" npx playwright testCypress
Add qarunbook-cypress.js (from the same download) to your project and hook it in. Set QARUNBOOK_PLATFORM to the platform the run covers.
// cypress.config.js
const qarunbook = require('./qarunbook-cypress');
module.exports = defineConfig({
e2e: { setupNodeEvents(on, config) { qarunbook(on, config); } },
});
// then, in CI:
// QARUNBOOK_API_KEY=$QARUNBOOK_KEY QARUNBOOK_APP="My app" QARUNBOOK_PLATFORM=WL npx cypress runAny other framework: JUnit XML
Almost every test runner can write a JUnit XML report. Send it as it is; each test is matched to a check by the ref in its name.
# Any test framework that writes JUnit XML (Jest with jest-junit, pytest --junitxml,
# JUnit/TestNG, Cypress, Playwright, Jenkins, Bitbucket…):
curl -X POST "https://qarunbook.com/api/v1/results/batch?app=My%20app&platform=WL" \
-H "Authorization: Bearer $QARUNBOOK_KEY" \
-H "Content-Type: application/xml" \
--data-binary @results.xml// Jenkinsfile — after the tests, send the JUnit report
post {
always {
withCredentials([string(credentialsId: 'qarunbook-key', variable: 'QARUNBOOK_KEY')]) {
sh 'curl -sf -X POST "https://qarunbook.com/api/v1/results/batch?app=My%20app&platform=WL" -H "Authorization: Bearer $QARUNBOOK_KEY" -H "Content-Type: application/xml" --data-binary @results.xml'
}
}
}
# bitbucket-pipelines.yml — the same curl as a step's last script line, with QARUNBOOK_KEY a secured variableYour own script
Or read any report yourself and send the results. This one reads Playwright's JSON reporter.
// scripts/report-to-qarunbook.mjs
// Reads Playwright's JSON report and records one result per tagged test.
import { readFileSync } from "node:fs";
const report = JSON.parse(readFileSync("results.json", "utf8"));
const platform = process.env.QA_PLATFORM ?? "WL";
// A test opts in by carrying its check ref in the title: "AUTH-01 signs in".
const REF = /\b([A-Z]{2,8}-\d{2,})\b/;
for (const suite of report.suites) {
for (const spec of suite.specs ?? []) {
const ref = spec.title.match(REF)?.[1];
if (!ref) continue;
const res = await fetch("https://qarunbook.com/api/v1/results", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.QARUNBOOK_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
app: "Checkout",
check: ref,
platform,
result: spec.ok ? "pass" : "fail",
}),
});
const { data, error } = await res.json();
console.log(ref, platform, error ? `error: ${error.message}` : data.status);
}
}In GitHub Actions
# .github/workflows/e2e.yml
name: e2e
on: [push]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 22 }
- run: npm ci && npx playwright install --with-deps
- run: npx playwright test --reporter=json > results.json
continue-on-error: true
- run: node scripts/report-to-qarunbook.mjs
if: github.ref == 'refs/heads/main'
env:
QARUNBOOK_KEY: ${{ secrets.QARUNBOOK_KEY }}
QA_PLATFORM: WLReport from your main branch only. A red run on somebody's half-finished branch is not news about the release, and would fail checks your testers passed an hour ago.
Clearing retests after a deploy
When a developer marks an issue fixed, its check reads retest until somebody confirms the fix. If a test covers that check, the first run after the deploy is the confirmation — its result clears the retest on its own. Checks no test covers stay in the queue for a person, which is exactly where they belong.
Staying under the limit
A key may make 120 requests a minute. A suite with more tagged tests than that should pause when X-RateLimit-Remaining reaches zero, or retry on 429 after Retry-After seconds — see Rate limits & errors.