What changed
Run with AI is now Run tests. Each check gets an automated test written from the app's code: a Playwright test for websites, a Maestro flow for phone apps, and a Cypress test if the app uses Cypress. A run plays those tests exactly as written, with no AI and nothing to pay.
Each app chooses its tools in App configuration → Test configuration → Automated tests. Playwright, Cypress and Maestro each have their own switch and don't depend on each other: a Cypress team gets Cypress tests written from the code as their own suite, not a copy of the Playwright ones. Cypress tests run on a connected computer, which installs Cypress the first time. With every switch off, the checklist is for people only, and Run tests says so.
Before it starts, Run tests says exactly what will run on each platform, such as "Playwright and Cypress tests · your connected computer". A checklist written from code now comes with its tests, written on the server in the background, and an existing checklist can have its missing tests written from the code.
Runs in your own Chrome are gone, along with the extension they needed. Plugged-in iPhones on a Mac and Limrun iPhones run the saved Maestro flows too. Desktop and HarmonyOS runs are refused with a plain reason.
Why it matters
A failed test is no longer the same as a bug. Before a fail is recorded, the AI takes one look at the screen the test stopped on. If the check's expected result holds there, the test was wrong, so it's rewritten instead of a false issue being raised. When that look can't happen, the check is marked couldn't check, with the reason.
When the app outgrows a test (a button moved, a word changed), the test is rewritten from the page it stopped on and run again, at most twice. Each check's mark says whether a test or the AI settled it, and results from a test are signed 'Automated test', with the test's own failure message.
Who it's for
Teams that want their checklist to run on every build without paying for AI each time, and teams that already write Playwright, Cypress or Maestro tests. The tests download as a Playwright project, a Cypress project and a Maestro folder, and results from your own pipeline can be sent back as JUnit XML or JSON.
How to try it
Open an app, go to App configuration → Test configuration → Automated tests, and turn on the tools you use. Then press Run tests on the board.
What shipped
Runs play each check's automated test
Every check gets a Playwright test, a Maestro flow or a Cypress test written from your code, and Run tests plays them with no AI. The AI only writes a missing test, rewrites one the app outgrew, or does a check a test can't.
Choose your test tools per app
Playwright, Cypress and Maestro each have their own switch in Test configuration → Automated tests. Cypress tests are written as their own suite and run on your connected computer. Integrations has a new "Where tests run" group with all three and your own pipeline, which can send its results back with a Playwright reporter, a Cypress plugin, or JUnit XML and JSON.
Every change
New9
- Automated tests for every check: Playwright for websites, Maestro for phone apps and Cypress when it's on. View a check's tests, or download them as a Playwright project, a Cypress project and a Maestro folder.
- Test tool switches in App configuration → Test configuration → Automated tests: Playwright, Cypress and Maestro, each on or off by itself. With none on, the checklist is for people only.
- Cypress tests are written from the code as their own suite and run on a connected computer, which installs Cypress the first time.
- A "Where tests run" group in Integrations: Playwright, Cypress, Maestro and your own pipeline.
- Checklists written from code come with their tests, written on the server in the background. For an existing checklist, write the missing tests from the code.
- Results from your own suites: POST /api/v1/results/batch takes JUnit XML or JSON, with a Playwright reporter and a Cypress plugin.
- Tests sign in and sign up the app's own way, saved once per app for web and phones, and read codes and links from the test inbox. No password or code is ever written into a test.
- Plugged-in iPhones on a Mac run their checks' Maestro flows.
- Assistants can start a run and read its results over MCP (run_tests, run_results), switch test tools (automated_tests), and assign an issue or attach screenshots when they raise it.
Improved7
- Run with AI is now Run tests, and it says exactly what will run on each platform before it starts.
- Before a fail is recorded, the AI checks the screen the test stopped on. If the expected result holds there, the test is rewritten instead of a false issue being raised.
- A test the app outgrew is rewritten from the page it stopped on, at most twice. One that couldn't be rewritten (no AI credit, no connection) stays as it is for the next run.
- Each check's mark says Test when its automated test settled it and AI only when the AI did. A failed test's own message is kept in the result and the issue.
- Tests never change a saved account's details, since other checks use it at the same time. Those checks sign up a fresh account.
- The helper sets up Maestro and its Java by itself on computers that test phones.
- When the AI does a web check itself, it reads the page as text, with passwords blanked out, and keeps the last screen as evidence.
Fixed6
- A check whose test needs a login that isn't saved for that platform is left untested, saying which login to save. It used to fail and raise an issue.
- A phone test that runs out of time is a test that didn't finish, never a failed check. Phone flows now get 8 minutes.
- Phone runs on one computer take turns with Maestro, so two devices no longer cut into each other.
- Asking for an emailed code again waits for a newer email, so resend checks no longer wait forever when the server sends the same code.
- A web app's phone view runs at iPhone size in Chromium, and a browser that can't start is a run error, not a fail.
- Emulators always start fresh, since a saved snapshot could hang them.