Running Tests
Run test suites across Chromium, Firefox, and WebKit. Live results, auto-heal, AI triage, single-test re-runs, and CI triggering.
Running Tests
Once you have test suites — generated automatically from a scan or built by hand — running them is one or two clicks. This guide covers the run modal, what happens during a run, the run detail page, and how AegisRunner handles failures.
Plan limits at a glance
| Plan | Test runs / month | Concurrent runs | Result retention |
|---|---|---|---|
| Free | 50 | 1 | 7 days |
| Starter | 500 | 2 | 30 days |
| Pro | 2,500 | 5 | 90 days |
| Business | 10,000 | 10 | 1 year |
| Enterprise | Unlimited | Unlimited | Unlimited |
Hitting the monthly cap shows an upgrade banner; the run is rejected with a 402. Hitting the concurrency cap queues the run until a slot frees up.
Running a suite
Click Test Suites in the sidebar. Use search or filter by scan to find the suite. Or run from inside a suite — open it and the Run button is on the header.
The Run modal opens. Pick which browsers you want to run on — one, two, or all three.
You can select any combination of Chromium, Firefox, and WebKit. Each browser runs as its own run; the run page shows them as separate browser tabs in the same view.
Click Start Run. You're redirected to the run page; results stream in live.
Browsers
| Browser | Engine | When to use it |
|---|---|---|
| Chromium | Blink | Default. Mirrors Chrome and Edge — what most of your users have. |
| Firefox | Gecko | Catches Gecko-only quirks, especially around CSS and form behavior. |
| WebKit | WebKit | Mirrors Safari and iOS Safari. Catches the things you don't see until iPhone users complain. |
Tests written from a mobile-device scan run in mobile viewport on whatever browser you pick.
What happens during a run
- The run is queued and a worker picks it up — usually within a couple of seconds.
- The browser launches in stealth mode (so anti-bot middleware doesn't trip), with the project's pre-auth cookies and login script applied if configured.
- Steps execute one by one. Each step's result is streamed to the run page in real time.
- If a step's locator can't find its target, AegisRunner attempts auto-heal — see below.
- If visual regression is on, screenshots are captured per step and diffed against the baseline.
- When the run finishes, you get an email and any configured Slack/Discord/Teams/webhook notifications.
The run detail page
Tabs across the top:
- Results — every test case with pass/fail/skip and duration. Click into a case to see its steps, screenshots, and assertion details.
- Triage — appears only if the run had failures. AI explains each failure: real regression, flake, environment issue, or auto-healed-and-passed. See Failure Triage.
- Recording — the full video of the run (paid plans).
- Issues — appears if any tests created GitHub or Jira tickets via your issue-tracking integration.
- Summary — pass-rate chart, slowest steps, browser breakdown.
- Logs — raw worker logs for the entire run. Useful when something weird happens.
Filtering results
The Results tab has filters at the top:
- All / Passed / Failed / Skipped
- Regressions only — tests that passed before and started failing in this run.
- New failures only — tests that have never passed.
Re-running a single failure
Open any failed test and click Re-run — only that one test runs, in the same browser, with the same setup. Avoids the 5-minute round trip of re-running the whole suite while you're iterating on a fix.
Auto-heal
UI changes — a button gets renamed, an icon moves from one container to another, a class is renamed. Without intervention, every locator that hit that element would break.
When a step's locator fails, AegisRunner:
- Takes an accessibility snapshot of the live page.
- Looks for the closest match by role + visible text (with a strict similarity threshold).
- If a confident match is found, the step continues against the new locator and the suite is updated for next time.
- If no confident match is found, the step fails and shows up in the Triage tab as a real failure.
You'll see a healed badge next to any step that auto-healed, with a link to the locator change. Auto-heal is on by default for all paid plans. See Debugging Failed Tests for the details.
Visual regression
Toggle Visual regression in the run modal to capture and compare screenshots step by step. The first run with visual regression on becomes the baseline; subsequent runs diff against it. Review diffs in the step detail panel — accept (update the baseline) or reject (the change is a bug). See Visual Regression Testing.
API steps
Suites can mix UI steps with api-request steps that hit REST or GraphQL endpoints directly. No browser involved — just an HTTP call with assertions on status code, JSON body fields (dot-path), and response time. See API Testing.
Exporting as Playwright
Pro and above can export any suite as a standalone Playwright spec file:
- Open the suite, click Export → Playwright.
- Download the
.spec.ts. - Drop it into your repo and run with
npx playwright test, or wire it into your CI.
The export includes step logic, assertions, and any test data, but doesn't include AegisRunner's auto-heal — once it's running locally, locator drift is on you.
Running from CI
POST to /api/v1/ci/trigger with a CI trigger token to kick off a run from any pipeline. See CI/CD Integration.
Related
- Debugging Failed Tests — what to do with a red run.
- Failure Triage — how the AI classifies failures.
- Scheduling Automated Tests — run on a cadence.
- Test Data Management — configure inputs for forms.
- Visual Regression Testing
- API Testing