Skip to content

Test Automation

How to Handle Flaky Tests in Playwright: A Triage Process That Actually Works

JL Jeremy Luong · Author
10 min read
Flaky tests in Playwright: a triage process that actually works (cover image)

Playwright flaky tests pass and fail on the same code, and they’re the most expensive tests you can own. Every red build that turns out to be “just flaky” teaches the team to ignore red builds. Once that habit sets in, your suite stops protecting you, and it still costs you CI minutes.

I’m not the only one bothered by this. The State of JS 2025 testing results list it among the top four testing pain points. The other three are mocking, configuration, and long run times.

The problem isn’t new, either. In 2016, Google reported that about 1.5% of all its test runs were flaky. About 16% of its tests showed some flakiness (Google Testing Blog).

Most flaky-test advice stops at “turn on retries.” Retries manage the symptom, though; they don’t cure it. So here’s the triage process I’d run to drive flaky tests in Playwright down and keep them down.

Step 1: Make Playwright flaky tests visible first

You can’t triage what you don’t measure. Playwright sorts every test into one of a few outcomes, and a test that fails and then passes on retry is reported as flaky, not passed. That’s how Playwright flaky tests announce themselves, and that label is your raw signal. Most teams just never look at it.

Diagram of the three outcomes for flaky tests in Playwright: passed on the first attempt, flaky when it failed and then passed on retry, and failed on every attempt, with flake rate defined as flaky outcomes divided by total test executions
A test that passes only on retry is reported as flaky. Count those to get your flake rate.

Start with a config that retries only in CI and captures evidence on the first retry:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [
    ['html', { open: 'never' }],
    ['json', { outputFile: 'test-results/results.json' }],
  ],
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
});

The JSON reporter is what makes measurement cheap. A short script counts flaky outcomes per test on every CI run:

// scripts/flake-report.ts
import fs from 'node:fs';

const report = JSON.parse(fs.readFileSync('test-results/results.json', 'utf8'));
const flaky: string[] = [];

function walk(suite: any, path: string[] = []) {
  for (const spec of suite.specs ?? []) {
    for (const test of spec.tests ?? []) {
      if (test.status === 'flaky') flaky.push([...path, spec.title].join(' › '));
    }
  }
  for (const child of suite.suites ?? []) walk(child, [...path, child.title]);
}

report.suites.forEach((s: any) => walk(s, [s.title]));
console.log(`Flaky tests this run: ${flaky.length}`);
flaky.forEach((t) => console.log(` - ${t}`));

Push that output somewhere persistent, like a CI artifact, a small database table, or a Slack message. You want a history per test, not a one-off snapshot.

The one KPI worth tracking: flake rate

Flake rate = flaky outcomes ÷ total test executions over a rolling window (a common window is the last 14 days of main-branch runs). Track it for the whole suite and for each test. A suite-level number tells leadership whether things are improving. The per-test number tells you where to spend the next hour.

Don’t set the target at zero on day one. Set it at “lower than last sprint,” and make the top five offenders a standing item in your team’s backlog.

Step 2: Reproduce on purpose

A test that flakes once a week in CI is hard to debug. One that flakes once in twenty local runs is easy. Playwright gives you the tools to reproduce them on demand:

# Hammer one test to expose intermittent failures
npx playwright test tests/checkout.spec.ts --repeat-each=20 --workers=4

# Fail the build if anything only passed on retry (great for PR gates)
npx playwright test --fail-on-flaky-tests

# Same gate, limited to tests affected by your branch
npx playwright test --only-changed=origin/main --fail-on-flaky-tests

# Open the trace captured on the first retry
npx playwright show-trace test-results/<test-folder>/trace.zip

Raising --workers while repeating is deliberate. Many flakes only show up under parallel load, when tests compete for shared data or a slow backend. If a test only fails with more workers, you’ve already narrowed the root cause to isolation or timing.

The trace viewer is the biggest advantage I’ve found in Playwright for this work. It gives you the DOM snapshot before and after each action, network calls, console logs, and the exact locator that resolved. Read the failed attempt and the passing retry side by side. The difference between them is usually the bug.

Step 3: The decision tree (retry, quarantine, or fix)

Every flaky test gets exactly one of three outcomes. To decide, run through these questions in order:

  1. Is the failure caused by the product, not the test? Think of a real race condition in the app, an API that sometimes returns 500, or a UI that renders twice. If so, it’s a bug. File it against the product and keep the test red, because the test is doing its job.
  2. Can you identify the root cause from the trace within about 30 minutes? If so, fix it now (see Step 5). That’s almost always cheaper than any other option.
  3. Does the test cover a critical user path, like login, checkout, or payment? If so, don’t quarantine it. Keep it in the main run with retries, assign an owner, and give it a fix-by date. Losing coverage on critical paths is worse than a noisy build.
  4. Otherwise, quarantine it with an owner and an expiry date (see Step 4).
Decision tree: if the product causes the failure, file a bug and keep the test red; if the root cause is clear from the trace in about 30 minutes, fix it now; if it covers a critical path, keep it in the main run with retries, an owner and a fix-by date; otherwise quarantine it with an owner and an expiry date
Ask the questions in order. Retries buy time; they never close the ticket.

Retries are never the fix. They’re a temporary cushion while you work through steps 2 to 4. If a test needs retries for more than a sprint, it has quietly become a quarantined test, and nobody decided that on purpose.

Step 4: A quarantine policy that expires

Quarantine done wrong is just deletion with extra steps. Teams tag flaky tests in Playwright, move them out of the gate, and forget them. A solid rule: a quarantined test has an owner, a ticket, and an expiry date, and it still runs.

Quarantine flow: tag the test @quarantine with owner, issue and expires annotations, run it in a non-blocking job with --grep @quarantine while the blocking gate uses --grep-invert @quarantine, and when the expiry date passes a CI check fails so the team fixes, extends or deletes the test
Quarantined tests keep running outside the merge gate until their expiry date forces a decision.

Playwright’s tag support makes this clean:

import { test, expect } from '@playwright/test';

test('applies a discount code at checkout', {
  tag: '@quarantine',
  annotation: [
    { type: 'owner', description: 'payments-team' },
    { type: 'issue', description: 'QA-1234' },
    { type: 'expires', description: '2026-11-15' },
  ],
}, async ({ page }) => {
  // ...
});

Then split the pipeline so quarantined tests still run but don’t block merges:

# Blocking gate: everything except quarantine
npx playwright test --grep-invert @quarantine

# Non-blocking job: quarantine only, results still collected
npx playwright test --grep @quarantine || true

Add a small CI check that fails if any expires annotation is in the past. That forces a decision when the date passes: fix it, extend it with a reason, or delete it because the coverage isn’t worth having. Any of those is fine. Leaving it to rot is not.

Step 5: The root causes behind most Playwright flaky tests

When teams audit a suite, most of the flakiness falls into a few buckets. Here they are, with the fix for each one.

Five common causes of flakiness mapped to fixes: hard waits to web-first assertions, brittle selectors to getByRole, getByLabel and getByTestId, not waiting for the network to waitForResponse, expect.poll and toPass with a timeout, shared state to per-test data in a fixture, and environment noise to page.route and pinned timezoneId and locale
Five buckets cover most of the flakiness you will find in an audit.

Hard waits and manual timing

page.waitForTimeout(2000) is a guess about how slow your app is. In CI, it eventually guesses wrong. Use web-first assertions instead. They retry until the condition is true or the timeout expires:

// Flaky
await page.click('#save');
await page.waitForTimeout(2000);
expect(await page.textContent('.toast')).toBe('Saved');

// Stable
await page.getByRole('button', { name: 'Save' }).click();
await expect(page.getByRole('status')).toHaveText('Saved');

Brittle selectors

CSS chains like div > div:nth-child(3) > span break when someone adds a wrapper div. Prefer getByRole, getByLabel, and getByTestId. Locators also resolve lazily, at the moment of action, so they don’t go stale the way stored element handles do.

Not waiting for the network

If the UI depends on an API response, wait for that response, not for a time interval:

const saved = page.waitForResponse(
  (r) => r.url().includes('/api/orders') && r.request().method() === 'POST'
);
await page.getByRole('button', { name: 'Place order' }).click();
expect((await saved).ok()).toBeTruthy();

For eventually-consistent backends, expect.poll or expect(...).toPass() is a clean way to wait for a state rather than a duration. Give toPass() a timeout, as in toPass({ timeout: 10_000 }). Without one it defaults to zero, so a block that never passes runs to the full test timeout instead of failing fast.

Shared state between tests

This is the cause behind tests that only fail under parallel load. Two workers edit the same user, one test leaves a cart full of items, or test order matters. Fixes:

  • Give every test its own data: create it through the API in a fixture, and clean it up afterwards.
  • Reuse authentication with storageState instead of logging in through the UI each time, but never share mutable accounts across workers.
  • Avoid test.describe.configure({ mode: 'serial' }) as a fix. It hides coupling instead of removing it.

(The relative URLs below assume baseURL is set in your config.)

// fixtures.ts
import { test as base } from '@playwright/test';

export const test = base.extend<{ order: { id: string } }>({
  order: async ({ request }, use) => {
    const res = await request.post('/api/test-data/orders', { data: { items: 1 } });
    const order = await res.json();
    await use(order);
    await request.delete(`/api/test-data/orders/${order.id}`);
  },
});

Environment and third parties

Animations, time zones, third-party scripts, and shared staging environments all add noise. Mock what you don’t own with page.route(), pin timezoneId and locale in the config, and keep end-to-end tests that hit real third parties to a small, clearly labeled set.

A word on AI “healers”

AI is making this problem bigger, not smaller. As coding agents speed up delivery, more of what we test is probabilistic. Vendors are calling that out too. Applitools notes that non-deterministic AI systems can produce inconsistent results. That shows up as flaky tests and false positives (Applitools).

Self-healing is now built into the tooling. Playwright’s Test Agents ship a healer that re-runs failing tests and patches them (docs). Other vendors offer similar features, and they’re useful for routine locator updates.

A healer that “fixes” a test by loosening an assertion can hide a real regression. Keeping self-healing tests “true to their intent” was on the agenda at CypressConf in October. That shows how common the problem is. Treat every healed test like a pull request from a new teammate: review the diff, and ask whether the test still checks what it was meant to check.

Your one-week starting plan

  • Day 1: Turn on the JSON reporter and trace: 'on-first-retry', and start logging flaky outcomes per test.
  • Days 2 and 3: Pick the five tests with the highest flake rate. Reproduce each with --repeat-each and read the traces.
  • Day 4: Put each one through the decision tree. Fix, quarantine with an expiry date, or file a product bug.
  • Day 5: Add --fail-on-flaky-tests to PR runs. The flag applies to the whole run, so pair it with --only-changed=origin/main if you only want to gate new and changed tests. Either way, new flakiness can’t sneak in. --only-changed needs origin/main in the local clone, and a default shallow CI checkout won’t have it, so set fetch-depth: 0 or fetch main first.

Repeat this every sprint. Playwright flaky tests then stop being a mystery, and the flake rate becomes a number you manage. If you want to compare notes with other practitioners, the r/Playwright and r/QualityAssurance threads on flaky tests are good reading.

Keep reading

If your team is losing hours every week to red builds that “just need a re-run,” the triage process above is a place to start. If you’re still working out whether Playwright is the right tool for your team, or how much of your coverage should be E2E in the first place, start with Playwright vs Selenium and contract testing vs E2E testing.

JL

Written by

Jeremy Luong

SDET Consultant

More about the author

Leave a Reply

Your email address will not be published.