Fixing Flaky Playwright Tests Against Preview Deploys

Running end-to-end tests against a preview deploy is the right idea: it tests the site as it will actually be served — through the host, the CDN, the redirects and the headers — rather than a local build. It is also where Playwright suites become flaky. On the documentation site in this guide, 41 Playwright tests ran against each pull request's Netlify preview, and about one run in six failed with a test that passed on retry. The team had started merging over red checks, which defeats the point.

This guide brings that failure rate below 1% by fixing the causes: tests running before the right deploy was live, cold CDN caches, timing assumptions in the tests, third-party scripts, and shared state between tests. It belongs to Testing Static Sites in CI; the suite itself is described in End-to-End Tests for Static Sites with Playwright, and build-level flakiness in Fixing Flaky Static Site Builds in GitHub Actions.

Step 1: Measure Which Tests Flake and How

Playwright's JSON reporter records each test's attempts. Over 200 CI runs, tests that failed and then passed on retry were counted, and their traces (recorded on first retry with trace: 'on-first-retry') were inspected:

Cause 1: Testing Before the Preview Is Live

The workflow waited for Netlify's deploy-preview status check and then started tests. Netlify reports the deploy as ready slightly before every edge serves it, and on a busy day the old preview for the same pull request was still served for a few seconds. Fourteen failures were assertions about a change that was not there yet. The fix: the build writes the commit into the HTML, and the test setup polls until the preview reports the commit under test:

// tests/global-setup.ts
export default async () => {
  const want = process.env.GITHUB_SHA!.slice(0, 7);
  for (let i = 0; i < 30; i++) {
    const html = await (await fetch(`${process.env.PREVIEW_URL}/?cb=${Date.now()}`)).text();
    if (html.includes(`name="build-id" content="${want}`)) return;
    await new Promise((r) => setTimeout(r, 2000));
  }
  throw new Error(`Preview never served ${want}`);
};

The cache-busting query parameter avoids reading a cached copy of the old page. The setup fails clearly if the preview never updates, which is a deploy problem rather than a test problem.

Cause 2: Cold Caches on Large Assets

The search tests opened the search dialog and expected results within the default 5-second timeout. On a fresh preview, the first request for the 900 KB search index went to the origin, not the edge cache, and occasionally took over 5 seconds. Two changes: the global setup warms the preview by requesting the search index and the largest assets once, and search tests use a 10-second timeout on the first result. Production readers never hit this — production caches are warm — so the longer timeout does not hide a real problem; a separate performance test measures search latency against production.

Cause 3: Racing Animations and Rendering

Nine failures came from tests asserting during a 250 ms menu animation or before a client-rendered component appeared. The pattern in each was a manual wait or an immediate read:

// flaky
await page.click('#menu-button');
await page.waitForTimeout(200);
expect(await page.isVisible('nav a[href="/guides/"]')).toBe(true);

// stable
await page.getByRole('button', { name: 'Menu' }).click();
await expect(page.getByRole('link', { name: 'Guides' })).toBeVisible();

Web-first assertions such as toBeVisible() retry until the condition holds or the timeout expires. Role-based locators also survive markup changes better. The test project also sets reducedMotion: 'reduce', and the site honours it by disabling transitions, which removes animation timing from tests altogether.

Cause 4: Third-Party Scripts

Previews loaded the same analytics, consent and chat scripts as production. Twice the consent banner covered a button the test clicked; four times a third-party script error was caught by a test that failed on any console error. Tests now block third parties by default:

await context.route(/plausible\.io|consent-vendor|chat-vendor/, (route) => route.abort());
await context.addCookies([{ name: 'consent', value: 'necessary', url: process.env.PREVIEW_URL! }]);

A small, separate test project exercises the consent banner and chat launcher with third parties allowed; those tests are tagged and allowed one retry. Testing Search and Navigation Flows With Playwright uses the same setup.

Cause 5: Shared State Between Tests

Four failures came from a theme test that set localStorage.theme = 'dark', which leaked into later tests in the same worker when a test reused a page. Each test now uses Playwright's default fresh browser context — the leak came from a helper that reused one page across tests for speed. The speed gain was 4 seconds over the suite; not worth the flakiness.

Reading Traces Instead of Guessing

The single most useful habit was opening the trace of every retried test before changing anything. Playwright's trace viewer (npx playwright show-trace trace.zip) shows a screenshot and DOM snapshot before and after each action, the network log and the console. For the "old deploy" failures, the snapshot showed the previous version of the page, which pointed straight at deployment timing rather than the test. For the cache failures, the network tab showed one request to the search index taking 6.2 seconds while everything else was fast. Without traces, several of these had been "fixed" earlier by raising timeouts across the suite, which made genuine failures slower to report and left the causes in place. CI uploads the trace files as artifacts for seven days, and the tracking issue links them, so whoever picks up a flaky test starts from evidence.

Step 2: Keep Retries Honest

retries: 1 remains in CI, so a genuine blip does not fail a pull request, and a trace is recorded for the retry. A small script reads the JSON report after each run and posts any test that needed a retry to a tracking issue; a test that needs retries three times in a week gets fixed or quarantined with an owner.

Measured Impact

Pitfalls & Rollback

• Starting tests when the host says "ready". Wait for the preview to serve your commit.
• Default timeouts on cold previews. Warm large assets first, or allow more time for first loads.
• Fixed waits and one-shot reads. Use web-first assertions.
• Third parties in every test. Block them except where they are under test.
• Rollback: each fix is in test setup or individual tests; the weekly flaky report shows whether reverting one brings failures back.

Conclusion

Playwright tests against preview deploys flake for reasons that have nothing to do with the site under test: the old build still being served, cold caches, timing assumptions, third-party scripts and leaked state. Waiting for the preview to report the right commit, warming large assets, using web-first assertions and role locators, blocking third parties and isolating tests took flaky runs from 17% to under 1%, and the team stopped merging over red checks.

FAQ

Why are Playwright tests flakier against preview deploys than locally?

Because a preview adds a network, a CDN with cold caches, deployment timing and third-party scripts that the local build may not load. Tests that assume a page is ready after load, or that the URL already serves the new build, pass locally and fail intermittently against the preview.

How do I make sure tests run against the right deploy?

Wait for the preview to report the commit under test before starting. Expose the build's commit in a meta tag or a small JSON file, poll it until it matches the commit being tested, then run the suite. Hosts that post a deployment status to GitHub also let the workflow trigger only when that deployment succeeds.

Should flaky tests be retried?

Playwright's retries are useful for recording traces of failures and for absorbing rare network blips, but a test that only passes on retry is still flaky. Keep retries at one in CI, track which tests needed them, and fix those tests rather than raising the retry count.

How do I stop third-party scripts making tests flaky?

Block or stub them in tests that do not exercise them, using route interception to abort requests to analytics, chat and consent domains or to return a fixture. Test third-party integrations in a small, dedicated set of tests where their behaviour is the point.

Related

• Parent: Testing Static Sites in CI — the testing pipeline.
• End-to-End Tests for Static Sites with Playwright — the suite these fixes apply to.
• Running Smoke Tests Against a Preview URL — lighter checks on the same previews.
• Fixing Flaky Static Site Builds in GitHub Actions — flakiness before the tests run.
• Visual Regression Testing on Preview Deploys — screenshot tests with similar timing issues.