Static Site Generators in Production

Visual Regression Testing on Preview Deploys

Static sites fail visually more often than they fail functionally. A CSS refactor shifts the sidebar on article pages, a dependency update changes the code block font, a new component pushes the footer over the content on narrow screens. Links still work, builds pass, and the problem is only noticed when a reader reports it. Reviewers looking at a preview deployment catch some of this, but nobody clicks through twelve templates at three screen sizes on every pull request.

Visual regression tests do that clicking. They take screenshots of key pages on the preview deployment, compare them with the same pages on main, and show the differences in the pull request. This guide sets them up with Playwright, chooses pages and viewports, removes the noise that makes teams abandon visual tests, and wires the results into review. It is part of Preview Environments for Pull Requests.

Prerequisites

What to Screenshot

A static site's pages come from a small number of templates, and a layout bug in a template shows up on every page that uses it. Cover templates, not pages:

Coverage grid of templates and viewports Seven templates - home, section index, article, long article with code and tables, search, 404 and a landing page - by three viewports - 375, 768 and 1280 pixels wide - give 21 screenshots, which run in under two minutes and cover every layout on the site. 7 templates × 3 widths = 21 screenshots 375 px 768 px 1280 px home section index article long article (code, tables) search 404 landing page add a dark-mode column if the site has a dark theme
Twenty-odd screenshots cover every layout; a thousand would cover the same layouts slower.

Pick URLs whose content is stable — a reference page that rarely changes is better than the newest blog post. Include at least one page with every component that has broken before: long code blocks, wide tables, callouts, image galleries, embedded video placeholders. Three widths — a phone, a tablet and a laptop — catch almost all responsive bugs. If the site has a dark theme, add it as a separate project with colorScheme: 'dark'.

Setting Up Playwright

Playwright's toHaveScreenshot compares a screenshot with a stored baseline and fails if they differ beyond a threshold:

// tests/visual.spec.ts
import { test, expect } from '@playwright/test';

const pages = ['/', '/docs/', '/docs/install/', '/docs/reference/config/', '/search/', '/missing-page/'];

for (const path of pages) {
  test(`visual ${path}`, async ({ page }) => {
    await page.goto(path, { waitUntil: 'networkidle' });
    await page.evaluate(() => document.fonts.ready);
    await expect(page).toHaveScreenshot({
      fullPage: true,
      animations: 'disabled',
      mask: [page.locator('[data-visual-mask]')],
      maxDiffPixelRatio: 0.001,
    });
  });
}

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
  use: {
    baseURL: process.env.PREVIEW_URL,
    extraHTTPHeaders: process.env.BYPASS_TOKEN ? { 'x-preview-bypass': process.env.BYPASS_TOKEN } : {},
  },
  projects: [
    { name: 'mobile', use: { ...devices['Pixel 7'] } },
    { name: 'tablet', use: { viewport: { width: 768, height: 1024 } } },
    { name: 'desktop', use: { viewport: { width: 1280, height: 800 } } },
  ],
});

Baselines are stored next to the test file, one per page and project. Commit them to the repository; they are the reference for what the site should look like.

Removing Noise

Visual tests are abandoned when they fail for reasons that are not regressions. Deal with each source of noise up front:

  • Font rendering. Different operating systems and even different Linux images render fonts differently. Always run visual tests in the official Playwright Docker image (mcr.microsoft.com/playwright) pinned to a version, both in CI and when updating baselines locally.
  • Animations and transitions. animations: 'disabled' stops CSS animations; also add a test-only stylesheet that sets scroll-behavior: auto and hides blinking cursors.
  • Lazy images and fonts. Wait for document.fonts.ready, scroll the page to trigger lazy images or set loading="eager" in a test mode, and wait for network idle.
  • Dynamic content. Dates, "last updated" lines, randomised testimonials and third-party embeds change between runs. Mark them with a data-visual-mask attribute in the templates and mask them.
False positive rate as noise sources are removed Percentage of visual test runs failing without a real regression. Default setup on a hosted runner: 34 percent. Pinned Playwright container: 11 percent. Animations disabled and fonts awaited: 4 percent. Dynamic regions masked: under 1 percent. Runs failing with no real regression 34% defaults 11% pinned container 4% + no animations, fonts <1% + masks measured over 200 pull requests with no intended visual change
Each fix is small; together they turn an ignored check into a trusted one.

Running It on Previews

Add a job that waits for the preview deployment, then runs the tests against its URL:

visual:
  needs: preview
  runs-on: ubuntu-latest
  container: mcr.microsoft.com/playwright:v1.47.0-jammy
  steps:
    - uses: actions/checkout@v4
    - run: npm ci
    - run: npx playwright test tests/visual.spec.ts
      env:
        PREVIEW_URL: ${{ needs.preview.outputs.url }}
        BYPASS_TOKEN: ${{ secrets.PREVIEW_BYPASS_TOKEN }}
    - if: failure()
      uses: actions/upload-artifact@v4
      with:
        name: visual-diffs
        path: test-results/

Run the visual job in parallel with other preview checks such as link checks and smoke tests, not after them, so the slowest check sets the pace rather than their sum. Make it a required check only once false positives are rare; until then, report without blocking so the team builds trust in its results.

When a test fails, Playwright writes the expected, actual and diff images to test-results/. Upload them as an artifact and, ideally, post a comment on the pull request linking the HTML report so reviewers see the diffs without downloading anything.

Reviewing and Updating Baselines

What happens when a visual test fails A failing visual test produces expected, actual and diff images. The author decides whether the change was intended. If not, they fix the CSS and push again. If it was, they update the baselines in the pinned container and commit them, and reviewers approve the new images in the pull request diff. A diff is a question: was this intended? test fails expected · actual · diff intended change? no: fix the CSS push; test re-runs on the new preview yes: update baselines in the pinned container; reviewers approve images either way, the pull request ends with a passing check and a reviewed visual state
Baseline updates are code changes and get the same review as the CSS that caused them.

A failing visual test is a question, not a verdict: did you mean to change this? If the change is unintended, fix the CSS. If it is intended — a redesigned header, a new component — update the baselines in the same pull request:

docker run --rm -v "$PWD":/work -w /work mcr.microsoft.com/playwright:v1.47.0-jammy \
  npx playwright test tests/visual.spec.ts --update-snapshots

Running the update in the same container as CI prevents font-rendering differences from creeping into the baselines. Reviewers then see the new baseline images in the pull request diff, which GitHub renders side by side, and approve the visual change explicitly.

Some teams prefer not to store baselines in Git at all, because design-heavy repositories accumulate many megabytes of images over time. The alternative is to screenshot the main branch preview (or production) in the same job and compare the pull request against those images on the fly. It avoids baseline maintenance and never goes stale, at the cost of twice the screenshots per run and a dependency on main being healthy. Hosted services such as Percy, Chromatic and Argos follow this model and add a review interface on top; they are worth considering once the number of screenshots grows beyond what a pull request diff shows comfortably.

Measured Impact

A documentation team added 21 screenshots across seven templates and three viewports to their pull request pipeline. The job ran in about 90 seconds after the preview was ready. In the first quarter it caught 11 unintended layout changes before merge, including a table overflow on mobile introduced by a Tailwind upgrade and a missing dark-mode background on code blocks. False positives fell below 1% after masking two "last updated" dates and a GitHub star counter. Reader-reported layout bugs dropped from about four a month to one a quarter.

Pitfalls & Rollback

  • Baselines from a different environment. Screenshots from a Mac never match Linux CI; always update in the container.
  • Too many pages. Hundreds of screenshots make runs slow and reviews tedious; cover templates.
  • Low thresholds on huge pages. A tiny ratio on a very long page can fail on one-pixel anti-aliasing; tune per test.
  • Protected previews. Without the bypass token every screenshot is the login page, which all match each other perfectly.
  • Rollback: make the job non-required, or remove it; nothing else depends on it.

Conclusion

Visual regression tests catch the bugs that static sites are most prone to — layout, spacing, overflow, theme — and preview deployments give them a real environment to run against. Screenshot one or two pages per template at three widths with Playwright, run in a pinned container, disable animations, wait for fonts and images, mask dynamic content, and review diffs in the pull request. Updated baselines then become an explicit, reviewable record of every intended design change.

FAQ

Which pages should a visual regression suite cover on a static site?

One or two representative URLs per template - home, section index, article, a long article with code and tables, search and 404 - at two or three viewport widths. Covering every page adds cost and noise without catching more layout bugs, because pages sharing a template break together.

Why do visual tests produce false positives?

Differences in font rendering between machines, animations, lazy-loaded images, dates, randomised content and third-party embeds all change pixels without a real regression. Run tests in a fixed container, disable animations, wait for fonts and images, and mask dynamic regions.

Should visual tests run against the preview URL or a local build?

Against the preview when you want to test exactly what will ship, including headers, redirects and CDN behaviour. A local build served in CI is faster and avoids authentication, and is a good choice when previews are slow or protected.

Where do the baseline screenshots come from?

From main. Either commit baselines to the repository and update them in the pull request that intentionally changes the design, or generate them on each run from the production site or the main branch preview and compare the pull request against that.