Accessibility Testing with axe and pa11y in CI

Static site generators make accessible HTML easy and do not guarantee it. A theme update lowers the contrast of link text, a new component ships an icon button without a label, a Markdown change skips from h2 to h4, a locale template loses its lang attribute. Each of these is caught instantly by an automated checker and each is invisible to a sighted developer reviewing a preview. Automated tools cannot certify a site as accessible, but they are very good at keeping it from getting worse.

This guide adds two complementary tools to CI: pa11y-ci for fast scans of many URLs from the sitemap, and axe-core through Playwright for pages that need interaction or a theme switch. It samples pages by template, tests both light and dark themes, fails pull requests on new violations only, and tracks the rest as a baseline. It is part of Testing Static Sites in CI. The numbers come from a 2,300-page Hugo documentation site and a 410-page Astro site.

Prerequisites

• A preview deploy per pull request, or a local static server over the build directory (npx serve dist).
• Node 20+, pa11y-ci, @playwright/test and @axe-core/playwright.
• A list of templates and one or two representative URLs for each.

Step 1: Choose What to Scan

Accessibility problems on static sites come from templates and components far more than from individual pages. Scanning every page repeats the same template findings thousands of times and takes long enough that people stop waiting for it. Instead, build a sample:

• two representative pages per template (home, section index, article, API reference, search, 404);
• every page changed in the pull request, found with git diff --name-only mapped to URLs;
• a handful of known-complex pages: the longest table, the page with the most code blocks, the page with embedded video.

On the documentation site this was 28 template pages plus, on average, 6 changed pages per pull request.

// tests/a11y/urls.mjs
import { execSync } from 'node:child_process';
const templates = ['/', '/guide/', '/guide/install/', '/api/', '/api/client/connect/', '/search/', '/404.html' /* … */];
const changed = execSync('git diff --name-only origin/main...HEAD -- content/')
  .toString().trim().split('\n').filter(Boolean)
  .map((f) => '/' + f.replace(/^content\//, '').replace(/(_index|index)?\.md$/, ''));
export const urls = [...new Set([...templates, ...changed])];

Step 2: pa11y-ci for Breadth

pa11y-ci takes a list of URLs and runs each through a headless browser with the chosen runners. Using axe as the runner keeps results consistent with the Playwright tests:

{
  "defaults": {
    "runners": ["axe"],
    "standard": "WCAG2AA",
    "timeout": 30000,
    "chromeLaunchConfig": { "args": ["--no-sandbox"] },
    "hideElements": "iframe[src*='youtube'], .third-party-widget"
  },
  "urls": []
}

node -e "import('./tests/a11y/urls.mjs').then(m => console.log(m.urls.map(u => process.env.BASE_URL + u).join('\n')))" > urls.txt
npx pa11y-ci --config .pa11yci.json $(cat urls.txt) --json > pa11y.json

hideElements excludes third-party embeds you cannot fix, so their violations do not drown your own. Track those separately with the vendor.

Step 3: axe Through Playwright for Interaction and Themes

Some violations only appear after interaction — an open mobile menu, an expanded disclosure, a search dialog with results — and contrast issues can exist in one theme only. Playwright can put the page in each state and run axe there:

// tests/a11y/axe.spec.ts
import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';
import { urls } from './urls.mjs';

for (const theme of ['light', 'dark'] as const) {
  for (const url of urls) {
    test(`${theme} ${url}`, async ({ page }) => {
      await page.emulateMedia({ colorScheme: theme });
      await page.goto(url);
      const results = await new AxeBuilder({ page })
        .withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa', 'wcag22aa'])
        .exclude('.third-party-widget')
        .analyze();
      expect(newViolations(results.violations, url, theme)).toEqual([]);
    });
  }
}

test('mobile menu open', async ({ page }) => {
  await page.setViewportSize({ width: 390, height: 844 });
  await page.goto('/guide/install/');
  await page.getByRole('button', { name: 'Menu' }).click();
  const results = await new AxeBuilder({ page }).include('#site-nav').analyze();
  expect(results.violations).toEqual([]);
});

On the Astro site, 61% of contrast violations found in the first year existed only in the dark theme — mostly muted text colours that met 4.5:1 on white and failed on the dark background.

Step 4: Fail on New Violations Only

Turning on a strict accessibility check against an existing site usually produces hundreds of failures, and the check gets disabled the same afternoon. A baseline avoids that: record every current violation as a stable fingerprint (rule ID, URL template, target selector), commit the file, and fail only on fingerprints not in it.

import baseline from './baseline.json' assert { type: 'json' };
const known = new Set(baseline.map((v) => `${v.rule}|${v.template}|${v.target}|${v.theme}`));
export function newViolations(violations, url, theme) {
  return violations.flatMap((v) => v.nodes.map((n) => ({ rule: v.id, template: templateOf(url), target: n.target.join(' '), theme })))
    .filter((v) => !known.has(`${v.rule}|${v.template}|${v.target}|${v.theme}`));
}

Fixing a baseline entry is then a normal pull request that removes it. A weekly job posts the baseline size to the team channel; on the documentation site it went from 212 entries to 4 in five months, with the last four waiting on a third-party search widget.

Step 5: Report Clearly

Failures should say what to fix, where, and how. axe results include a helpUrl per rule and the failing element's selector and HTML snippet; print those in the test failure and upload the full JSON as an artifact. A summary table in $GITHUB_STEP_SUMMARY — rule, count, example page — is what reviewers actually read.

What Automated Tests Cannot Tell You

Every green run still needs periodic human testing. Automated checks cannot judge whether alt text is meaningful, whether the focus order follows the visual order, whether a custom widget is usable with a screen reader, or whether content makes sense when zoomed to 400%. Both sites scheduled a quarterly manual review — keyboard-only navigation through each template, a VoiceOver and NVDA pass on the most-used flows, zoom and reflow checks — and each review found two or three issues no rule covers, most often in interactive components. Record those findings as Playwright tests where possible (focus lands on the dialog heading, Escape returns focus to the trigger), so the next regression is caught automatically.

Contrast checks also depend on fonts being loaded: a fallback font renders thinner and at different sizes, which can change axe's contrast verdict for large-text thresholds. Waiting for document.fonts.ready before running axe removed a class of results that differed between runs.

Measured Impact

Pitfalls & Rollback

• Scanning before hydration or fonts. Contrast on fallback fonts and pre-hydration states can differ; wait for networkidle or a ready marker.
• Testing only light mode. Most dark-mode contrast failures go unseen.
• Excluding too much. Exclusions for third-party widgets should be narrow selectors, not whole regions.
• Treating a pass as compliance. Automated rules cover a minority of WCAG criteria.
• Rollback: the checks are CI jobs; making them non-required or reverting the workflow change removes the gate without touching the site.

Conclusion

Automated accessibility testing is a ratchet, not a certificate. Scan a sample by template plus changed pages, in both themes, with pa11y-ci for breadth and axe through Playwright for interactive states; fail on new violations and burn down a baseline. On these sites that took sampled violations from 212 to 4 and all but stopped new regressions from reaching readers.

FAQ

How much of WCAG can automated tools check?

Automated rules reliably catch a subset, commonly estimated at 30 to 50 percent of issues, including contrast, missing alternative text, form labels, heading order, the lang attribute, duplicate IDs and invalid ARIA. Keyboard flow, focus order that makes sense, meaningful alternative text and reading order still need manual review.

Should I use axe-core or pa11y?

pa11y-ci is the quickest way to scan a list of URLs from a sitemap and can run axe as its engine. axe-core through Playwright is better when pages need interaction first, such as opening a menu or switching to dark mode, and integrates with the rest of an end-to-end suite. Many teams use both.

How do I introduce accessibility tests on a site with many existing violations?

Record the current violations as a baseline, fail the build only on new violations, and burn the baseline down over time. Failing on every existing issue from day one usually leads to the check being disabled.

Why test dark mode separately?

Colour contrast is computed from the actual rendered colours, and a dark theme uses different ones. On the sites measured here, most contrast regressions existed in only one theme.

Related

• Parent: Testing Static Sites in CI — the full test suite.
• End-to-End Tests for Static Sites with Playwright — the same Playwright setup for flows.
• Validating HTML Output in CI — markup errors that also harm accessibility.
• Evaluating SSG Accessibility Defaults — starting from an accessible theme.
• Visual Regression Testing on Preview Deploys — catching visual changes axe does not.