Lighthouse CI vs WebPageTest for Regression Gates

A performance budget is only as good as the check that enforces it. The two tools most static sites consider for that check are Lighthouse CI, which runs Lighthouse inside the CI job against a preview build, and WebPageTest, which loads the preview on real browsers and devices from test locations and reports back through its API. Both can fail a pull request. They differ in how noisy they are, what they catch, how long they take and what they cost — and a gate that is too noisy gets ignored, while one that is too slow gets skipped.

This comparison ran both as gates on every pull request of a 1,900-page Astro documentation and marketing site for eight weeks, alongside a set of deliberately introduced regressions, and recorded variance, catches, false alarms, time and cost. It belongs to Performance Budgets and Lighthouse CI; the Lighthouse CI setup is in Setting Up Lighthouse CI for a Static Site.

Method

• Pages: six URLs covering every template — home, docs article, docs index, blog post, pricing, changelog.
• Lighthouse CI 0.14 on the GitHub Actions standard runner, against the static build served locally, mobile preset, 5 runs per URL, median assertions.
• WebPageTest (hosted, API) against the pull request's preview deploy, Moto G Power on 4G from one location, 3 runs per URL, first view, median.
• Assertions, both tools: LCP, Total Blocking Time (TBT), CLS, total JavaScript bytes and total transfer bytes per template, with tolerances set from two weeks of baseline variance.
• Regressions: 12 deliberate changes introduced in throwaway pull requests over the eight weeks, each with a known effect, plus whatever real regressions occurred.

Variance

Two weeks of runs on unchanged main gave each tool's noise:

TBT under Lighthouse CI was the noisiest metric: shared runners vary in CPU speed, and simulated throttling multiplies that variation. Byte counts were perfectly stable in both, which is why they make the best hard gates.

Regressions Caught

Lighthouse CI caught 9 of 12; WebPageTest caught 11. The two Lighthouse misses are structural: a 250 ms LCP change was inside its noise band, and a caching problem cannot show when Lighthouse CI loads a locally served build. Neither caught the interaction regression — a page-load test does not interact; that needs an interaction test such as the one in Measuring Web Vitals in Playwright Tests.

False Alarms

Over eight weeks of real pull requests (214 PRs):

Nine false alarms in 214 PRs was tolerable but noticeable; after the team dropped the TBT assertion's tolerance from 20% to 35% and kept TBT as a warning rather than an error, Lighthouse CI false alarms fell to 2 in the following month without missing any of the deliberate regressions that had triggered on bytes.

Time and Cost

WebPageTest's time depended on queueing at the test location, which made it the slowest check on most pull requests — long enough that developers merged before it finished on several occasions.

Making Results Readable for Developers

A gate is only useful if developers can act on a failure without a performance specialist. Both tools produce rich reports, but the default output differed in how quickly someone could see why. Lighthouse CI's temporary public storage gives a link to the full report per URL, and its assertion output names the metric and the threshold; the team added a short PR comment that lists only failing assertions, with a link to the report and a diff of resource sizes against main generated from the Lighthouse JSON. WebPageTest's strength is the filmstrip and waterfall comparison between two tests, which made the font-preload and caching regressions obvious at a glance — a developer looking at the comparison could see the font request start half a second later, or the HTML coming from the origin instead of the edge.

The combination the team settled on posts one comment per pull request: Lighthouse CI's failing assertions and size diff immediately, and, for the nightly WebPageTest run on main, a message in the team channel only when a metric moved beyond its tolerance, with the comparison link. In the month after that change, the median time from a failing gate to a fix pushed was 40 minutes, down from most of a day when failures required someone to dig through raw reports.

Verdict

• Gate pull requests with Lighthouse CI, asserting byte budgets as errors and LCP, TBT and CLS as warnings with tolerances from your own variance. It is fast, free beyond CI minutes, and caught every byte-driven regression.
• Run WebPageTest nightly and before releases against production or the release candidate, on a real device and through the CDN. It caught the two regressions Lighthouse CI structurally could not.
• Add an interaction test for INP; neither page-load tool covers it.

Pitfalls

• Asserting on the Lighthouse score. Assert on bytes and individual metrics.
• Setting tolerances from guesswork. Measure two weeks of variance first.
• Testing a local build only. CDN and header problems need a test through the real host.
• Gates slower than code review. Developers merge around them.

Conclusion

As pull request gates on the same site, Lighthouse CI was quicker, cheaper and noisier, catching 9 of 12 deliberate regressions; WebPageTest was slower and steadier, catching 11, including a CDN caching problem that a local build cannot show. The arrangement that worked was both: Lighthouse CI on every pull request with byte budgets as hard failures, and WebPageTest on a schedule through the real CDN.

FAQ

Which is better as a pull request gate, Lighthouse CI or WebPageTest?

Lighthouse CI for most static sites. It runs inside the CI job, takes a couple of minutes, and with several runs per URL and assertions on stable metrics it caught most real regressions in this comparison. WebPageTest measured more realistically and caught two regressions Lighthouse missed, but it is slower and costs more, so it fits better as a nightly or pre-release check.

Why do Lighthouse scores vary so much between CI runs?

Lighthouse simulates a slow device and network from a trace recorded on whatever machine runs it. Shared CI runners vary in CPU speed and load, so the simulated timings vary too. Running three to five times per URL and taking the median, asserting on byte budgets that do not vary, and using a dedicated runner reduce the noise.

What does WebPageTest measure that Lighthouse CI does not?

WebPageTest loads pages on real devices or real browsers with traffic-shaped network connections in chosen locations, records a filmstrip and full waterfall, and can test repeat views, scripted multi-step journeys and different connection profiles. Lighthouse CI simulates throttling on the CI machine with a single first-view load.

Should a performance gate fail on the Lighthouse score?

Better not. The score is a weighted blend that moves with small changes in any metric and has the highest run-to-run variance. Assert on resource sizes and counts, which are deterministic, and on individual metrics such as LCP and Total Blocking Time with tolerances measured from your own variance.

Related

• Parent: Performance Budgets and Lighthouse CI — budgets and how to enforce them.
• Reducing Lighthouse Score Variance in CI — lowering the noise measured here.
• Running WebPageTest Scripts Against Preview Deploys — the WebPageTest setup.
• Per-Template Performance Budgets — the budgets both tools asserted.
• Measuring Web Vitals in Playwright Tests — covering interactions.