Reducing Lighthouse Score Variance in CI

Run Lighthouse ten times on the same page and you get ten different answers. On a laptop the spread is annoying; in CI, where a number above a threshold fails someone's pull request, it is the difference between a check people trust and one they mute. The fix is not a single setting but a handful of changes that each remove one source of noise, until the spread across unchanged builds is small enough to put a threshold above it.

This guide measures the spread on a real setup, removes the noise one source at a time, and reports what each change was worth. It supports the timing-budget approach in Performance Budgets and Lighthouse CI.

Prerequisites

• Lighthouse CI running in your pipeline — see Setting Up Lighthouse CI for a Static Site.
• The ability to trigger the workflow repeatedly on an unchanged commit (workflow_dispatch).
• Somewhere to collect results — the uploaded .lighthouseci/ JSON is enough.

Measure the Spread First

Before changing anything, quantify the problem. Trigger the Lighthouse job twenty times on the same commit against the same preview URL, and compute the spread (max minus min, and the standard deviation) for each metric.

for i in $(seq 1 20); do gh workflow run lighthouse.yml -f ref=$SHA; sleep 5; done
# after they finish:
gh run list -w lighthouse.yml -L 20 --json databaseId -q '.[].databaseId' \
  | xargs -I{} gh run download {} -n lighthouse -D runs/{}
node scripts/spread.mjs runs/   # prints min, median, max, stdev per metric

The starting point for a blog post template, single run per job, GitHub Actions ubuntu-latest, Lighthouse 12 mobile:

With an 800 ms spread, any LCP threshold low enough to catch a real regression also fails healthy builds.

The Six Sources of Noise

Fix 1: Median of Several Runs

The cheapest and largest improvement. Set numberOfRuns: 5 and assert with aggregationMethod: 'median'. Lighthouse CI also offers median-run, which picks the run with the median performance score and uses all its metrics together; plain median takes the median of each metric independently, which is what you want for per-metric budgets.

Fix 2: Warm the CDN Before Measuring

A fresh preview deploy has a cold edge cache. The first Lighthouse run fetches every asset from origin; later runs hit the edge. Discard that effect by requesting each URL and its main assets once before collecting:

for u in $(node -e "require('./lighthouse/urls')(process.env.PREVIEW_URL).forEach(x=>console.log(x))"); do
  curl -s -o /dev/null "$u"
done

Fix 3: Block Nondeterministic Third Parties

Analytics, chat widgets, consent managers and A/B testing scripts behave differently on each load. In CI, measure your own code and block them with blockedUrlPatterns, then measure third-party cost separately and deliberately, as described in Auditing Third-Party Scripts with Lighthouse.

collect: {
  settings: {
    blockedUrlPatterns: ['*googletagmanager.com*', '*intercom*', '*hotjar*', '*optimizely*'],
  },
},

Fix 4: Pin Chrome and Lighthouse

A Chrome or Lighthouse update can shift metrics by several percent. Pin @lhci/cli to a minor version and install a specific Chrome build (browser-actions/setup-chrome with a version input). Upgrade both deliberately, re-measure the spread, and re-baseline budgets in the same pull request.

Fix 5: Detect or Avoid Slow Runners

Lighthouse records a benchmarkIndex for the machine it ran on. On shared runners it varied from about 1,100 to 1,900 across the twenty jobs, and TBT correlated with it strongly. Two options: discard runs whose benchmark index is below a floor and re-run, or move the job to a dedicated runner. A self-hosted runner, or a larger GitHub-hosted runner, with no other jobs scheduled, removed most of the TBT spread.

// scripts/check-benchmark.mjs — fail fast on an unusually slow runner
import { readdirSync, readFileSync } from 'node:fs';
const idx = readdirSync('.lighthouseci').filter((f) => f.startsWith('lhr-'))
  .map((f) => JSON.parse(readFileSync(`.lighthouseci/${f}`)).environment.benchmarkIndex);
const min = Math.min(...idx);
console.log(`benchmarkIndex min ${min}`);
if (min < 1300) { console.log('Slow runner — re-run the job'); process.exit(78); }

Fix 6: Budget Bytes Alongside Timings

Byte sizes do not vary at all between runs of the same build. For every timing budget, add the byte budget that usually drives it — script bytes for TBT, above-the-fold image bytes for LCP on image-led templates. Then the timing budget can stay a warning with a generous threshold while the byte budget is a hard error, and most real regressions are still caught deterministically. The pattern is laid out in Writing a Performance Budget That Fails Builds.

Measured Impact

The same twenty-job experiment after each cumulative change:

With a 90 ms LCP spread around a 1.9 s median, an error threshold at 2.2 s sits more than three spreads above the median. That is safe to enforce, and it would still catch the 300 ms regression a mis-sized hero image typically causes.

Keeping Variance Low Over Time

Variance creeps back. A new third-party script is added and nobody updates the block list; the runner image is upgraded and the benchmark floor no longer matches; a template gains an animation that makes LCP timing depend on when the animation frame lands. Re-run the twenty-job spread measurement once a quarter and after any Lighthouse, Chrome or runner change, and keep the result next to the budgets in the repository so the next person can see what "normal" looks like.

It also helps to separate the performance job from everything else. Running Lighthouse in the same job as the build, or in parallel with a heavy test suite on the same self-hosted machine, reintroduces CPU contention you removed by choosing a dedicated runner. A job that does nothing but warm the CDN, run Lighthouse and upload results is easier to reason about and measurably steadier.

Finally, make noise visible in reports. Lighthouse CI's output includes every run's values; surfacing the min–max range in the pull request comment next to the median lets reviewers see at a glance whether a result is clear-cut or borderline, and they stop treating a single number as more precise than it is.

Pitfalls & Rollback

• Blocking third parties and forgetting them. Blocked scripts still affect readers. Measure them in a separate scheduled job so their cost is visible.
• Averaging instead of taking the median. One outlier run drags a mean; the median ignores it.
• Re-running until green. Automatic retries on failure hide real regressions. Retry only on a detected slow runner, not on a failed assertion.
• Comparing across Lighthouse versions. Re-baseline after every upgrade instead of assuming continuity.
• Rollback: each fix is a config line or a workflow step. Revert any one of them and re-measure the spread to see what it was contributing.

Conclusion

Lighthouse variance in CI is not mysterious; it comes from a few identifiable sources, and each has a direct fix. Median of five runs, a CDN warm-up, blocked third parties, pinned versions, a quiet runner and byte budgets alongside timing budgets took LCP spread from 800 ms to 90 ms and TBT spread from 210 ms to 35 ms on the same unchanged page. At that level, timing budgets can fail builds without failing healthy ones.

FAQ

How many Lighthouse runs per URL are enough?

Three is the practical minimum and five is better for timing metrics. In our measurements, moving from one run to the median of three cut the LCP spread by about two thirds; going to five cut it by a further quarter.

Does simulated or applied throttling vary less?

Simulated throttling, the default, usually varies less because it models the network from an unthrottled trace. Applied throttling with DevTools is more realistic but more sensitive to runner load.

Why is Total Blocking Time so noisy?

TBT depends on how long main-thread tasks take, which depends directly on CPU speed. Shared CI runners vary in CPU availability from job to job, so TBT moves more than network-bound metrics. Lighthouse's benchmark index can be used to detect slow runners.

Should I use a dedicated runner?

If timing budgets must block merges, yes. A self-hosted or larger dedicated runner with nothing else scheduled on it cut TBT spread from 170 ms to 35 ms in our tests. Byte budgets do not need it because sizes do not vary.

Related

• Parent: Performance Budgets and Lighthouse CI — why timing budgets need low variance.
• Writing a Performance Budget That Fails Builds — when to promote a warning to an error.
• Running WebPageTest Scripts Against Preview Deploys — real devices when lab emulation is not enough.
• Measuring Build Time Regressions in CI — the same noise problem for build durations.
• Auditing Third-Party Scripts with Lighthouse — measuring what CI blocks.