Static Site Generators in Production

Measuring Build-Time Regressions in CI

Build times do not degrade in one step; they degrade in a hundred small ones. A partial that scans every page, a plugin added for one feature, an image pipeline that stopped hitting its cache. Each change adds a second, nobody notices, and eighteen months later the build takes four minutes and someone proposes a migration to fix it.

The cure is measurement with a threshold attached: time the build step specifically, compare against a rolling baseline of comparable runs, and fail a pull request that makes it dramatically worse. This guide sets that up. It sits under Incremental Builds and Build Caching for SSGs.

Prerequisites

  • A CI pipeline where the build is a distinct step you can time.
  • Somewhere to keep a history: a cache entry, a repository file, or a small metrics store.
  • A cache-restore step whose hit or miss status is available to the workflow.

Time the Build, Not the Job

Job duration is the wrong number: it includes checkout, dependency installation, tests and deploy, and it moves with runner contention. Measure the build step alone and record the context that explains it:

#!/usr/bin/env bash
# scripts/timed-build.sh — emits build-metrics.json
set -euo pipefail
START=$(date +%s%3N)
npm run build
END=$(date +%s%3N)

PAGES=$(find dist -name '*.html' | wc -l)
DURATION=$(( END - START ))

cat > build-metrics.json <<JSON
{
  "duration_ms": ${DURATION},
  "pages": ${PAGES},
  "ms_per_page": $(( DURATION / (PAGES > 0 ? PAGES : 1) )),
  "cache_hit": "${CACHE_HIT:-unknown}",
  "sha": "${GITHUB_SHA:-local}",
  "branch": "${GITHUB_REF_NAME:-local}"
}
JSON
cat build-metrics.json

ms_per_page is the field that survives a growing site. A corpus that doubles legitimately doubles total build time; if milliseconds per page also doubles, something in the templates got slower — the quadratic-scan pattern described in Profiling Hugo Templates With Template Metrics.

Total build time versus milliseconds per page Two series over twelve months. Total build time rises steadily from 18 to 52 seconds, which looks alarming. Milliseconds per page stays flat at about 12 until month eight, then rises to 19, isolating the month where a template genuinely got slower rather than the site merely growing. Growth or regression? Only one series tells you total build · 18 s → 52 s ms/page · 12 → 19 at month 8 month 1 month 6 month 12 Solid: total seconds — rises with every new page, so it always looks like a regression Dashed: per page — flat while the site grows, and steps exactly where a template changed
Normalising by page count is what turns "the build keeps getting slower" into "the build got slower in month eight, and here is the commit range".

Compare Against a Rolling Baseline

Comparing against the previous run guarantees false alarms, because runner performance varies by 10-20% between jobs. Compare against a median of recent runs on the default branch:

// scripts/check-build-budget.mjs — compares against a rolling median
import { readFile, writeFile } from 'node:fs/promises';

const current = JSON.parse(await readFile('build-metrics.json', 'utf8'));
let history = [];
try { history = JSON.parse(await readFile('.build-history.json', 'utf8')); } catch { /* first run */ }

// Compare warm builds against warm builds only
const comparable = history.filter((h) => h.cache_hit === current.cache_hit).slice(-20);

if (comparable.length >= 5) {
  const values = comparable.map((h) => h.ms_per_page).sort((a, b) => a - b);
  const median = values[Math.floor(values.length / 2)];
  const delta = (current.ms_per_page - median) / median;
  const pct = (delta * 100).toFixed(1);

  console.log(`ms/page: ${current.ms_per_page} vs median ${median} (${pct > 0 ? '+' : ''}${pct}%)`);

  if (delta > 0.25) {
    console.error(`build regression: ${pct}% slower per page than the 20-run median`);
    process.exit(1);
  }
  if (delta > 0.10) console.warn(`::warning::build ${pct}% slower per page than median`);
}

if (current.branch === 'main') {
  history.push(current);
  await writeFile('.build-history.json', JSON.stringify(history.slice(-50), null, 2));
}

Three design choices make this usable rather than noisy. Only comparable runs enter the baseline, so a cold build after a dependency change is never measured against warm ones. The threshold is a percentage, because absolute seconds mean different things at different site sizes. And only default-branch runs update the history, so a slow experimental branch cannot poison the baseline.

Wire It In

# .github/workflows/build.yml (excerpt)
- name: Restore build cache
  id: cache
  uses: actions/cache@v4
  with:
    path: |
      node_modules/.astro
      .cache
    key: build-${{ hashFiles('package-lock.json') }}-${{ hashFiles('content/**') }}
    restore-keys: build-${{ hashFiles('package-lock.json') }}-

- name: Build (timed)
  env:
    CACHE_HIT: ${{ steps.cache.outputs.cache-hit == 'true' && 'warm' || 'cold' }}
  run: bash scripts/timed-build.sh

- name: Build budget
  run: node scripts/check-build-budget.mjs

- name: Persist history
  if: github.ref == 'refs/heads/main'
  uses: actions/cache/save@v4
  with:
    path: .build-history.json
    key: build-history-${{ github.run_id }}

Storing the history in a cache entry rather than in the repository keeps the default branch free of commits that exist only to record a number. If you prefer the history to be reviewable, commit it — but do it from a scheduled job rather than on every merge, or the noise is worse than the problem.

How a build-time gate classifies a run A decision flow. If the cache state differs from the baseline, the run is not comparable and the check is skipped. Otherwise the run is compared with the median: under ten percent passes silently, ten to twenty-five percent emits a warning, and above twenty-five percent fails the pull request. Three outcomes, one of which is silence same cache state? warm vs warm only no → skip not comparable < 10% slower pass, silently 10–25% slower warning annotation > 25% slower fail the pull request Thresholds are per page against a 20-run median on the default branch, not against the previous run A gate that fires on runner noise gets disabled within a fortnight, which is worse than having none
The silent case is the important one. A check that comments on every pull request trains people to ignore it long before it has anything useful to say.

Read the Regression, Then Fix It

When the gate fires, the number tells you there is a problem and not what it is. Three questions resolve nearly every case.

Did the page count change? If pages grew and milliseconds per page held, there is no regression — the site is bigger. Update nothing.

Did dependencies change? A new plugin or a major version bump is the most common single cause, and it usually shows as a step change rather than a drift. Compare the lockfile diff against the run where the number moved.

Did a template change? This is the expensive case and the one worth profiling. Generator-specific profilers point straight at it: --templateMetrics in Hugo, the build analyser in Astro, DEBUG=Eleventy:Benchmark* in Eleventy.

DEBUG=Eleventy:Benchmark* npx @11ty/eleventy 2>&1 | grep -E 'Benchmark.*[0-9]{4,}ms' | head -10

Make the Number Visible Without Nagging

A gate that only speaks when something is wrong is easy to forget; one that comments on every pull request gets muted. The middle path is a job summary — visible to anyone who looks at the run, invisible to everyone else.

{
  echo "### Build"
  echo ""
  echo "| metric | value |"
  echo "|---|---|"
  echo "| duration | $(jq -r '.duration_ms/1000 | floor' build-metrics.json) s |"
  echo "| pages | $(jq -r .pages build-metrics.json) |"
  echo "| ms/page | $(jq -r .ms_per_page build-metrics.json) |"
  echo "| cache | $(jq -r .cache_hit build-metrics.json) |"
} >> "$GITHUB_STEP_SUMMARY"
Where build metrics should appear Three channels ranked. A job summary is always available and never intrusive. A warning annotation appears only on a moderate regression. A failed check appears only on a large regression. A pull request comment on every run is marked as the option that gets muted. Visible when wanted, loud only when needed Job summary — every run duration, pages, ms/page, cache state · read by anyone who opens the run Warning annotation — 10 to 25% regression appears in the diff view, does not block the merge Failed check — above 25% blocks the merge, names the median it was compared against A comment on every pull request is the fourth option, and the one that gets muted within a month
The summary is what makes the gate trustworthy: when it does fail, the number it failed on has been visible on every run leading up to it.

Measured Impact

An Eleventy documentation site tracked for four quarters after adding the gate:

QuarterPagesBuild timems/pageRegressions caught
Q1 (baseline)62018.4 s29.7—
Q281023.9 s29.52 (both reverted in the PR)
Q31,04031.2 s30.01 (plugin swapped)
Q41,31039.1 s29.83 (2 reverted, 1 accepted)

Total build time rose 112% across the year, and milliseconds per page moved by less than 2%. That is the outcome the gate exists to produce: the build got slower only because the site got bigger, and each of the six genuine regressions was caught in the pull request that introduced it rather than a year later.

Pitfalls & Rollback

  • Timing the whole job. Install, tests and deploy swamp the signal and vary with runner contention.
  • Comparing cold against warm. A cache miss is not a regression; record cache state and compare like with like.
  • Absolute thresholds. Five seconds means something different on a 20-second build than on a four-minute one.
  • Comparing against the previous run. Runner variance guarantees false positives; use a rolling median.
  • Failing on small regressions. A gate that fires often gets disabled. Warn below 25%, fail above it.
  • Rollback: the check is one script and one workflow step. Removing the step restores the previous pipeline, and the history file is inert data that can simply be deleted.

Conclusion

Build times decay by accumulation, so the only defence is a number that someone sees on every pull request. Time the build step alone, normalise by page count, compare against a rolling median of comparable runs, and reserve a hard failure for regressions large enough to be real. When it fires, check page count, then dependencies, then templates — in that order, because that is the order of how often each is the cause. The caching and incrementality that keep the number low in the first place are in Incremental Builds and Build Caching for SSGs.

FAQ

Why not just watch the CI job duration?

Because job duration mixes checkout, dependency install, build, tests and deploy, and it varies with runner contention. Timing the build step alone, and recording it as a number rather than reading it from a log, is what makes a trend visible.

How do I stop runner variance from causing false alarms?

Compare against a rolling median of recent runs on the default branch rather than against the previous run, and set the threshold as a percentage rather than a fixed number of seconds. A 25 percent regression against a 20-run median is a real signal; a 3-second jump against one previous run is noise.

Should a build-time regression fail the pull request?

Fail it for a large regression and warn for a small one. A hard failure at 25 percent catches an accidental quadratic template; a warning at 10 percent keeps the number visible without blocking work that legitimately adds pages.

How do I separate a cache miss from a real slowdown?

Record whether the cache was restored as part of the measurement, and compare warm builds against warm builds. A cold run after a dependency change is expected to be slow and should never be compared against a warm baseline.

What should I do when the site legitimately gets bigger?

Normalise by page count. Tracking milliseconds per page instead of total seconds means adding 200 pages does not look like a regression, while a template that got slower still does.