Static Site Generators in Production

Logging 404s at the Edge

Every static site has a list of URLs that readers try and fail to reach: pages moved during a restructure without a redirect, links in old blog posts on other sites, bookmarks from a previous generator's URL scheme, typos in a popular forum answer. Those 404s are invisible to the build, rarely reported by readers, and — because they come from outside — impossible to find by crawling your own site. The one place they are all recorded is the CDN's request log.

This guide collects 404s from the edge on Cloudflare and CloudFront, filters out bot noise, ranks paths by reader impact and turns the top of the list into redirects on a weekly cadence. It is part of Monitoring Static Sites in Production.

Prerequisites

Step 1: Capture 404s With Their Context

A 404 count per path is not enough to act on; you need to know where readers came from. Record path, referrer, user agent class and country for every 404. On Cloudflare, a small Worker in front of static assets can write that to Workers Analytics Engine without any log pipeline:

// worker.js — wraps static assets, records 404s
export default {
  async fetch(request, env, ctx) {
    const res = await env.ASSETS.fetch(request);
    if (res.status === 404) {
      const url = new URL(request.url);
      const ua = request.headers.get('user-agent') ?? '';
      ctx.waitUntil(Promise.resolve(env.NOTFOUND.writeDataPoint({
        blobs: [url.pathname, request.headers.get('referer') ?? '', uaClass(ua), request.cf?.country ?? ''],
        indexes: [url.pathname.slice(0, 96)],
      })));
    }
    return res;
  },
};
const uaClass = (ua) => /bot|crawl|spider|scan|curl|python|go-http/i.test(ua) ? 'bot'
  : /Mozilla\/5\.0/.test(ua) ? 'browser' : 'other';

On CloudFront, standard logs already contain status, URI, referrer and user agent; point an Athena table at the log bucket and query it directly.

Step 2: Filter Out the Noise

Most raw 404s on the public internet are not readers. Vulnerability scanners probe /wp-login.php, /.env, /.git/config and thousands of similar paths on every site. On the documentation site measured here, 91% of raw 404 requests were bots. Filter before ranking:

-- Athena over CloudFront logs: last 7 days of reader-facing 404s
SELECT uri, count(*) AS hits,
       count(DISTINCT referrer) AS referrers,
       approx_percentile(length(referrer), 0.5) > 1 AS usually_referred
FROM cloudfront_logs
WHERE status = 404 AND date >= current_date - interval '7' day
  AND user_agent LIKE 'Mozilla/5.0%'
  AND NOT regexp_like(user_agent, '(?i)bot|crawl|spider|scan')
  AND NOT regexp_like(uri, '(?i)\.(php|asp|aspx|env|git|bak|sql|zip)$|wp-|/cgi-bin/|/\.')
GROUP BY uri
ORDER BY hits DESC
LIMIT 50;
Raw 404s versus reader 404s A stacked bar of 404 requests in one week. Of 184,000 raw 404s, 167,000 were bots and scanners, 4,500 were browsers requesting probe-like paths, and 12,500 were readers following real links, which is the part worth acting on. One week of 404s, before and after filtering bots and scanners · 167,000 (91%) readers 12,500 probe-like paths from browsers · 4,500 Ranking unfiltered 404s puts /wp-login.php at the top of a site that has never run WordPress
The actionable part of the 404 log is under a tenth of it; filtering is what makes the report readable.

Step 3: Rank by Reader Impact and Decide

For each path in the top fifty, decide one of four outcomes:

SituationActionExample
Content moved; clear equivalent exists301 redirect/docs/install → /guides/install/
Old URL pattern from a previous generatorpattern redirect/blog/2021/03/slug.html → /blog/slug/
Content deliberately removed410 Gone, or 404 with a helpful pageretired product docs
Typo in an external link, heavy trafficredirect, and ask the linking site to fix it/guides/intsall/

Referrers tell you which: a 404 whose referrer is your own site is an internal broken link to fix at the source; one whose referrer is a search engine means an indexed URL that needs a redirect; one whose referrer is a single external page means one link somewhere is wrong.

The referrer decides the fix Three referrer types lead to three fixes. A referrer on your own site means an internal broken link: fix the link in the source. A search engine referrer means an indexed old URL: add a redirect. A single external page as referrer means one wrong link elsewhere: add a redirect if traffic is significant and contact the linking site. Where the reader came from tells you what to fix your own site search engine one external page fix the link in your source; no redirect needed add a 301 to the equivalent page redirect if busy; ask the linking site to update
Same-site referrers are the most valuable find: they are bugs in your own content that every reader of that page can hit.

On the restructured docs site, 23 of the top fifty 404 paths had same-site referrers — links inside the docs that the restructure had not updated, mostly in pages written in raw HTML where the link checker's Markdown parser had not looked. Fixing those at the source removed the 404 and the need for a redirect at the same time. Another 19 came from search engines and got redirects. The remaining eight each came from one external page, usually a forum answer or a partner's documentation; for the five with meaningful traffic, a redirect plus a short note to the linking site's maintainer fixed it permanently.

Step 4: Make It a Weekly Loop

Run the query weekly, post the top twenty to the team channel with a suggested action for each, and add redirects in a pull request the same week. Redirect targets are verified by the build, as described in Keeping Redirects Working After an SSG Migration. Add an alert for spikes: a path that goes from zero to more than 100 reader 404s in an hour almost always means a deploy removed or renamed something popular.

Reader 404s per day over eight weeks of weekly fixes A falling line of reader 404s per day. It starts at about 1,800 per day in week one after a site restructure, drops to about 700 after the first batch of 15 redirects, to about 300 after week three, and levels off around 180 per day from week five onward. Reader 404s per day, weekly redirect batches 0 1,000 2,000 ~1,800 / day 15 redirects +22 ~180 / day wk 1 wk 8
Two weekly batches of redirects removed most of the reader-facing 404s; the remainder is a long tail of one-off typos.

Other Status Codes Worth Watching

The same log pipeline answers adjacent questions cheaply. Count 410 and 404 responses on asset paths separately from pages: a spike in missing .js or .css files usually follows a deploy that removed old hashed assets while cached HTML still referenced them, which is covered in Atomic Deploys vs Incremental Uploads. Count 301 responses by path: a redirect that serves thousands of hits a week points at an internal link or sitemap entry that should be updated to the final URL, saving every reader a round trip. And watch 5xx responses, which on a static host should be essentially zero; any at all usually mean an edge function throwing or an origin misconfiguration on self-hosted setups.

Make the 404 Page Useful

Some 404s will always remain, so the page readers land on matters. A static 404 page can still help: include a search box, links to the main sections, and — because it is generated at build time — a short list of pages whose paths share words with common mistyped URLs. A few lines of client-side script can suggest the closest existing path by comparing the requested URL with the sitemap. On this site, adding a search box and "did you mean" suggestion to the 404 page meant 41% of readers who hit a 404 clicked through to a real page, against 12% before. Make sure the page returns a real 404 status — hosts serve custom 404 pages correctly by default, but single-page-app fallback settings can turn them into soft 200s that search engines index.

Measured Impact

Eight weeks after a documentation restructure moved 340 pages:

MeasureWeek 1Week 8
Reader 404s per day~1,800~180
Redirects added from the log061
Internal broken links found via same-site referrers230
Search Console "Not found (404)" URLs29034
404-page click-through to a real page12%41%

Pitfalls & Rollback

  • Ranking unfiltered logs. Bot probes dominate raw 404s. Filter by user agent and path pattern before looking.
  • Redirecting everything to the homepage. Search engines treat that as a soft 404, and readers lose context. Redirect to an equivalent page or return 404.
  • Keeping raw IPs indefinitely. They are personal data in many jurisdictions; truncate or drop them.
  • A 404 page that returns 200. Check the status explicitly; SPA fallbacks often break it.
  • Rollback: the logging Worker or Athena table is independent of the site. Removing it stops collection without affecting pages.

Conclusion

Edge logs are the only complete record of the URLs readers fail to reach. Capturing each 404 with its referrer and user agent, filtering the 91% that are bots, and acting weekly on the top of the ranked list cut reader-facing 404s on a restructured docs site by 90% in eight weeks, with 61 redirects. A 404 page with search and suggestions turned many of the rest into successful visits.

FAQ

Why log 404s at the edge instead of in analytics?

Analytics scripts run in the browser and often do not load on error pages, and many readers block them. The CDN sees every request, including those from links in other sites, feed readers and bots, so its logs are the complete record.

How do I tell bot 404s from reader 404s?

Look at the referrer and user agent. Reader 404s usually have a referrer from a search engine, another site or your own pages, and a browser user agent. Bot probes for paths like /wp-admin or /.env have no referrer and scanner user agents. Filter them out before ranking.

Should every 404 get a redirect?

No. Redirect paths that readers actually reach and that have a clear equivalent page. Leave probes, typos with no pattern and paths for content that was deliberately removed as 404 or 410.

How long should 404 logs be kept?

Thirty to ninety days is enough to see patterns and seasonal traffic. Aggregate counts per path can be kept longer; drop raw IP addresses or truncate them to respect privacy.