Crawling for Broken Links on a Schedule
Links rot. A crawl of a four-year-old documentation site found that 11% of its external links returned errors, soft-404s or redirects to unrelated pages; a three-year-old engineering blog was at 17%. None of that breakage happened in a commit, so no pull-request link check could have caught it. Readers find it one click at a time, and each broken link quietly tells them the page is out of date.
A scheduled crawl of the live site catches rot as it happens and turns it into a short queue of fixes. This guide sets up a weekly crawl with lychee in GitHub Actions, separates internal from external breakage, files issues rather than posting a wall of red, and keeps false positives low enough that people act on the results. It is part of Monitoring Static Sites in Production and complements the pull-request checks in Checking Links in Pull Requests.
Prerequisites
- A live site with a
sitemap.xmllisting every page. - GitHub Actions (or any scheduler) and permission to create issues.
- An idea of which external domains the site links to most — the first crawl will tell you.
Step 1: Crawl From the Sitemap
Crawling from the sitemap guarantees every published page is checked, including pages not linked from navigation. lychee is a fast link checker that accepts a list of URLs to fetch and extracts and checks every link on each:
curl -s https://docs.example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' > pages.txt
lychee --no-progress --format json --output report.json \
--max-concurrency 8 --max-retries 3 --retry-wait-time 5 \
--timeout 20 --accept '200..=299,429' \
--user-agent 'Mozilla/5.0 (compatible; docs-linkcheck/1.0; +https://docs.example.com/about/)' \
--exclude-path 'pagefind' --exclude '^https://(www\.)?linkedin\.com' \
--cache --max-cache-age 3d \
$(cat pages.txt)
--cache keeps results for three days, so a re-run after fixing one link does not re-check thousands of healthy ones. Accepting 429 (rate limited) as non-broken avoids reporting sites that merely asked the crawler to slow down; they are re-checked next week.
Step 2: Split Internal From External
Internal broken links on a live site mean something went wrong in routing or deployment — a redirect missing after a page moved, a section excluded from the build. They should fail the job and alert the team the same day. External broken links are expected rot, handled at a steady pace.
// scripts/triage-links.mjs
const r = JSON.parse(readFileSync('report.json', 'utf8'));
const failures = Object.entries(r.fail_map ?? {}).flatMap(([page, links]) =>
links.map((l) => ({ page, url: l.url, status: l.status?.code ?? l.status?.text })));
const internal = failures.filter((f) => new URL(f.url).host === 'docs.example.com');
const external = failures.filter((f) => new URL(f.url).host !== 'docs.example.com');
writeFileSync('external.json', JSON.stringify(external));
if (internal.length) { console.error(internal); process.exit(1); }
Step 3: File Issues, Grouped and Deduplicated
Posting 200 broken links into a chat channel produces no fixes. Group external failures by domain, check them against open issues with a broken-link label, and open one issue per domain listing every affected page:
- name: File issues for new external breakage
if: always()
uses: actions/github-script@v7
with:
script: |
const ext = require('./external.json');
const byHost = {};
for (const f of ext) (byHost[new URL(f.url).host] ??= []).push(f);
const open = await github.paginate(github.rest.issues.listForRepo,
{ owner: context.repo.owner, repo: context.repo.repo, labels: 'broken-link', state: 'open' });
for (const [host, items] of Object.entries(byHost)) {
if (open.some((i) => i.title.includes(host))) continue;
await github.rest.issues.create({ owner: context.repo.owner, repo: context.repo.repo,
title: `Broken links to ${host} (${items.length})`, labels: ['broken-link'],
body: items.map((i) => `- [ ] ${i.url} (${i.status}) on ${i.page}`).join('\n') });
}
The checklist format lets whoever picks up the issue tick off each link as it is fixed.
Anchors and Redirect Chains
Two quieter kinds of breakage are worth including once the basic crawl is stable. Fragment links — /guides/config/#cache-dir — keep returning 200 when the heading is renamed, so a status-code check never notices that the reader lands at the top of a long page instead of the section they wanted. lychee's --include-fragments option checks that the fragment exists as an id on the target page; on this site it found 41 stale anchors, almost all internal and caused by heading edits. Redirect chains to external sites are not broken yet but often will be: a link that now goes through two redirects to reach its destination usually means the target site has reorganised, and updating the link to its final URL removes latency for readers and a future failure point. Report chains of two or more hops as low-priority items in the same weekly issue rather than as failures.
Step 4: Keep False Positives Low
Link checkers produce false positives from sites that block bots, return 403 to anything without a browser fingerprint, rate-limit, or serve soft-404s with a 200 status. On the first run against the docs site, 38% of reported external failures were false. Four measures brought that down to 4%:
| Measure | False positives removed |
|---|---|
| Realistic user agent with a contact URL | 31% of the false set |
| Accept 429, retry with backoff | 22% |
| Exclude domains that always block checkers (checked manually quarterly) | 35% |
| Treat 403 from known-good domains as a warning, not a failure | 12% |
Soft-404s — pages that return 200 with "page not found" content — are the one category lychee cannot detect by status. A small post-check fetches a sample of 200-status external URLs and flags bodies under 2 KB or containing "not found" in the title, which found 17 more dead links on the first run.
Measured Impact
Six months of weekly crawls on the 2,000-page docs site:
| Measure | Start | After 6 months |
|---|---|---|
| Broken external links | 212 | 14 (open issues) |
| Broken internal links on live site | 23 (from a restructure) | 0 |
| Median age of a broken link when fixed | unknown (years) | 9 days |
| False positives per crawl | 130 | 5–10 |
| Crawl duration | 22 min | 14 min (with cache) |
| Engineer time per week | — | ~40 min working through issues |
About a third of fixes replaced the link with the page's new location, a third linked an archived copy (the Internet Archive had most), and a third removed the link and adjusted the sentence.
Fixing Rot Well
How a broken link is fixed matters as much as finding it. Each option has a different effect on the reader and on future maintenance.
Two habits reduce future rot. When writing, prefer links to stable, versioned documentation over blog posts and marketing pages, which move most often; the crawl data showed vendor blog links broke at four times the rate of versioned docs links. And for citations that must survive, submit the page to an archive service at the time of writing, so an archived copy exists from the day the link is added rather than whatever snapshot happened to be taken later. A small pre-commit hook can do the submission automatically for new external links in changed Markdown files.
Pitfalls & Rollback
- Crawling as a bot that looks like an attack. High concurrency without a user agent gets you blocked. Crawl politely and identify yourself.
- Alerting on external breakage. It is gradual and not urgent; file issues instead.
- One issue per link. Two hundred issues is noise. Group by domain.
- Checking only the build. Production has redirects, headers and CDN rules the build does not; crawl the live site.
- Rollback: the crawl is a scheduled workflow. Disable the schedule to stop it; issues already filed remain as a record.
Conclusion
A weekly crawl from the sitemap, split into urgent internal failures and gradual external rot, filed as grouped and deduplicated issues, and tuned until false positives are rare, turns link rot from an invisible decay into a forty-minute weekly chore. On a 2,000-page docs site, 212 broken external links fell to 14 open issues within six months, and new breakage was fixed in a median of nine days rather than whenever a reader happened to complain.
FAQ
If links are checked in pull requests, why crawl production too?
Pull-request checks catch links you break. A scheduled crawl catches links that break on their own, mostly external pages that move or disappear, plus production-only problems such as redirects and CDN rules that differ from the build.
How often should the crawl run?
Weekly for external links, which rot slowly, and daily for internal links on sites that deploy often. A weekly full crawl of a 2,000-page site with rate limiting took about 14 minutes.
How do I handle sites that block link checkers?
Some sites return 403 or 429 to automated clients. Send a realistic user agent, respect rate limits, retry with backoff, and keep an allow-list of domains known to block checkers so they are verified manually instead of reported as broken.
What should happen to a broken external link?
Replace it with the moved page if it exists, link an archived copy if the content matters, or remove the link and adjust the text. Track each as an issue so the fix is owned and visible.
Related
- Parent: Monitoring Static Sites in Production — the other monitoring signals.
- Checking Links in Pull Requests — catching the links you break.
- Logging 404s at the Edge — broken links pointing into your site.
- Keeping Redirects Working After an SSG Migration — preventing internal breakage in the first place.
- Scheduling Content Publication with Cron-Triggered Builds — other scheduled jobs in the same workflow style.