Fixing Pages Crawled but Not Indexed on Static Sites

"Crawled – currently not indexed" is the most frustrating line in Search Console's page indexing report, because it is not an error. The search engine fetched the page, read it, and decided not to keep it. There is no broken tag to fix. On static sites, the status tends to grow in a particular way: a generator that produces pages from templates — tags, categories, authors, dates, pagination, near-duplicate variants — produces thousands of pages with little of their own to say, and the search engine declines to index them. Sometimes that is correct. Sometimes the same signal starts excluding pages that matter.

This guide works through 4,300 crawled-but-not-indexed URLs on an 8,000-page developer resource site built with Hugo, sorts them by template, and fixes the ones that should be indexed while deliberately removing the ones that should not exist. It belongs to Technical SEO for Static Sites.

Step 1: Export and Group by Template

Export the URLs from Search Console's page indexing report (Pages → Crawled – currently not indexed → Export). Group them by URL pattern, which on a static site maps directly to the template that produced them:

import pandas as pd, re
df = pd.read_csv('crawled-not-indexed.csv')
def template(u):
    for name, pat in [('tag', r'/tags/'), ('page', r'/page/\d+/'), ('author', r'/authors/'),
                      ('snippet', r'/snippets/'), ('guide', r'/guides/'), ('changelog', r'/changelog/')]:
        if re.search(pat, u): return name
    return 'other'
print(df['URL'].map(template).value_counts())

The grouping changed the question. 3,110 of the URLs were tag, pagination and author pages, mostly thin, which the site did not need indexed. The real problems were 290 guides and 760 snippet pages — content the site wanted found.

Step 2: Decide What Should Exist

For each template, decide: improve it, consolidate it, or stop producing it.

Step 3: Find Why Good Guides Were Excluded

The 290 excluded guides were full articles, so thinness was not the explanation. Three patterns appeared when comparing them with indexed guides from the same section:

Near-duplicates. 96 were versions of the same guide for different frameworks — "Deploy to X with Hugo", "Deploy to X with Jekyll" — generated from one template with a few words swapped. The search engine indexed one or two per group and excluded the rest. These were merged into one guide per task with framework-specific sections, and the old URLs redirected to the matching section anchor.

Weak internal links. 148 were reachable only through paginated lists and tag pages — no guide, section page or navigation linked to them. Their crawl dates in Search Console were months old. Adding them to their section overview pages and to "related" blocks on similar guides gave each at least three contextual links.

Content hidden by the template. 46 used a tabbed layout that rendered only the first tab's content in HTML; the other tabs were populated by a script from a JSON file. The rendered page the search engine evaluated was short. Rendering all tabs in HTML, hidden with CSS until selected, put the full content in the page.

Step 4: Improve the Snippet Pages

Snippet pages were 5–20 lines of code with a title and nothing else. The site's own analytics showed readers valued them — they had high click-through from internal search — but a page of code alone looks thin. Each snippet got two short prose sections, written by the team over three weeks: when to use it, and what to watch out for. 210 near-identical snippets (the same function in slightly different forms) were merged into 70 pages with variants. The snippet template also gained a "used in" list linking to guides that include the snippet, which doubled as internal links in both directions.

Step 5: Stop Generating Pages Nobody Needs

For the tag pages with fewer than five items, Hugo stopped rendering them:

[[cascade]]
  [cascade.build]
    render = "never"
  [cascade._target]
    kind = "term"
    path = "/tags/{a-tag,another-tag}"   # generated list of thin tags

A build script computes the list of thin tags and writes it into the cascade target; the remaining 180 tag pages got a short hand-written intro from a data file. Removed tag URLs return 410, which tells search engines the removal is deliberate. Author pages redirected to a single contributors page. The sitemap dropped every page that should not be indexed, so it lists only pages the site stands behind — see Generating Sitemaps for Large Static Sites.

Step 6: Verify on a Sample, Then Wait

After deploying, indexing was requested for 20 representative URLs from each fixed group, and their status checked with URL Inspection a week later. 17 of 20 guides and 14 of 20 snippets were indexed within ten days, which confirmed the fixes before waiting on the rest. The full report updated over about six weeks.

Keeping New Thin Pages From Appearing

A generator will keep producing pages from templates long after a cleanup, so the build now reports the word count of each page's main content by template and flags any template whose median falls below 150 words.

Measured Impact

Pitfalls & Rollback

• Treating the whole report as a problem. Group by template; many excluded pages should be excluded.
• Requesting indexing without changing anything. The decision repeats.
• Template-generated near-duplicates. Merge them; one strong page beats five similar ones.
• Content loaded by script. Render it in HTML.
• Rollback: each template decision is a config or template change; removed tag pages can be restored by removing them from the cascade list, though their redirects or 410s should be reverted at the same time.

Conclusion

"Crawled – currently not indexed" on a static site is mostly a template question. Group excluded URLs by template, decide which templates should produce pages at all, and then fix the pages that matter: merge near-duplicates, link to orphans, render hidden content, and add substance to thin pages. On this site that cut the excluded count from 4,300 to 1,180 — nearly all intended — and raised organic clicks to guides and snippets by 31%.

FAQ

What does crawled currently not indexed mean?

Google fetched the page but decided not to add it to the index, at least for now. It is not an error and there is no single fix. It usually means the page looked low-value, too similar to another page, or unimportant relative to the rest of the site. Improving or consolidating the page, and linking to it from relevant pages, are the levers that change the decision.

Is it a problem if some pages are crawled but not indexed?

Not always. Tag pages, paginated archives and near-empty utility pages are often excluded, and that can be the right outcome. It is a problem when the excluded pages are ones you want people to find, such as guides, product pages or documentation. Check which templates the excluded URLs belong to before acting.

Will requesting indexing in Search Console fix it?

Rarely on its own. Requesting indexing asks for a recrawl, but if the page is still judged low-value it stays excluded. Change the page or its context first, then request indexing for a few representative URLs to confirm the change worked before applying it to the rest.

Do static sites have more of these problems than dynamic ones?

They can, because generators make it easy to produce many pages from templates, such as one per tag, per author, per date or per paginated list, each with little unique content. A static site that generates thousands of such pages often sees them excluded, and sometimes drags down how often its important pages are crawled.

Related

• Parent: Technical SEO for Static Sites — the technical SEO checklist for generators.
• SEO-Safe Pagination and Tag Pages — designing list pages that help.
• Canonical URLs and Duplicate Content on Static Sites — handling near-duplicates.
• Fixing Soft 404s on Static Hosts — the neighbouring indexing status.
• Fixing Hugo Builds That Slow Down After Adding Taxonomies — the build-time side of too many tag pages.