Search Index Size Budgets for Large Docs

Search indexes grow silently. Nobody decides to ship a four-megabyte index; it happens one generated API page, one retained version and one changelog entry at a time, until the search box on a phone takes three seconds to show anything and readers conclude the docs have no search at all. Performance teams budget JavaScript bundles for exactly this reason. Search deserves the same treatment, with one twist: the number to budget is not the index size on disk but the bytes a reader must download before their first result.

This guide defines that budget, measures it on three real documentation sites, shows the five changes that brought one 6,000-page site back under it, and wires the check into CI. It builds on the tool comparison in Search for Static Sites.

Prerequisites

• A documentation site with client-side search (Pagefind, Lunr, MiniSearch or a framework's built-in local search).
• Chrome DevTools, or a Lighthouse CI setup that can run a scripted search — see Setting Up Lighthouse CI for a Static Site.
• A list of your ten most common queries, to measure realistic fragment downloads.

Define the Budget in Reader Terms

The budget exists to protect one experience: a reader opens search, types, and sees results before they give up. Usability sessions and field data on documentation sites suggest the tolerable wait from first keystroke to first result is about one second, and that anything under half a second feels instant.

Working backwards on a Moto G Power-class phone over fast 4G (roughly 1.6 Mbps effective with 150 ms latency), the time to first result breaks into connection overhead, download, and parse-and-initialise:

That gives two lines: 150 KB as the target, where search feels immediate, and 300 KB as the hard limit, beyond which a noticeable share of readers give up before results appear.

Measure the First-Query Payload

For a monolithic index (Lunr, MiniSearch, VitePress local search) the first-query payload is the whole index plus the search library. For Pagefind it is the entry script, the WebAssembly engine, the metadata file and the fragments touched by the query. Measure it with a scripted search in a headless browser rather than guessing:

// scripts/search-payload.mjs — Playwright
import { chromium } from 'playwright';
const browser = await chromium.launch();
const page = await browser.newPage();
let bytes = 0;
page.on('response', async (r) => {
  if (/\/(pagefind|search)\//.test(r.url())) {
    bytes += Number(r.headers()['content-length'] ?? (await r.body()).length);
  }
});
await page.goto(process.argv[2]);
await page.click('#search-open');
await page.keyboard.type('deploy', { delay: 80 });
await page.waitForSelector('.pagefind-ui__result');
console.log(`first-query payload: ${(bytes / 1024).toFixed(0)} KB`);
await browser.close();

Run it against a preview deploy so compression and HTTP caching behave as in production, and repeat for your ten most common queries — fragment-based indexes download different amounts per query.

Three documentation sites measured this way:

Site C, the largest, had the smallest payload. Site B, a third of its size, was over the hard limit. Total size predicted nothing; what was indexed decided everything.

Find What Is Bloating the Index

Site B's payload came from five sources, identified by rebuilding the index with each suspect excluded and re-running the measurement:

Apply the Fixes

The five fixes above, in order of payload saved:

Index one version. Old versions stay browsable but are not indexed by default. A small version-specific index can be loaded when a reader switches to an old version, filtered by data-pagefind-filter="version:...", as outlined in Versioned Documentation with Docusaurus.

Draw the content boundary. Navigation text repeated on 6,000 pages is noise with a large footprint. Mark only the article body as indexable.

Index API pages selectively. A reference page for a type with 400 properties produces hundreds of unique identifiers, each a separate index term. Index the page title, summary and headings, and add data-pagefind-ignore to property tables. Readers searching for a specific identifier are better served by the API page's own filter box.

Split the changelog. Release notes mention every feature and match everything. Give them their own index loaded from the changelog page, so they never compete with guides in the main search.

Skip code blocks. Shell output and configuration keys inflate the vocabulary. Keep them out unless your readers genuinely search for code strings, in which case index only inline code in prose.

Enforce It in CI

Budgets that are not enforced become aspirations. Run the payload script after every preview deploy and fail the check when the median of the top ten queries exceeds the budget:

- name: Search payload budget
  run: |
    node scripts/search-payload.mjs "${{ steps.preview.outputs.url }}" > payload.txt
    KB=$(grep -oP '\d+(?= KB)' payload.txt)
    echo "Search first-query payload: ${KB} KB (budget 150, limit 300)"
    [ "$KB" -le 300 ] || { echo "::error::Search payload over 300 KB"; exit 1; }
    [ "$KB" -le 150 ] || echo "::warning::Search payload over 150 KB target"

Record the value per build in the same place as your other performance metrics, as described in Writing a Performance Budget That Fails Builds. A slow upward drift is a signal to look at what new content is being indexed before it crosses the line.

Measured Impact

Site B after the five fixes, measured with the same script and device profile:

Relevance improved as a side effect: removing navigation text and changelog noise let the pages that genuinely matched rise to the top.

Pitfalls & Rollback

• Budgeting total index size. For fragmented indexes it is the wrong number. Budget the first-query payload.
• Measuring one query. Fragment downloads vary by query. Use the median of your ten most common.
• Testing locally without compression. A dev server may not gzip or brotli JSON. Measure a real preview deploy.
• Excluding pages rather than regions. Dropping API pages from search entirely hides content readers need; exclude the heavy tables, not the pages.
• Rollback: every fix is a markup attribute or an index configuration change. Reverting one restores that content to the index on the next build.

Conclusion

A search index is a performance budget line like any bundle. Budget the bytes before the first result — 150 KB as the target, 300 KB as the limit — measure them with a scripted search on every preview deploy, and when the number climbs, look at what is being indexed rather than at the tool. On a 6,000-page docs site, indexing one version, drawing a content boundary, trimming API tables and splitting the changelog took the first-query payload from 342 KB to 118 KB and the wait from 1.5 seconds to half a second.

FAQ

What is a reasonable size budget for a search index?

Budget the bytes a reader downloads before the first result, not the total index size. Around 150 KB gzipped keeps the first result under half a second on a mid-range phone over 4G; 300 KB is the outer limit before the search box feels unresponsive.

Why does total index size not matter for Pagefind?

Because Pagefind splits its index into fragments and loads only those a query needs. A 40 MB index on disk can still serve a first query with under 200 KB of downloads. What matters is the size of the entry files and the fragments a typical query touches.

What usually makes a search index too big?

Indexing things readers never search for. Navigation repeated on every page, generated API pages with thousands of identifiers, changelogs, code blocks and every old version of the docs are the usual culprits.

How do I enforce the budget?

Measure it in CI after every build. Compute the gzipped size of the files a first query downloads and fail the build when it passes the budget, the same way you would fail a build for an oversized JavaScript bundle.

Related

• Parent: Search for Static Sites — choosing the search tool in the first place.
• Building a Lunr Index at Build Time in Eleventy — a monolithic index kept under budget.
• Adding Pagefind to an Astro Site — content boundaries and weights in practice.
• Tracking Bundle Size per Pull Request — the same budgeting discipline for JavaScript.
• Choosing an SSG for API Reference Documentation — where large generated reference sections come from.