Static Site Generators in Production

Multilingual Search on Static Sites

Search on a single-language site is a solved problem. Add a second language and three things break at once: English stemming mangles German, stop-word lists remove the wrong words, and readers of the French docs get English results they did not ask for. Add Japanese and a fourth thing breaks — there are no spaces between words, so a naive indexer treats whole sentences as single tokens and finds nothing.

This guide builds per-language search for a documentation site in six languages (English, German, French, Spanish, Japanese and Simplified Chinese), compares Pagefind's automatic handling with a hand-built Lunr setup, and measures relevance per language. It extends Search for Static Sites and pairs with the site-structure decisions in Picking an SSG for a Multi-Language Documentation Site.

Prerequisites

  • A static site with one URL prefix per locale (/de/, /ja/), each page carrying a correct lang attribute on <html>.
  • A way to know which pages are translations and which are fallbacks.
  • Twenty or more real queries per language with the expected page, from native speakers or support logs, to measure relevance.

Why One Index Fails

A shared index built with English processing does four kinds of damage:

Wrong stemming. The English Porter stemmer reduces "deployment" and "deploying" to "deploy". Applied to German it turns "Bereitstellungen" into nonsense and fails to relate it to "Bereitstellung". Applied to French it strips endings that are not suffixes.

Wrong stop words. English removes "die" as a stop word. In German, "die" is the feminine article — also a stop word, luckily — but English also drops "is", "on" and "was", which are content words in other languages ("was" is German for "what").

Compound words. German glues nouns together: "Zwischenspeicherkonfiguration" (cache configuration) is one token. A query for "Zwischenspeicher" does not match it unless the indexer decomposes compounds or supports substring matching.

No word boundaries. Japanese "キャッシュの設定" (cache settings) has no spaces. Without segmentation, the whole phrase is one token, and a query for "キャッシュ" finds nothing.

Four ways a single English-processed index fails other languages Four panels. Stemming: German Bereitstellungen is stemmed incorrectly and does not match Bereitstellung. Stop words: German was, meaning what, is removed as an English stop word. Compounds: Zwischenspeicherkonfiguration does not match a query for Zwischenspeicher. Segmentation: the Japanese phrase for cache settings stays one token and a query for cache finds nothing. One index, English rules, four failure modes Stemming Bereitstellungen → "bereitstellungen" query "Bereitstellung" → no match English suffix rules do not apply to German Stop words "Was ist ein Build-Hook?" English list drops "was" (= what) content words removed, noise words kept Compounds Zwischenspeicherkonfiguration query "Zwischenspeicher" → no match one token, never decomposed Segmentation キャッシュの設定 → one token query キャッシュ → no match no spaces means no word boundaries
Each failure is invisible to an English-speaking team testing search in English, which is why multilingual search regressions often ship unnoticed.

Option 1: Pagefind's Per-Language Indexes

Pagefind handles this with no configuration beyond correct markup. During indexing it reads each page's lang attribute, groups pages by language, applies that language's stemmer (it bundles Snowball stemmers for about twenty-five languages) and segments CJK text. The output contains one index per language:

dist/pagefind/
├── pagefind-entry.json          # lists languages and their index hashes
├── index/                       # per-language index chunks
│   ├── en_7a1f...pf_index
│   ├── de_c912...pf_index
│   └── ja_04bd...pf_index
└── fragment/

At query time, pagefind.js reads document.documentElement.lang and loads only the matching language's index. A reader on /de/ never downloads English fragments. To search across languages deliberately, initialise with pagefind.options({ language: 'en' }), but that is rarely what readers want.

The things you still have to get right are all markup:

<html lang="de">                 <!-- not "de-DE" on some pages and "de" on others -->
<article data-pagefind-body>
  <p data-pagefind-ignore>Diese Seite ist noch nicht übersetzt.</p>

Inconsistent language codes split one language into two indexes; Pagefind treats de and de-DE as different languages unless normalised. Pick one form in your layout and assert it in a build check.

Option 2: Lunr With lunr-languages

For a site already on Lunr, lunr-languages adds stemmers and stop words per language. Build one index per locale in the build hook, as in Building a Lunr Index at Build Time in Eleventy:

import lunr from 'lunr';
import stemmerSupport from 'lunr-languages/lunr.stemmer.support.js';
import de from 'lunr-languages/lunr.de.js';
import ja from 'lunr-languages/lunr.ja.js';
import tinyseg from 'lunr-languages/tinyseg.js';
stemmerSupport(lunr); de(lunr); tinyseg(lunr); ja(lunr);

function buildIndex(locale, docs) {
  return lunr(function () {
    if (locale !== 'en') this.use(lunr[locale]);
    this.ref('id');
    this.field('title', { boost: 10 });
    this.field('body');
    docs.forEach((d) => this.add(d));
  });
}

The browser must load the same language plugin before lunr.Index.load(), or queries are processed with English rules against a German index and silently return worse results. That coupling is the main maintenance cost of this option, and the Japanese segmenter (TinySegmenter) adds about 25 KB gzipped to the Japanese search bundle.

Measured Relevance per Language

Each language had 25 queries from native-speaking support staff, scored on whether the expected page ranked first. The same 480 pages per language were indexed three ways.

LanguageSingle English-processed indexLunr + lunr-languagesPagefind per-language
English20 / 2520 / 2521 / 25
German9 / 2517 / 2519 / 25
French13 / 2518 / 2519 / 25
Spanish14 / 2519 / 2519 / 25
Japanese2 / 2515 / 2517 / 25
Chinese1 / 25not supported16 / 25
Correct first results per language, three indexing approaches Grouped bars per language. With a single English index, German scores 9, Japanese 2 and Chinese 1 out of 25. With Lunr and lunr-languages, German reaches 17 and Japanese 15, with Chinese unsupported. With Pagefind per-language indexes, German reaches 19, Japanese 17 and Chinese 16. English stays around 20 to 21 in all three. Right page first, out of 25 queries per language 0 12 25 EN DE FR ES JA n/a ZH single English index Lunr + lunr-languages Pagefind per-language
English barely moves; every other language roughly doubles, and the CJK languages go from unusable to workable.

The remaining German misses were compound words that neither tool decomposes. For the four worst offenders, adding the decomposed terms as hidden keywords on the target page (data-pagefind-meta or a hidden indexed span) recovered three. CJK misses were mostly technical loanwords written in katakana in queries and in English in the docs, or the reverse — a content consistency problem more than a search problem.

Index Size per Locale

Per-language indexes also keep downloads proportional to one language rather than all six. With Pagefind the first-query payload for a German reader was 94 KB; a shared all-language index built for comparison needed 310 KB for the same query, because fragments contained words from every language. Japanese and Chinese indexes were the largest per page (about 1.4× the English size) because segmentation produces many short tokens.

First-query payload for a German reader Two bars. A shared all-language index made a German reader download 310 kilobytes before the first result. Per-language indexes reduced that to 94 kilobytes because only German fragments are fetched. A German reader's first query, same content Shared index fragments carry all six languages · 310 KB Per-language 94 KB, German only Pagefind 1.3, 480 pages per language, query "Zwischenspeicher", gzip, median of 5
Per-language indexes are a relevance fix first, but they also cut the download by more than two thirds.

Localise the Search Interface Too

An index in the right language inside an English search box still feels broken. The placeholder text, the "no results" message, the result count announced to screen readers and the filter labels all need translating. Pagefind UI ships translations for its own strings and picks them from the page's lang; custom interfaces should read their strings from the same locale files as the rest of the site. Keyboard shortcuts deserve a check as well: / is awkward on several European keyboard layouts, so offer Ctrl+K alongside it. Finally, log queries per locale separately. Zero-result queries in Japanese or German are often the first sign that a translation is missing or uses different terminology from the product UI.

Pitfalls & Rollback

  • Inconsistent lang values. de and de-DE become separate indexes. Normalise in the layout and assert with a build check.
  • Missing lang on generated pages. Tag pages and API pages produced by plugins often inherit the default locale. Check them.
  • Indexing fallback pages in every locale. An untranslated English page served under /ja/ lands in the Japanese index with English text. Either mark it lang="en" inside the article or exclude it, and label fallback results.
  • Testing only in English. Relevance regressions in other languages are invisible to English testers. Keep per-language query sets and run them in CI.
  • Mixed-script content. Japanese pages full of English product names need both scripts to be searchable; test queries in each script, not only the native one.
  • Rollback: per-language indexing is driven by markup and configuration. Reverting the language attributes or the Lunr plugin setup restores the previous single index on the next build.

Conclusion

Multilingual search on a static site is mostly a markup problem: correct, consistent lang attributes, per-language indexes and a query that loads the reader's language only. Pagefind does the language processing automatically and handled all six languages in the test, including Chinese; Lunr with lunr-languages works for European languages and Japanese but needs its plugins kept in step between build and browser. Either way, per-language indexing roughly doubled correct first results outside English, and a per-language query set in CI is what keeps it that way.

FAQ

Should all languages share one search index?

No. Stemming and stop words are language-specific, and a shared index mixes results readers cannot use. Build one index per language and query the index matching the page the reader is on.

How does Pagefind know which language a page is in?

It reads the lang attribute on the html element. Pages with lang="de" go into the German index, pages with lang="ja" into the Japanese one, and the browser library loads the index matching the current page automatically.

Does static search work for Chinese and Japanese?

Yes, with limits. These languages do not separate words with spaces, so the tool needs a segmenter. Pagefind segments CJK text during indexing; recall is good for common terms but weaker than a hosted service with a tuned dictionary.

What should happen when a translation is missing?

Index untranslated fallback pages in the reader's language index only if the page itself is served under that locale. Otherwise readers see results they cannot read. Mark fallback pages clearly in the result snippet.