Incremental Builds in Astro with the Content Layer

Astro's content collections used to re-read and re-render every Markdown file on every build. For a 300-page blog that is fine. For a 10,000-page documentation site or a site pulling thousands of entries from a headless CMS, it made builds long and CI expensive. Astro 5's content layer changed that. Collections are now populated by loaders, entries are stored with a digest in a persistent data store, and unchanged entries are reused on the next build instead of being parsed and rendered again.

This guide explains what the content layer caches and what it does not, shows how to write a custom loader that takes advantage of digests, and sets up CI so the cache survives between runs. It is part of Incremental Builds and Build Caching for SSGs.

Prerequisites

Astro 5 or later with collections defined in src/content.config.ts.
A CI system that can persist a directory between runs.
For custom loaders: familiarity with the collection schema API.

What the Content Layer Caches

Each collection is defined with a loader. The built-in glob() loader reads files from disk; the file() loader reads a single JSON or YAML file; custom loaders fetch from anywhere. On each build, the loader decides what to add, update or remove in the collection's store. Astro saves the store to node_modules/.astro/data-store.json, including each entry's data, its digest and, for Markdown, its rendered HTML.

The digest check is what turns a full content reload into an incremental one.

What is incremental:

Loading. Unchanged files or API items are not re-parsed or re-validated.
Markdown and MDX rendering. Rendered HTML for unchanged entries is reused, which is where most of the saving comes from on content-heavy sites.
Image processing. Optimised images live in the same cache directory and are reused if unchanged.

What is not:

Page generation. Astro still runs every page's component tree and writes every HTML file. For sites with thousands of pages this is now usually the largest part of the build.
Client bundling. Vite rebuilds client JavaScript and CSS each time, although it is fast for most content sites.

The glob Loader

For files on disk, the built-in glob() loader handles digests for you:

// src/content.config.ts
import { defineCollection, z } from 'astro:content';
import { glob } from 'astro/loaders';

const docs = defineCollection({
  loader: glob({ pattern: '**/*.{md,mdx}', base: './src/content/docs' }),
  schema: z.object({
    title: z.string(),
    description: z.string(),
    updated: z.coerce.date().optional(),
  }),
});

export const collections = { docs };

It hashes each file's contents, compares with the store and only processes changed files. Deleted files are removed from the store. In development, it also watches the folder and updates single entries in place.

Writing a Loader With Digests

Custom loaders get the same benefit if they pass a digest. Here is a loader for a headless CMS that fetches all entries, but only re-processes those whose content changed:

import type { Loader } from 'astro/loaders';

export function cmsLoader({ endpoint, token }): Loader {
  return {
    name: 'cms-loader',
    async load({ store, parseData, generateDigest, meta, logger }) {
      const since = meta.get('lastSync');
      const url = new URL(endpoint);
      if (since) url.searchParams.set('updated_since', since);
      const res = await fetch(url, { headers: { Authorization: `Bearer ${token}` } });
      const { items, deleted } = await res.json();

      for (const item of items) {
        const digest = generateDigest(item);
        const data = await parseData({ id: item.slug, data: item });
        store.set({ id: item.slug, data, digest, rendered: { html: item.bodyHtml } });
      }
      for (const id of deleted ?? []) store.delete(id);

      meta.set('lastSync', new Date().toISOString());
      logger.info(`synced ${items.length} changed, ${deleted?.length ?? 0} deleted`);
    },
  };
}

Two techniques combine here. The meta store persists a lastSync timestamp, so the loader asks the CMS only for entries changed since the last build — a much smaller response. And generateDigest plus store.set with a digest means that even if the CMS returns an unchanged entry, Astro skips reprocessing it. If the CMS cannot filter by date, the digest alone still saves the parsing and rendering work.

Digests make processing incremental; a sync timestamp makes the fetch incremental too.

Run a full sync occasionally anyway — for example on a nightly schedule with the cache key bumped — so any entries missed by the timestamp filter, such as items restored from the CMS trash, are picked up. Content modelling for CMS-backed sites is covered in Wiring a Headless CMS to a Static Build.

Keeping the Cache in CI

The store lives in node_modules/.astro, which CI runners discard. Persist it explicitly:

- uses: actions/cache@v4
  with:
    path: node_modules/.astro
    key: astro-${{ hashFiles('package-lock.json', 'astro.config.*', 'src/content.config.*') }}-${{ github.sha }}
    restore-keys: |
      astro-${{ hashFiles('package-lock.json', 'astro.config.*', 'src/content.config.*') }}-

Order matters: npm ci deletes the whole node_modules folder, including anything restored into it. Restore the cache after npm ci and before astro build. Including the content config in the key means a schema change starts from an empty store, which Astro would do anyway, and avoids restoring a large store that will be discarded. The full workflow is in Building Astro Sites with GitHub Actions.

With a warm store, page generation — not content processing — dominates the build.

When the Cache Is Cleared

Astro discards the store and reloads everything when the Astro version changes, when content.config.ts or collection schemas change, or when the store file is missing or corrupt. Integrations that modify content, such as remark and rehype plugins, are part of the config, so adding a plugin also clears it — which is correct, because rendered HTML would otherwise be stale. If output ever looks stale after an unusual change, delete node_modules/.astro locally or bump the CI cache key.

Reducing Page Generation Time

Once content processing is incremental, page generation is the remaining cost. Three common improvements:

Avoid per-page collection queries. Calling getCollection('docs') inside every page to build a sidebar re-sorts thousands of entries thousands of times. Compute navigation once in a module and import it.
Keep components cheap. Syntax highlighting inside components at render time repeats work the Markdown pipeline already did.
Split very large sites. Separate versioned docs into their own builds, or build only the latest version on each commit, as described in Matrix Builds for Multi-Site Monorepos.

Measured Impact

A documentation site with 6,000 Markdown entries moved from Astro 4's legacy collections to the content layer with the glob() loader and a persisted CI cache. Cold builds were unchanged at about four minutes. Typical pull request builds, which change a handful of files, fell from 4 minutes 15 seconds to 51 seconds. A second site loading 3,500 entries from a headless CMS with a lastSync loader cut its build's API time from 70 seconds to under 3 and stopped hitting the CMS's rate limit during busy editing days.

Pitfalls & Rollback

Restoring the cache before npm ci. It is deleted with node_modules; restore after install.
Loaders without digests. Every entry is re-processed each build; pass digest to store.set.
Relying on lastSync without handling deletions. Deleted CMS entries stay in the store unless the loader removes them.
Stale rendered HTML after plugin changes. Changing plugins through a separate file the config imports may not clear the store; bump the cache key when you do.
Rollback: delete the cache step; builds become full builds again, with identical output.

Conclusion

Astro's content layer makes content processing incremental: loaders store each entry with a digest, unchanged entries reuse their data and rendered HTML, and the store persists in node_modules/.astro. Use the glob() loader for files, pass digests from custom loaders and sync only changed items where the source allows it, and cache the store in CI after installing dependencies. Content-heavy builds then spend most of their time generating pages, which is the next thing to optimise.

FAQ

Does Astro support incremental builds?

Partly. Astro 5's content layer caches loaded and rendered collection entries between builds in a data store, so unchanged entries are not re-parsed or re-fetched. Astro still writes every page on each build, so page rendering time remains, but content loading and Markdown processing become incremental.

Parent: Incremental Builds and Build Caching for SSGs — caching across generators.
Enabling Incremental Builds in Eleventy — the Eleventy equivalent.
Caching Hugo Builds in GitHub Actions — the Hugo equivalent.
Building Astro Sites with GitHub Actions — the full workflow.
Measuring Build-Time Regressions in CI — noticing when the cache stops working.