Static Site Generators in Production

Loading Remote Content With Astro Content Layer Loaders

Many static sites get some of their content from somewhere other than the repository: release notes from GitHub, product data from an internal API, articles from a headless CMS, changelog entries from a database. Before Astro 5, the usual pattern was to fetch in each page's frontmatter or in a shared helper, which meant every build refetched everything, nothing was validated, and pages had to know where their data came from. The content layer replaces that with loaders: functions that fill a collection from any source, store the entries with a schema, and can update incrementally between builds.

This guide writes a custom loader that pulls about 1,800 integration articles from an internal REST API into an Astro 5 collection, renders their Markdown bodies, and syncs only changes on subsequent builds. It belongs to Astro vs Eleventy for Documentation Sites, where remote data handling is one of the practical differences between the two generators.

Prerequisites

How Loaders Fit Into a Build

Sequence of a content layer loader syncing from a remote API Four actors, the Astro build, the loader, the data store and the remote API. The build calls the loader's load function. The loader reads the last sync cursor from the meta store. It asks the API for items changed since the cursor. The API returns 37 changed items and 2 deleted IDs. The loader renders Markdown for changed items, sets them in the store with digests, deletes the two removed entries and saves the new cursor. Pages then query the collection from the store. A loader during a build Astro build Loader Data store Remote API load(context) read sync cursor GET ?changed_since=cursor 37 changed, 2 deleted set entries, delete 2, save cursor getCollection at render time
On a warm build the loader touches 39 items instead of 1,800.

A loader is an object with a name and a load function. Astro calls load once per build with a context that includes the store (entries), meta (key-value storage for things like sync cursors), parseData (schema validation), generateDigest, renderMarkdown and a logger.

Step 1: Define the Collection With a Schema

// src/content.config.ts
import { defineCollection, z } from 'astro:content';
import { integrationsLoader } from './loaders/integrations';

const integrations = defineCollection({
  loader: integrationsLoader({ baseUrl: import.meta.env.INTEGRATIONS_API, token: import.meta.env.INTEGRATIONS_TOKEN }),
  schema: z.object({
    title: z.string(),
    vendor: z.string(),
    category: z.enum(['auth', 'storage', 'messaging', 'analytics', 'payments']),
    updatedAt: z.coerce.date(),
    logo: z.string().url().optional(),
  }),
});

export const collections = { integrations };

The schema is the contract with the remote source. If the API starts returning a category the site does not know, the build fails with the entry ID and field, instead of rendering a broken page.

Step 2: Write the Loader

// src/loaders/integrations.ts
import type { Loader } from 'astro/loaders';

export function integrationsLoader(opts: { baseUrl: string; token: string }): Loader {
  return {
    name: 'integrations',
    async load({ store, meta, parseData, generateDigest, renderMarkdown, logger }) {
      const cursor = meta.get('cursor');
      const url = new URL('/v1/integrations', opts.baseUrl);
      if (cursor) url.searchParams.set('changed_since', cursor);
      else store.clear();

      const res = await fetch(url, { headers: { Authorization: `Bearer ${opts.token}` } });
      if (!res.ok) throw new Error(`integrations API ${res.status}`);
      const { items, deleted, next_cursor } = await res.json();

      for (const id of deleted ?? []) store.delete(id);
      for (const item of items) {
        const data = await parseData({ id: item.slug, data: item.meta });
        const digest = generateDigest(item);
        store.set({ id: item.slug, data, digest, rendered: await renderMarkdown(item.body) });
      }
      meta.set('cursor', next_cursor);
      logger.info(`${items.length} updated, ${deleted?.length ?? 0} deleted (cursor ${cursor ? 'incremental' : 'full'})`);
    },
  };
}

store.set with a digest skips the write when the digest matches the stored entry, so re-sending unchanged items costs almost nothing. Storing rendered means pages can use render() and get Content and headings as they would for local Markdown.

Step 3: Use the Collection in Pages

---
// src/pages/integrations/[slug].astro
import { getCollection, render } from 'astro:content';
export async function getStaticPaths() {
  const items = await getCollection('integrations');
  return items.map((entry) => ({ params: { slug: entry.id }, props: { entry } }));
}
const { entry } = Astro.props;
const { Content, headings } = await render(entry);
---
<h1>{entry.data.title}</h1>
<Content />

The page has no idea the content came from an API. The index page filters by category with getCollection('integrations', (e) => e.data.category === 'auth').

Step 4: Handle Failure Without Publishing an Empty Site

The worst outcome is a build that succeeds with no content because the API returned an empty list during an incident. The loader guards against it in two ways: it throws on any non-2xx response, failing the build; and on a full sync it refuses to clear the store if the API returns fewer than 80% of the previously stored count:

const previous = Number(meta.get('count') ?? 0);
if (!cursor && previous && items.length < previous * 0.8)
  throw new Error(`integrations API returned ${items.length} items, expected about ${previous}`);
meta.set('count', String(store.keys().length));

A failed build leaves the previous deploy live, which is the right outcome.

Decision tree for handling API responses in the loader The root asks what the API returned. A non-2xx status throws and fails the build, keeping the previous deploy live. A full list under 80 percent of the previous count also throws. A valid incremental response applies changes and deletions. A valid full response clears the store and loads everything. What the loader does with each API response What did the API return? Non-2xx status Throw build fails, old deploy stays Full list under 80% Throw suspected partial data Valid incremental Apply changes set changed, delete removed Valid full list Reload clear store and set all
Both failure branches fail loudly, so a partial outage cannot publish a half-empty site.

Step 5: Refresh Content Without a Code Change

Content changes in the API need a build. The API's admin publishes a webhook on change, which triggers a GitHub Actions repository_dispatch build with the Astro cache restored. Because the loader syncs incrementally, the content-only build spends about 3 seconds in the loader instead of 41. For a cheaper option, a scheduled build every hour does the same with no webhook, as in Scheduled Content Publishing With GitHub Actions.

Testing the Loader Locally

The loader is ordinary TypeScript, so it can be tested without running a build. A Vitest test calls load with a fake context — an in-memory store and meta, a parseData that applies the Zod schema, and a stub renderMarkdown — against a mocked fetch that returns fixture responses for a full sync, an incremental sync with deletions, a 500 error and a suspiciously short list. Four tests cover the behaviour that matters in production, and they run in under a second. During development, astro dev runs the loader at startup and astro sync runs it on demand, which makes it quick to see real entries; setting INTEGRATIONS_API to a staging URL in .env.development keeps local builds off the production API.

Measured Impact

The previous implementation fetched the full list in a shared helper used by 1,800 pages, memoised per build.

MeasureFetch in pagesLoader, coldLoader, warmTool
Time in data fetching41 s38 s3.1 sAstro build log
API requests per build19 (paged)191API access log
Full build2 min 52 s2 min 49 s2 min 14 sCI timing
Invalid entries caught0 (rendered broken)4 on first run—Schema errors

The four invalid entries on the first run were integrations with a category the site did not support. Previously they had rendered as pages with an empty category badge and no index listing.

Seconds spent fetching integration content per build by approach Vertical bars. Fetching in pages took 41 seconds. The loader on a cold build took 38 seconds. The loader on a warm build with the cache restored took 3.1 seconds. Time spent fetching content per build 41 s Fetch in pages 38 s Loader cold 3.1 s Loader warm
Measured from Astro's build log timestamps on GitHub Actions, median of five builds.

Pitfalls & Rollback

  • No cache in CI. Without node_modules/.astro restored, every build is a full sync.
  • Ignoring deletions. Deleted remote items keep their pages unless the loader removes them.
  • Clearing the store on a bad response. Throw instead; a failed build is better than an empty site.
  • Secrets in client code. The token is read in the loader at build time only; never pass it to a component that ships to the browser.
  • Rollback: the pages depend only on the collection, so the loader can be replaced by the old fetch helper or by a glob loader over exported files without touching any page.

Conclusion

A content layer loader turns remote content into a validated, cached collection that pages use like local files. For 1,800 API-backed pages it caught four invalid entries on its first run, cut warm-build fetching from 41 seconds to 3, and made a partial API outage fail the build rather than publish an incomplete site.

FAQ

What is the difference between a content layer loader and fetching in a page?

Fetching in a page's frontmatter runs for every page that does it, on every build, with no caching or validation. A loader runs once per build, stores entries in Astro's data store with a schema, and can skip entries whose digest has not changed. Pages then query the collection like local content, so they do not know or care where it came from.

Do loader results persist between builds?

Yes, when the data store in node_modules/.astro is preserved between builds. A loader can read its stored entries and a sync token from the meta store, ask the API only for what changed since then, and update or delete entries accordingly. In CI the cache directory has to be saved and restored for this to work.

How do I handle content deleted in the remote source?

Either clear the store and reload everything, which is simple and safe for small collections, or have the API return deleted IDs since the last sync and call store delete for each. Without one of these, deleted items stay in the collection and their pages keep being built.

Can a loader render Markdown from the API?

Yes. The loader context includes a renderMarkdown function, and storing its result in the entry's rendered field lets pages use the render function and the Content component exactly as with local Markdown files, including headings for a table of contents.

Filed underHow-tosAstro