Rolling Back S3 and CloudFront Deploys With Versioned Prefixes

Managed static hosts make deploys atomic and rollbacks instant: every deployment is kept, and switching between them is a pointer change. A static site on S3 and CloudFront gets neither by default. aws s3 sync uploads and deletes files one at a time, so for a minute or two the bucket holds half of each build, and rolling back means syncing the old build again with the same exposure. An incident from exactly that exposure is described in Recovering From a Deploy That Deleted Live Assets.

Versioned prefixes give S3 the managed-host model: upload each complete build to its own prefix, then switch the live site to it in one step. This guide sets it up for an Astro documentation site with a CloudFront Function and CloudFront KeyValueStore holding the current build ID, so a deploy or rollback is one key update that takes effect in seconds. It belongs to Rollbacks and Deploy Safety for Static Sites; the base setup is in Deploying a Static Site to S3 and CloudFront.

Prerequisites

  • An S3 bucket behind a CloudFront distribution with origin access control.
  • A CI role with permission to write to the bucket, update a CloudFront KeyValueStore and create invalidations.
  • Hashed filenames for assets, so builds' assets never collide.

The Layout

s3://acme-docs/
  builds/
    3f9a1c2/   ← previous build, complete
    8b41e07/   ← current build, complete
    c72d5aa/   ← new build, being uploaded
How a request is routed to the current build prefix Four layers. The reader requests docs install on the production hostname. A CloudFront Function reads the current build ID from the KeyValueStore and rewrites the path to builds slash the build ID slash docs install index.html. CloudFront fetches that object from S3 on a cache miss. S3 holds each build complete under its own prefix. Request path through the prefix switch Request: /docs/install/ production hostname unchanged URLs CloudFront Function reads current = 8b41e07 viewer request rewrite CloudFront cache keyed by original URL invalidate HTML on switch S3: builds/8b41e07/… complete build previous builds kept
Changing one value in the KeyValueStore moves every request to a different complete build.

Step 1: The Edge Function

A CloudFront Function on viewer requests reads the current build from a KeyValueStore and prefixes the path. It also handles directory indexes, which S3's REST endpoint does not:

import cf from 'cloudfront';
const kvs = cf.kvs();
async function handler(event) {
  const req = event.request;
  const build = await kvs.get('current');
  let uri = req.uri;
  if (uri.endsWith('/')) uri += 'index.html';
  else if (!uri.split('/').pop().includes('.')) uri += '/index.html';
  req.uri = `/builds/${build}${uri}`;
  return req;
}

The function runs in under a millisecond per request at the edge. KeyValueStore reads are local to the edge location; updates propagate globally within seconds.

Step 2: Deploy = Upload, Verify, Switch

BUILD=$(git rev-parse --short HEAD)
aws s3 sync dist "s3://$BUCKET/builds/$BUILD/" \
  --exclude '*.html' --cache-control 'public, max-age=31536000, immutable'
aws s3 sync dist "s3://$BUCKET/builds/$BUILD/" \
  --exclude '*' --include '*.html' --cache-control 'public, max-age=0, s-maxage=3600, must-revalidate'

# verify the new build directly before switching (internal header lets the function pick a build)
node scripts/verify-build.mjs --build "$BUILD"

ETAG=$(aws cloudfront-keyvaluestore describe-key-value-store --kvs-arn "$KVS" --query ETag --output text)
aws cloudfront-keyvaluestore put-key --kvs-arn "$KVS" --key current --value "$BUILD" --if-match "$ETAG"
aws cloudfront create-invalidation --distribution-id "$DIST" --paths '/*.html' '/*/' '/'

The live site keeps serving the previous build until put-key. The verification step requests 30 pages and their assets from the new prefix through a staging hostname whose function reads a preview key instead of current, so the exact files that will go live are tested first. --if-match makes concurrent deploys fail safely instead of overwriting each other.

Steps of a deploy with versioned prefixes, and the rollback path Five steps. Upload the new build to its own prefix while the old one keeps serving. Verify the new prefix through a staging hostname. Update the current key in the KeyValueStore to the new build ID. Invalidate cached HTML. To roll back, set the key back to the previous build ID and invalidate again. Deploy and rollback are the same operation Upload to builds/ID old build still live Verify via staging 30 pages + assets put-key current=ID switch in seconds Invalidate HTML cached pages refresh Rollback: put-key previous + invalidate
A rollback skips the upload and verification entirely, because the previous build is still complete in S3.

Step 3: Rollback

Rolling back is setting the key to the previous build ID and invalidating HTML:

PREV=$(aws s3 ls "s3://$BUCKET/builds/" | awk '{print $2}' | tr -d / | grep -v "$CURRENT" | tail -1)   # or read from deploy log
aws cloudfront-keyvaluestore put-key --kvs-arn "$KVS" --key current --value "$PREV" --if-match "$ETAG"
aws cloudfront create-invalidation --distribution-id "$DIST" --paths '/*.html' '/*/' '/'

A GitHub Actions workflow with a manual trigger and a build-ID input wraps it, listing the last ten builds with their commit messages. Readers with HTML cached at the edge get the rolled-back HTML once the invalidation completes, typically within a minute; assets for both builds exist, so nothing in between breaks.

Step 4: Pruning Old Builds

Each build is about 180 MB (mostly images). A scheduled job keeps the last 10 builds and any build deployed in the last 14 days, and deletes the rest. Because hashed assets are duplicated across build prefixes, storage is higher than with a single synced folder — about 1.8 GB for ten builds, around $0.04 a month at standard S3 pricing. A variant that stores hashed assets in one shared assets/ prefix and only HTML per build cuts that to a few hundred megabytes, at the cost of a slightly more complex function.

Testing a Build Before It Goes Live

The staging hostname is the quiet benefit of this layout. It is a second CloudFront distribution — or a second alternate domain name on the same one — whose function reads the preview key instead of current. The deploy sets preview to the new build first, runs the verification suite, Lighthouse CI and a short visual comparison against that hostname, and only then sets current. Because both hostnames serve the very same objects from the same prefix, what passed on staging is byte-for-byte what readers get. Previously, checks ran against a local build, and differences in headers, redirects and CDN behaviour only showed up after a deploy.

The staging hostname sends X-Robots-Tag: noindex from a response headers policy, and its CloudFront distribution is restricted to the company's IP ranges with a WAF rule, so it never becomes a second public copy of the site.

Recording What Is Live

The KeyValueStore holds the current build ID, but people need more than a hash. Each deploy writes a small JSON record — build ID, commit, author, time, verification results — to a deploys/ prefix, and the rollback workflow reads the last ten records to offer a readable list. The same record is exposed at /_build.json on the site, which the post-deploy checks and the support team use to confirm which build a reader saw.

Measured Impact

Compared with the previous aws s3 sync --delete deploys:

MeasureSync deploysVersioned prefixesSource
Window with mixed old/new files40–110 s per deploy0Deploy logs
Rollback time (decision to live)4–6 min (rebuild + sync)35 s (key + invalidation)Runbook drills
Asset 404s after deploysup to 6% (incident)<0.01%CloudFront metrics
Builds testable before going liveNoYes (staging hostname)—
Storage180 MB1.8 GB (10 builds)S3 metrics
Seconds from rollback decision to the old build serving, before and after Paired bars. With sync deploys, a rollback took 300 seconds on average across five drills, and with versioned prefixes 35 seconds. The time readers could see mixed files during a deploy went from 75 seconds on average to 0. Rollback time in drills Sync deploys Versioned prefixes Rollback time 300 s 35 s Mixed-file window per deploy 75 s 0 s
Five rollback drills each; mixed-file window from deploy logs, averaged over 20 deploys.

Pitfalls & Rollback

  • Forgetting directory indexes. S3's REST endpoint does not resolve index.html; the function must.
  • Switching before the upload finishes. Verify the new prefix, then switch.
  • Not invalidating HTML after a switch. The cache is keyed by URL and keeps old pages.
  • Concurrent deploys. Use --if-match on the key update.
  • Rollback: this setup is the rollback mechanism; the previous build stays complete until pruned.

Conclusion

Uploading each build to its own S3 prefix and pointing CloudFront at it through a CloudFront Function and KeyValueStore gives an S3-hosted static site the deploy model of a managed host: no window of mixed files, a full build verified before it goes live, and rollback in 35 seconds by changing one key. The cost is a few cents a month of extra storage for kept builds.

FAQ

Why are S3 sync deploys not atomic?

A sync uploads and deletes files one by one over many seconds or minutes. During that time the bucket holds a mix of old and new files, so readers can receive new HTML that references assets not yet uploaded, or old HTML whose assets were just deleted. Rolling back means syncing the old build again, which has the same problem in reverse.

How do versioned prefixes make deploys atomic?

Each build is uploaded in full to its own prefix, such as builds slash a commit hash, while the live site keeps serving the previous prefix. When the upload is complete, one change points the site at the new prefix. Readers see either the old build or the new one, never a mix, and the previous prefix stays intact for rollback.

How is the switch between prefixes made?

Either by changing the CloudFront distribution's origin path, which takes a few minutes to propagate, or by a CloudFront Function or Lambda at the edge that reads the current build identifier from a key value store and rewrites request paths, which switches in seconds. This guide uses a CloudFront Function with KeyValueStore.

What happens to cached files when the prefix changes?

CloudFront caches by request URL, not by origin path, so HTML cached from the old prefix is served until it expires or is invalidated. Invalidate HTML after switching. Hashed assets have unique names, so old and new builds' assets never collide in the cache.

Filed underHow-tosAWS (S3 and CloudFront)