Commenting on Preview Deploys

Preview deploys are only useful if people look at them. The default experience on most hosts is a status check with a link to the preview's home page, and a reviewer of a pull request that changed three pages deep in the documentation has to navigate there, guess which pages changed, and remember to check them all. Writers learn that reviewers skip previews, and reviewers learn that previews take effort. The fix is small: one comment on the pull request, updated on every push, that links directly to the changed pages, shows check results and thumbnails, and makes the preview the obvious place to review.

This guide builds that comment for a documentation site previewed on Cloudflare Workers, with the list of changed pages derived from both source changes and built output, check results, visual thumbnails, and a way for reviewers to leave feedback on the preview itself. It is part of Preview Environments for Pull Requests, and builds on Automating Preview Deploy Pipelines with GitHub Actions.

Prerequisites

• A preview deploy per pull request with a stable URL per pull request (a branch alias or a pr-N subdomain).
• GitHub Actions with pull-requests: write permission.
• A build that can be compared with the base branch build (the base build's output cached as an artifact, or rebuilt).

Step 1: Find the Pages That Changed

Source changes map to URLs directly for content files: content/guide/install.md becomes /guide/install/. But a pull request that edits a shared partial, a navigation data file or a shortcode changes many pages without touching their source. Comparing built output catches those:

// scripts/changed-pages.mjs — compare the PR build with the base build by content hash
const base = await hashHtml('base-dist');     // { '/guide/install/': 'a91f…', … }
const head = await hashHtml('dist');
const changed = Object.keys(head).filter((u) => base[u] && base[u] !== head[u]);
const added = Object.keys(head).filter((u) => !base[u]);
const removed = Object.keys(base).filter((u) => !head[u]);
await writeFile('changed-pages.json', JSON.stringify({ changed, added, removed }));

The hash is computed over the normalized main content only (the same normalization as in Snapshot Testing Build Output), so pages that differ only in a sidebar entry do not all appear as changed. When more than 20 pages changed, the comment lists the first 20 and a count, which is itself useful information for the reviewer of a "small" template change.

Step 2: One Comment, Updated in Place

A new comment on every push buries the conversation. A hidden marker identifies the bot's comment so each run updates it:

- name: Build comment body
  run: node scripts/preview-comment.mjs > comment.md   # reads changed-pages.json, check results, preview URL
- uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728   # v2.9.1
  with:
    header: preview
    path: comment.md

The comment body, generated from the data gathered earlier:

### Preview for 3f9c1a2 — https://pr-482.docs-preview.example.com

**Changed pages (3)**
- [Install on Linux](https://pr-482.docs-preview.example.com/guide/install/linux/) — text and 1 code block
- [Configure SSO](https://pr-482.docs-preview.example.com/guide/sso/) — new section "Okta"
- [Troubleshooting](https://pr-482.docs-preview.example.com/guide/troubleshooting/) — link added

**Checks:** ✅ links · ✅ HTML · ✅ accessibility (3 pages) · ⚠️ LCP +120 ms on /guide/sso/

The per-page summaries ("new section", "link added") come from comparing heading outlines and link counts between the two builds. The commit SHA in the heading tells reviewers which push they are looking at.

Step 3: Thumbnails of Changed Pages

For content reviewers, a picture of the changed section is often enough. The workflow captures screenshots of each changed page on the preview (the top of the page and the first changed section, located by heading), uploads them as workflow artifacts or to a small bucket, and adds them to the comment in a collapsed <details> block, so the comment stays short. Visual differences from the base branch, when there are any, come from the visual regression check in Visual Regression Testing on Preview Deploys and are linked rather than embedded.

Step 4: Feedback on the Preview Itself

Reviewers from outside engineering — product managers, support, legal — find it easier to comment on the page than on a Markdown diff. Previews include a small feedback widget, loaded only on preview hostnames, that lets a reviewer select text and leave a comment. The widget posts to an edge function that creates a pull request review comment through the GitHub API, quoting the selected text and linking to the page and heading. Vercel's comments toolbar and Netlify's collaborative deploy previews offer the same idea on those platforms. On this site, 40% of review comments from non-engineers arrived through the widget in its first three months.

Step 5: Clean Up When the Pull Request Closes

When the pull request is merged or closed, the comment is updated a final time — "Preview removed; merged in 9b7d013" — and the preview is deleted, following Cleaning Up Stale Preview Deployments. Updating rather than deleting the comment keeps the history of what was reviewed.

Writing the Comment for Its Readers

The comment is read by different people for different reasons, and its layout reflects that. Writers want confirmation that their change built and where to check it, so the preview link and the changed pages come first. Reviewers want to know what to look at and whether anything is broken, so each changed page has a one-line summary of what changed and the check results sit on one line with symbols that scan quickly. Maintainers want the details only when something fails, so full check output, screenshots and the list of indirectly changed pages are folded into <details> blocks. Keeping the visible part under about fifteen lines was a deliberate rule: an early version with every check's full output inline was scrolled past rather than read.

The comment also avoids noise when nothing reviewable changed. A pull request that only touches CI configuration or the README gets a one-line comment saying no pages changed, rather than a preview link that shows an identical site. And when the build fails, the comment says so in its first line with a link to the failing step, instead of silently keeping the previous push's preview link, which had once led a reviewer to approve content from an older commit.

The comment script itself is tested like the rest of CI, with fixture builds that exercise zero, few and many changed pages.

Measured Impact

Pitfalls & Rollback

• A new comment per push. Use a marker and update in place.
• Listing every page after a template change. Cap the list and show a count; compare main content only.
• Linking the preview root. The value is in direct links to changed pages.
• Feedback widgets on production. Load them only on preview hostnames.
• Rollback: the comment is one workflow step; removing it leaves the host's default status check in place.

Conclusion

A preview is only reviewed if it is easy to review. One self-updating comment that links straight to the pages a pull request changed, summarizes what changed on each, shows check results and thumbnails, and lets reviewers comment on the page itself doubled the share of pull requests whose preview was actually opened on this site, and cut time to first review by more than half.

FAQ

Why post a custom comment when the host already adds a deploy status?

A deploy status shows one link to the site root. A custom comment can link straight to the pages the pull request changed, include check results and screenshots, and update itself on each push, which is what reviewers need to review content quickly.

How do I avoid a new comment on every push?

Include a hidden marker in the comment body, find an existing comment with that marker, and update it instead of creating a new one. Actions such as marocchino/sticky-pull-request-comment implement this pattern.

How are changed pages found?

Map changed source files from the pull request diff to URLs, for example content paths to page paths, and add pages that changed indirectly by comparing a structural summary or hashes of the built HTML against the base branch build.

Can reviewers comment on the preview page itself?

Yes. Tools such as Vercel's comments toolbar, Netlify's Drawer or a lightweight feedback widget on previews let reviewers leave comments on specific parts of a page, which are then linked back to the pull request.

Related

• Parent: Preview Environments for Pull Requests — previews in general.
• Preview Deploys for Monorepos — one comment for several sites.
• Docs-as-Code Review Workflow for Writers — the review process around the comment.
• Branch Deploy Controls on Cloudflare Pages — stable preview aliases.
• Per-Template Performance Budgets — the performance line in the comment.