Files
supabase/.github/workflows/www-e2e.yml
Miranda Limonczenko 6d3a4bcc48 feat(www) Add scaffolding for WWW E2E tests and CI check (#48861)
Closes DOCS-1278

## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature. Adds E2E test scaffolding and a CI check for the marketing
site.

## What is the current behavior?

Closes [FE-4047](https://linear.app/supabase/issue/FE-4047).

The marketing site has no E2E coverage. Docs has a suite in `e2e/docs`,
but its
runner, git helpers and axe reporting are private to that package, so a
second
site cannot reuse them.

## What is the new behavior?

* **A www suite scoped to changed content.** Changed `.mdx` files in
`_blog`,
`_events`, `_customers` and `_alternatives` map to the URLs they render.
Pages
with `disable_page_build: true` are skipped because they 404 by design.
Capped
at 20 pages. Enforces `heading-order` and `page-has-heading-one`,
matching docs.
* **`e2e/shared` The docs site is also static with similar needs. This
folder shares the docs logic with www.
* **A CI check that is safe to mark required.** Path scoping lives in a
`Detect changed paths` step rather than a `paths:` trigger, so the check
  reports on every pull request instead of being skipped.
`waitForVercelDocsPreview.js` becomes `waitForVercelPreview.js`, shared
by both
  workflows.

## How the check behaves

The job always reports a check run, so it is safe to mark required. Path
scoping
happens in a step rather than a `paths:` trigger, which would leave
non-www pull
requests waiting on a check that never reports.

| Case | Behavior |
| --- | --- |
| Fork pull request adds new pages | Passes without testing. The Vercel
wait is gated on `head.repo.full_name == github.repository`, so forks
resolve no preview URL. The job emits a `::warning` and a job summary
containing a ready-to-run `gh workflow run www-e2e.yml` command with the
resolved page paths, so a maintainer can run it against the preview. |
| Vercel preview times out or fails | Passes without testing. The wait
step is `continue-on-error: true`, so a 900s timeout or a failed
deployment leaves the URL unset and the suite skips. Vercel's own
`Vercel – zone-www-dot-com` check already reports the failure. |
| Draft pull request | Job does not run at all, gated at the job level
on `pull_request.draft == false`. `ready_for_review` is in the trigger's
`types`, so marking it ready runs the check. |
| Another app changed, www untouched | Job runs and every step skips.
The `www` filter matches only the four content directories, `e2e/www`,
`e2e/shared`, the lockfile, and this workflow. |
| Only the harness changed | Passes without testing. Scope resolves to
zero pages, and the Vercel wait is additionally gated on `www_app`, so
it does not wait for a preview Vercel skipped. |
| No preview resolves, any reason | Skips rather than falling back to
production. Production does not serve pages the pull request adds, so
testing it would fail a valid change. |

### Not covered

Changes to `apps/www` components and routes do not trigger this check —
only the
four content directories do. A follow-up can check global components
such as the navigation and the footer.

## Manual testing

1. Start the site: `pnpm dev:www`
2. Run `pnpm e2e:www` with no www content changed. It should resolve
zero pages
   and skip Playwright, not fail.
3. Touch a post, then run `pnpm e2e:www` again:
`echo "" >> apps/www/_blog/2024-01-01-some-post.mdx`. The resolved
`/blog/...`
   path should be listed before Playwright starts.
4. Run against production with no local server:
`PLAYWRIGHT_BASE_URL=https://supabase.com
WWW_E2E_PAGE_PATHS=/blog/postgres-language-server pnpm e2e:www`
5. Point step 4 at a page with a known heading problem. The failure
should name
   the rule, the CSS selector and the markup.
6. Confirm docs still passes on the shared runner: `pnpm dev:docs`, then
   `pnpm e2e:docs`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added WWW end-to-end testing for affected content pages, including
accessibility checks.
* Added standard and full-site test commands, configurable preview
testing, and failure reports.
* Added shared utilities for page discovery, accessibility scanning, and
test execution.

* **Documentation**
* Documented WWW test setup, coverage, debugging, CI behavior, and
running checks against production or preview environments.

* **Improvements**
* Updated documentation test workflows to better identify affected
changes and handle preview environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 22:06:51 +00:00

209 lines
7.9 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
name: WWW E2E Tests
# Path scoping lives in "Detect changed paths" rather than a `paths` trigger, so
# this reports a check run on every pull request and stays safe to mark required.
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, converted_to_draft]
branches: ['master']
workflow_dispatch:
inputs:
base_url:
description: 'Base URL to test against'
required: false
default: 'https://supabase.com'
type: string
page_paths:
description: 'Comma-separated site-relative paths to test (required for manual runs)'
required: false
default: ''
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
statuses: read
pull-requests: read
env:
CI: true
jobs:
e2e:
name: WWW E2E
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.draft == false
timeout-minutes: 30
runs-on: blacksmith-4vcpu-ubuntu-2404
steps:
- name: Detect changed paths
id: changes
if: github.event_name == 'pull_request'
uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
with:
filters: |
www:
- 'apps/www/_blog/**/*.mdx'
- 'apps/www/_events/**/*.mdx'
- 'apps/www/_customers/**/*.mdx'
- 'apps/www/_alternatives/**/*.mdx'
- 'e2e/www/**'
- 'e2e/shared/**'
- 'pnpm-lock.yaml'
- '.github/workflows/www-e2e.yml'
www_app:
- 'apps/www/**'
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
if: github.event_name == 'workflow_dispatch' || steps.changes.outputs.www == 'true'
with:
persist-credentials: false
# String '0' — numeric 0 is falsy in Actions expressions.
fetch-depth: ${{ github.event_name == 'pull_request' && '0' || '1' }}
sparse-checkout: |
e2e/www
e2e/shared
scripts
patches
apps/www/_blog
apps/www/_events
apps/www/_customers
apps/www/_alternatives
- name: Use Node.js
if: github.event_name == 'workflow_dispatch' || steps.changes.outputs.www == 'true'
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version-file: '.nvmrc'
- name: Resolve www E2E scope
id: scope
if: github.event_name == 'workflow_dispatch' || steps.changes.outputs.www == 'true'
env:
EVENT_NAME: ${{ github.event_name }}
BASE_REF: ${{ github.base_ref }}
PAGE_PATHS_INPUT: ${{ inputs.page_paths }}
run: |
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
if [ -z "$PAGE_PATHS_INPUT" ]; then
echo "skip=true" >> "$GITHUB_OUTPUT"
echo "paths=" >> "$GITHUB_OUTPUT"
echo "Manual run requires the page_paths input."
exit 0
fi
echo "skip=false" >> "$GITHUB_OUTPUT"
printf 'paths=%s\n' "$PAGE_PATHS_INPUT" >> "$GITHUB_OUTPUT"
exit 0
fi
git diff --name-only --diff-filter=ACMR "origin/$BASE_REF"...HEAD \
| node --experimental-strip-types e2e/www/scripts/resolve-www-scope.ts
- name: Skip Playwright (no in-scope pages)
if: steps.scope.outputs.skip == 'true'
run: echo "No in-scope www pages changed; skipping Playwright suite."
- uses: pnpm/action-setup@0ebf47130e4866e96fce0953f49152a61190b271 # v6.0.9
if: steps.scope.outputs.skip == 'false'
name: Install pnpm
with:
run_install: false
- name: Enable pnpm store cache
if: steps.scope.outputs.skip == 'false'
uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0
with:
node-version-file: '.nvmrc'
cache: 'pnpm'
# Vercel skips the preview when only the harness changed, so wait for one
# only when apps/www changed. See scripts/waitForVercelPreview.js.
- name: Wait for Vercel www preview
if: steps.scope.outputs.skip == 'false' && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && steps.changes.outputs.www_app == 'true'
id: deployment
continue-on-error: true
run: node scripts/waitForVercelPreview.js
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
VERCEL_STATUS_CONTEXT: 'Vercel zone-www-dot-com'
VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
VERCEL_TEAM_ID: ${{ secrets.VERCEL_TEAM_ID }}
- name: Resolve base URL
if: steps.scope.outputs.skip == 'false'
id: base-url
env:
EVENT_NAME: ${{ github.event_name }}
BASE_URL_INPUT: ${{ inputs.base_url }}
DEPLOYMENT_URL: ${{ steps.deployment.outputs.deployment-url }}
PAGE_PATHS: ${{ steps.scope.outputs.paths }}
run: |
set -euo pipefail
if [ "$EVENT_NAME" = "workflow_dispatch" ]; then
printf 'url=%s\n' "$BASE_URL_INPUT" >> "$GITHUB_OUTPUT"
if [ "$BASE_URL_INPUT" = "https://supabase.com" ]; then
echo "use_bypass=false" >> "$GITHUB_OUTPUT"
else
echo "use_bypass=true" >> "$GITHUB_OUTPUT"
fi
echo "should_test=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ -n "$DEPLOYMENT_URL" ]; then
printf 'url=%s\n' "$DEPLOYMENT_URL" >> "$GITHUB_OUTPUT"
echo "use_bypass=true" >> "$GITHUB_OUTPUT"
echo "should_test=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# Production lacks pages this pull request adds, so it is not a fallback.
echo "url=" >> "$GITHUB_OUTPUT"
echo "use_bypass=false" >> "$GITHUB_OUTPUT"
echo "should_test=false" >> "$GITHUB_OUTPUT"
echo "::warning::No Vercel www preview URL for this pull request. Skipping Playwright rather than testing production, which does not have pages this pull request adds."
{
echo "### WWW E2E skipped: no preview to test against"
echo
echo "Fork pull requests reach this path because they run without repository"
echo "secrets. A maintainer can run the suite against the preview manually:"
echo
echo '```'
echo "gh workflow run www-e2e.yml \\"
echo " -f base_url=<preview-url> \\"
echo " -f page_paths=$PAGE_PATHS"
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
- name: Install dependencies
if: steps.base-url.outputs.should_test == 'true'
run: pnpm install --frozen-lockfile --filter=e2e-www...
- name: Install Playwright Chromium
if: steps.base-url.outputs.should_test == 'true'
run: pnpm -C e2e/www exec playwright install chromium --with-deps --only-shell
- name: Run www E2E
if: steps.base-url.outputs.should_test == 'true'
working-directory: e2e/www
run: pnpm run e2e:www
env:
PLAYWRIGHT_BASE_URL: ${{ steps.base-url.outputs.url }}
WWW_E2E_PAGE_PATHS: ${{ steps.scope.outputs.paths }}
VERCEL_AUTOMATION_BYPASS_SECRET: ${{ steps.base-url.outputs.use_bypass == 'true' && secrets.VERCEL_AUTOMATION_BYPASS_WWW || '' }}
- name: Upload Playwright report
if: failure() && steps.scope.outputs.skip == 'false'
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: www-playwright-report
path: |
e2e/www/playwright-report/
e2e/www/test-results/
retention-days: 7