mirror of https://github.com/supabase/supabase.git synced 2026-09-09 19:42:46 +08:00

Files

Miranda Limonczenko 6d3a4bcc48 feat(www) Add scaffolding for WWW E2E tests and CI check (#48861 )

Closes DOCS-1278

## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature. Adds E2E test scaffolding and a CI check for the marketing
site.

## What is the current behavior?

Closes [FE-4047](https://linear.app/supabase/issue/FE-4047).

The marketing site has no E2E coverage. Docs has a suite in `e2e/docs`,
but its
runner, git helpers and axe reporting are private to that package, so a
second
site cannot reuse them.

## What is the new behavior?

* **A www suite scoped to changed content.** Changed `.mdx` files in
`_blog`,
`_events`, `_customers` and `_alternatives` map to the URLs they render.
Pages
with `disable_page_build: true` are skipped because they 404 by design.
Capped
at 20 pages. Enforces `heading-order` and `page-has-heading-one`,
matching docs.
* **`e2e/shared` The docs site is also static with similar needs. This
folder shares the docs logic with www.
* **A CI check that is safe to mark required.** Path scoping lives in a
`Detect changed paths` step rather than a `paths:` trigger, so the check
  reports on every pull request instead of being skipped.
`waitForVercelDocsPreview.js` becomes `waitForVercelPreview.js`, shared
by both
  workflows.

## How the check behaves

The job always reports a check run, so it is safe to mark required. Path
scoping
happens in a step rather than a `paths:` trigger, which would leave
non-www pull
requests waiting on a check that never reports.

| Case | Behavior |
| --- | --- |
| Fork pull request adds new pages | Passes without testing. The Vercel
wait is gated on `head.repo.full_name == github.repository`, so forks
resolve no preview URL. The job emits a `::warning` and a job summary
containing a ready-to-run `gh workflow run www-e2e.yml` command with the
resolved page paths, so a maintainer can run it against the preview. |
| Vercel preview times out or fails | Passes without testing. The wait
step is `continue-on-error: true`, so a 900s timeout or a failed
deployment leaves the URL unset and the suite skips. Vercel's own
`Vercel – zone-www-dot-com` check already reports the failure. |
| Draft pull request | Job does not run at all, gated at the job level
on `pull_request.draft == false`. `ready_for_review` is in the trigger's
`types`, so marking it ready runs the check. |
| Another app changed, www untouched | Job runs and every step skips.
The `www` filter matches only the four content directories, `e2e/www`,
`e2e/shared`, the lockfile, and this workflow. |
| Only the harness changed | Passes without testing. Scope resolves to
zero pages, and the Vercel wait is additionally gated on `www_app`, so
it does not wait for a preview Vercel skipped. |
| No preview resolves, any reason | Skips rather than falling back to
production. Production does not serve pages the pull request adds, so
testing it would fail a valid change. |

### Not covered

Changes to `apps/www` components and routes do not trigger this check —
only the
four content directories do. A follow-up can check global components
such as the navigation and the footer.

## Manual testing

1. Start the site: `pnpm dev:www`
2. Run `pnpm e2e:www` with no www content changed. It should resolve
zero pages
   and skip Playwright, not fail.
3. Touch a post, then run `pnpm e2e:www` again:
`echo "" >> apps/www/_blog/2024-01-01-some-post.mdx`. The resolved
`/blog/...`
   path should be listed before Playwright starts.
4. Run against production with no local server:
`PLAYWRIGHT_BASE_URL=https://supabase.com
WWW_E2E_PAGE_PATHS=/blog/postgres-language-server pnpm e2e:www`
5. Point step 4 at a page with a known heading problem. The failure
should name
   the rule, the CSS selector and the markup.
6. Confirm docs still passes on the shared runner: `pnpm dev:docs`, then
   `pnpm e2e:docs`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added WWW end-to-end testing for affected content pages, including
accessibility checks.
* Added standard and full-site test commands, configurable preview
testing, and failure reports.
* Added shared utilities for page discovery, accessibility scanning, and
test execution.

* **Documentation**
* Documented WWW test setup, coverage, debugging, CI behavior, and
running checks against production or preview environments.

* **Improvements**
* Updated documentation test workflows to better identify affected
changes and handle preview environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

2026-08-11 22:06:51 +00:00

6.5 KiB

Raw Permalink Blame History

WWW E2E tests

This guide explains how to run Playwright end-to-end checks against marketing site content pages.

Use this suite when you change blog posts, events, customer stories, or alternatives pages under apps/www. It loads each in-scope page, checks that it returns a successful status, and scans it for a single accessibility rule.

This page covers:

Set up — install the browser once
Run the tests — the usual local command
Choose a target URL — production, preview, or local www
Override which pages run — when the default git scope is wrong
What the suite covers — in-scope paths and limits
Debug failures — reports and traces
How CI uses this suite — pull request behavior

Set up

From this directory, install the Playwright Chromium browser once:
```
cd e2e/www
pnpm exec playwright install chromium
```

Run the tests

By default, pnpm e2e:www tests pages affected by your current changes: commits since origin/master, plus staged and unstaged working-tree files. If nothing in scope changed, the command exits successfully without starting Playwright.

From the repository root, point the suite at a deployed site and run it:
```
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:www
```

Extra arguments pass through to Playwright, so pnpm e2e:www --ui opens UI mode and pnpm e2e:www --grep @a11y runs only the accessibility assertions.

Run every in-scope page

To test every content page instead of a changed-files scope — for example, a periodic full-site check — run:

PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:www:all

This ignores WWW_E2E_PAGE_PATHS and the 20-page cap described in Limits, and tests every page across all four content directories — around 480 as of this writing. --all runs also default to --max-failures=0, so a full run isn't cut short by playwright.config.ts's global maxFailures: 3. The suite runs one worker by default, so pass --workers to parallelize it:

PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:www:all -- --workers=4

Against production, raising workers does not pay off: a serial full run finishes in about 17 minutes with no failures, while four workers took longer and timed out on 34 of 480 navigations. Those timeouts are load, not page defects. Prefer the default single worker unless you are pointed at a preview or a local server.

Choose a target URL

Tests use PLAYWRIGHT_BASE_URL. When unset, they default to the local www dev server at http://localhost:3000.

Prefer a deployed site for day-to-day checks. Use the local server only when you need unpublished content that production does not serve yet.

For a protected Vercel preview, also set VERCEL_AUTOMATION_BYPASS_SECRET.

To use the local server, start it with pnpm dev:www from the repository root and run the suite without PLAYWRIGHT_BASE_URL.

Override which pages run

Leave WWW_E2E_PAGE_PATHS unset to keep the default changed-files scope.

To test specific pages instead of the git diff:

WWW_E2E_PAGE_PATHS=/blog/supabase-steve-chavez \
  PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:www

WWW_E2E_PAGE_PATHS accepts a comma- or newline-separated list of site-relative paths. WWW_E2E_BASE_REF overrides the base ref the git diff compares against.

What the suite covers

In scope

Changed path	Behavior
`apps/www/_blog/*.mdx`	Test `/blog/<slug>`
`apps/www/_events/*.mdx`	Test `/events/<slug>`
`apps/www/_customers/*.mdx`	Test `/customers/<slug>`
`apps/www/_alternatives/*.mdx`	Test `/alternatives/<slug>`

Slugs come from the filename, matching getAllPostSlugs in apps/www/lib/posts.tsx. Blog and event filenames drop their YYYY-MM-DD- prefix; customers and alternatives use the filename as-is.

Each page gets one test: it must return a successful status, and an axe scan must report no page-has-heading-one violations. That is the only rule enforced today — add more to ENFORCED_RULES in features/www-pages.spec.ts once a class of issue reaches zero across the site.

ENFORCED_RULES is deliberately separate from the docs suite's list. Docs enforces heading-order as well; www cannot yet. A full-site scan found heading-order violations on the large majority of content pages, almost all from the same two shared components — the related-posts card (h4 under an h2) and a trailing h6. Enforcing it here would fail nearly every pull request.

Out of scope

Events with disable_page_build: true, which return a 404 by design
Static marketing routes under apps/www/pages and apps/www/app
Index and listing pages such as /blog and /customers
Link checking, and every accessibility rule other than the one above

Limits

Resolved scope is capped at 20 pages so a large content drop cannot explode runtime. If a change resolves to more pages than that, only the first 20 in sorted order are tested. To test beyond the cap, use pnpm e2e:www:all or set WWW_E2E_PAGE_PATHS explicitly.

Debug failures

Open the HTML report after a run:

pnpm -C e2e/www exec playwright show-report

Inspect traces and screenshots under test-results/ for failed runs.

How CI uses this suite

The workflow at .github/workflows/www-e2e.yml runs on pull requests that touch owned www content, e2e/www, or e2e/shared.

Diff the pull request against its base branch and resolve in-scope page paths.
Skip Playwright when nothing in scope changed.
When apps/www changed, wait for the Vercel www preview and set PLAYWRIGHT_BASE_URL to that preview. When no preview resolves, skip rather than test against production, which does not have pages the pull request adds.
Run the suite with WWW_E2E_PAGE_PATHS set to the resolved list.

Draft pull requests stay skipped until you mark them ready for review. Manual workflow_dispatch runs require a page_paths input and accept an optional base_url, which defaults to production.

Shared helpers

e2e/shared holds the pieces this suite and e2e/docs both use: git diff collection, page-path parsing, the runner, and the scope-resolver CLI. Suite directories keep only what is specific to them — for www, that is the content-file-to-URL mapping in utils/resolve-www-scope.ts.

6.5 KiB Raw Permalink Blame History