Files
supabase/e2e/shared/run-suite.ts
Miranda Limonczenko 6d3a4bcc48 feat(www) Add scaffolding for WWW E2E tests and CI check (#48861)
Closes DOCS-1278

## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature. Adds E2E test scaffolding and a CI check for the marketing
site.

## What is the current behavior?

Closes [FE-4047](https://linear.app/supabase/issue/FE-4047).

The marketing site has no E2E coverage. Docs has a suite in `e2e/docs`,
but its
runner, git helpers and axe reporting are private to that package, so a
second
site cannot reuse them.

## What is the new behavior?

* **A www suite scoped to changed content.** Changed `.mdx` files in
`_blog`,
`_events`, `_customers` and `_alternatives` map to the URLs they render.
Pages
with `disable_page_build: true` are skipped because they 404 by design.
Capped
at 20 pages. Enforces `heading-order` and `page-has-heading-one`,
matching docs.
* **`e2e/shared` The docs site is also static with similar needs. This
folder shares the docs logic with www.
* **A CI check that is safe to mark required.** Path scoping lives in a
`Detect changed paths` step rather than a `paths:` trigger, so the check
  reports on every pull request instead of being skipped.
`waitForVercelDocsPreview.js` becomes `waitForVercelPreview.js`, shared
by both
  workflows.

## How the check behaves

The job always reports a check run, so it is safe to mark required. Path
scoping
happens in a step rather than a `paths:` trigger, which would leave
non-www pull
requests waiting on a check that never reports.

| Case | Behavior |
| --- | --- |
| Fork pull request adds new pages | Passes without testing. The Vercel
wait is gated on `head.repo.full_name == github.repository`, so forks
resolve no preview URL. The job emits a `::warning` and a job summary
containing a ready-to-run `gh workflow run www-e2e.yml` command with the
resolved page paths, so a maintainer can run it against the preview. |
| Vercel preview times out or fails | Passes without testing. The wait
step is `continue-on-error: true`, so a 900s timeout or a failed
deployment leaves the URL unset and the suite skips. Vercel's own
`Vercel – zone-www-dot-com` check already reports the failure. |
| Draft pull request | Job does not run at all, gated at the job level
on `pull_request.draft == false`. `ready_for_review` is in the trigger's
`types`, so marking it ready runs the check. |
| Another app changed, www untouched | Job runs and every step skips.
The `www` filter matches only the four content directories, `e2e/www`,
`e2e/shared`, the lockfile, and this workflow. |
| Only the harness changed | Passes without testing. Scope resolves to
zero pages, and the Vercel wait is additionally gated on `www_app`, so
it does not wait for a preview Vercel skipped. |
| No preview resolves, any reason | Skips rather than falling back to
production. Production does not serve pages the pull request adds, so
testing it would fail a valid change. |

### Not covered

Changes to `apps/www` components and routes do not trigger this check —
only the
four content directories do. A follow-up can check global components
such as the navigation and the footer.

## Manual testing

1. Start the site: `pnpm dev:www`
2. Run `pnpm e2e:www` with no www content changed. It should resolve
zero pages
   and skip Playwright, not fail.
3. Touch a post, then run `pnpm e2e:www` again:
`echo "" >> apps/www/_blog/2024-01-01-some-post.mdx`. The resolved
`/blog/...`
   path should be listed before Playwright starts.
4. Run against production with no local server:
`PLAYWRIGHT_BASE_URL=https://supabase.com
WWW_E2E_PAGE_PATHS=/blog/postgres-language-server pnpm e2e:www`
5. Point step 4 at a page with a known heading problem. The failure
should name
   the rule, the CSS selector and the markup.
6. Confirm docs still passes on the shared runner: `pnpm dev:docs`, then
   `pnpm e2e:docs`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added WWW end-to-end testing for affected content pages, including
accessibility checks.
* Added standard and full-site test commands, configurable preview
testing, and failure reports.
* Added shared utilities for page discovery, accessibility scanning, and
test execution.

* **Documentation**
* Documented WWW test setup, coverage, debugging, CI behavior, and
running checks against production or preview environments.

* **Improvements**
* Updated documentation test workflows to better identify affected
changes and handle preview environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-11 22:06:51 +00:00

109 lines
3.3 KiB
TypeScript

import { spawn } from 'node:child_process'
import { collectChangedFiles, repoRootFrom } from './git.ts'
export type ScopeResult = { pages: string[]; skip: boolean }
export type SuiteConfig = {
root: string
label: string
devCommand: string
pagePathsEnv: string
baseRefEnv: string
defaultBaseUrl: string
resolveScope: (options: { changedFiles: string[]; repoRoot: string }) => Promise<ScopeResult>
resolveAllPages?: (repoRoot: string) => Promise<string[]>
}
const PREFLIGHT_TIMEOUT_MS = 3_000
async function isBaseUrlReachable(baseUrl: string): Promise<boolean> {
const controller = new AbortController()
const timeout = setTimeout(() => controller.abort(), PREFLIGHT_TIMEOUT_MS)
try {
await fetch(baseUrl, { signal: controller.signal })
return true
} catch {
return false
} finally {
clearTimeout(timeout)
}
}
async function resolvePagePaths(config: SuiteConfig): Promise<string[] | null> {
const explicit = process.env[config.pagePathsEnv]?.trim()
if (explicit) {
return explicit
.split(/[\n,]/)
.map((path) => path.trim())
.filter(Boolean)
}
const baseRef = process.env[config.baseRefEnv]?.trim() || 'origin/master'
const repoRoot = repoRootFrom(config.root)
const changedFiles = collectChangedFiles(repoRoot, baseRef)
const result = await config.resolveScope({ changedFiles, repoRoot })
if (result.skip) {
console.error(
`No in-scope ${config.label} pages changed vs ${baseRef} (including working tree). Skipping Playwright.`
)
return null
}
console.error(
`Resolved ${result.pages.length} ${config.label} page(s) from changes vs ${baseRef}:`
)
for (const page of result.pages) {
console.error(` ${page}`)
}
return result.pages
}
export async function runSuite(config: SuiteConfig): Promise<void> {
const rawArgs = process.argv.slice(2)
const runAll = rawArgs.includes('--all')
const playwrightArgs = rawArgs.filter((arg) => arg !== '--all' && arg !== '--')
let pages: string[] | null
if (runAll && config.resolveAllPages) {
pages = await config.resolveAllPages(repoRootFrom(config.root))
console.error(`Resolved all ${pages.length} in-scope ${config.label} page(s).`)
} else {
pages = await resolvePagePaths(config)
}
if (pages === null) process.exit(0)
const hasMaxFailuresArg = playwrightArgs.some(
(arg) => arg === '-x' || arg.startsWith('--max-failures')
)
const finalArgs =
runAll && !hasMaxFailuresArg ? [...playwrightArgs, '--max-failures=0'] : playwrightArgs
const baseUrl = process.env.PLAYWRIGHT_BASE_URL?.trim() || config.defaultBaseUrl
if (!(await isBaseUrlReachable(baseUrl))) {
console.error(`No ${config.label} server responding at ${baseUrl}.`)
if (!process.env.PLAYWRIGHT_BASE_URL) {
console.error(`Start it with \`${config.devCommand}\`, or point at a deployed site.`)
} else {
console.error('Check that the URL is correct and reachable.')
}
process.exit(1)
}
const child = spawn('pnpm', ['exec', 'playwright', 'test', ...finalArgs], {
cwd: config.root,
env: { ...process.env, [config.pagePathsEnv]: pages.join(',') },
stdio: 'inherit',
shell: process.platform === 'win32',
})
child.on('exit', (code, signal) => {
if (signal) {
process.kill(process.pid, signal)
return
}
process.exit(code ?? 1)
})
}