lywsvip/supabase
1
0
Fork 0
mirror of https://github.com/supabase/supabase.git synced 2026-09-06 01:49:12 +08:00
Code Issues Packages Projects Releases Wiki Activity
38,329 Commits 1 Branch 60 Tags
create-pull-request/patch
Commit Graph

3 Commits

Author SHA1 Message Date
Miranda Limonczenko
6d3a4bcc48 feat(www) Add scaffolding for WWW E2E tests and CI check (#48861) 
Closes DOCS-1278

## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Feature. Adds E2E test scaffolding and a CI check for the marketing
site.

## What is the current behavior?

Closes [FE-4047](https://linear.app/supabase/issue/FE-4047).

The marketing site has no E2E coverage. Docs has a suite in `e2e/docs`,
but its
runner, git helpers and axe reporting are private to that package, so a
second
site cannot reuse them.

## What is the new behavior?

* **A www suite scoped to changed content.** Changed `.mdx` files in
`_blog`,
`_events`, `_customers` and `_alternatives` map to the URLs they render.
Pages
with `disable_page_build: true` are skipped because they 404 by design.
Capped
at 20 pages. Enforces `heading-order` and `page-has-heading-one`,
matching docs.
* **`e2e/shared` The docs site is also static with similar needs. This
folder shares the docs logic with www.
* **A CI check that is safe to mark required.** Path scoping lives in a
`Detect changed paths` step rather than a `paths:` trigger, so the check
  reports on every pull request instead of being skipped.
`waitForVercelDocsPreview.js` becomes `waitForVercelPreview.js`, shared
by both
  workflows.

## How the check behaves

The job always reports a check run, so it is safe to mark required. Path
scoping
happens in a step rather than a `paths:` trigger, which would leave
non-www pull
requests waiting on a check that never reports.

| Case | Behavior |
| --- | --- |
| Fork pull request adds new pages | Passes without testing. The Vercel
wait is gated on `head.repo.full_name == github.repository`, so forks
resolve no preview URL. The job emits a `::warning` and a job summary
containing a ready-to-run `gh workflow run www-e2e.yml` command with the
resolved page paths, so a maintainer can run it against the preview. |
| Vercel preview times out or fails | Passes without testing. The wait
step is `continue-on-error: true`, so a 900s timeout or a failed
deployment leaves the URL unset and the suite skips. Vercel's own
`Vercel – zone-www-dot-com` check already reports the failure. |
| Draft pull request | Job does not run at all, gated at the job level
on `pull_request.draft == false`. `ready_for_review` is in the trigger's
`types`, so marking it ready runs the check. |
| Another app changed, www untouched | Job runs and every step skips.
The `www` filter matches only the four content directories, `e2e/www`,
`e2e/shared`, the lockfile, and this workflow. |
| Only the harness changed | Passes without testing. Scope resolves to
zero pages, and the Vercel wait is additionally gated on `www_app`, so
it does not wait for a preview Vercel skipped. |
| No preview resolves, any reason | Skips rather than falling back to
production. Production does not serve pages the pull request adds, so
testing it would fail a valid change. |

### Not covered

Changes to `apps/www` components and routes do not trigger this check —
only the
four content directories do. A follow-up can check global components
such as the navigation and the footer.

## Manual testing

1. Start the site: `pnpm dev:www`
2. Run `pnpm e2e:www` with no www content changed. It should resolve
zero pages
   and skip Playwright, not fail.
3. Touch a post, then run `pnpm e2e:www` again:
`echo "" >> apps/www/_blog/2024-01-01-some-post.mdx`. The resolved
`/blog/...`
   path should be listed before Playwright starts.
4. Run against production with no local server:
`PLAYWRIGHT_BASE_URL=https://supabase.com
WWW_E2E_PAGE_PATHS=/blog/postgres-language-server pnpm e2e:www`
5. Point step 4 at a page with a known heading problem. The failure
should name
   the rule, the CSS selector and the markup.
6. Confirm docs still passes on the shared runner: `pnpm dev:docs`, then
   `pnpm e2e:docs`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added WWW end-to-end testing for affected content pages, including
accessibility checks.
* Added standard and full-site test commands, configurable preview
testing, and failure reports.
* Added shared utilities for page discovery, accessibility scanning, and
test execution.

* **Documentation**
* Documented WWW test setup, coverage, debugging, CI behavior, and
running checks against production or preview environments.

* **Improvements**
* Updated documentation test workflows to better identify affected
changes and handle preview environments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-08-11 22:06:51 +00:00
Miranda Limonczenko
777c02c205 test(docs): scan changed pages for WCAG 2.1 A/AA in warn mode (#48727) 
Closes DOCS-1233

## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Test coverage. The docs accessibility check now covers the full WCAG 2.1
A/AA rule set instead of two rules.

**Note:** This PR tests _only_ the main article of changed pages
(meaning, the content itself). A follow-up Linear issue is to address
scanning the pieces outside of that: header, navigation, and interactive
elements.

## What is the current behavior?

The `@a11y` test in `e2e/docs` runs two axe rules against each in-scope
page, `heading-order` and `page-has-heading-one`. Both already pass
everywhere, so the check only guards a result we have. Nothing else in
WCAG A/AA is checked.

## What is the new behavior?

The same test runs the full WCAG 2.1 A/AA rule set.

- **Existing debt does not block PRs.** Only the two heading rules fail.
Everything else reports.
- **The check stays fast.** It scans the article only and skips nine
rules that cannot fire there. Scan time drops from 2405ms to 981ms.
- **Findings belong to us.** Legacy mode excludes cross-origin frames.
YouTube embeds were counting against us, 11 of 15 violations on one
page.
- **A pass carries meaning.** A 404 reports as a load failure, not an
a11y bug. A page scanned before it hydrates warns instead of quietly
reporting clean.

## How the findings appear

The test is named `has no blocking accessibility violations`, so a
failure listed by CI is always something to fix. It is not named for the
full rule set, because a green check would then claim more than the
check verifies.

| | Rules | Where you see it |
| --- | --- | --- |
| Blocking | `heading-order`, `page-has-heading-one` | Test failure, so
the runner reports it on the PR |
| Reported | Everything else in WCAG A/AA | `::warning` annotation on
the run |

An annotation looks like this, on a run that still passes:

```
::warning title=Accessibility::/docs/guides/database/functions has 1 non-blocking accessibility finding(s): frame-title (4)
```

The full axe result for each page is attached to the report as
`axe-results.json`.

## Matching the Studio ratchet

This follows the ESLint ratchet in `apps/studio`. That pattern warns on
pre-existing debt rather than blocking on it, surfaces findings as
annotations rather than PR comments, and promotes a rule to an error
once its violations reach zero.

The mechanism here is `ENFORCED_RULES` in `utils/axe-helpers.ts`. The
two heading rules are on it because the heading-hierarchy work drove
them to zero site-wide.

The intent is to migrate rules into that list one at a time. Pick a
rule, fix its violations, then move it into `ENFORCED_RULES` so it
cannot come back. An exhaustive scan of the site groups the current
backlog by root cause to sequence that work, and two fixes cover 99.1%
of it.

Studio keeps per-file baseline counts, which this does not. A whole-rule
list is coarser, and it works here because docs violations reach zero
across the site rather than per file.

## Manual testing

Install the browser once, then run each step from the repo root. Every
command scans production, so you do not need a local docs server.

```bash
pnpm -C e2e/docs exec playwright install chromium
```

1. Confirm a reported finding does not fail the check.

   ```bash
DOCS_E2E_PAGE_PATHS=/docs/guides/database/functions
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
   ```

Expect `1 passed`, and the `::warning` annotation above in the output.

2. Confirm the scan finds that violation. Same page, now failing on
every rule.

   ```bash
A11Y_ENFORCE_ALL=1 DOCS_E2E_PAGE_PATHS=/docs/guides/database/functions
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
   ```

Expect `1 failed`, reporting `frame-title (serious, 4 node(s))`. Steps 1
and 2 together are the point of this PR.

3. Confirm the skipped rules stay skipped.

   ```bash
A11Y_ENFORCE_ALL=1
DOCS_E2E_PAGE_PATHS=/docs/guides/getting-started/quickstarts/nextjs
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
   ```

Expect `button-name (critical, 2 node(s))` and `label (critical, 2
node(s))`, and no `color-contrast`.

4. Confirm a page that does not load reports a load failure.

   ```bash
DOCS_E2E_PAGE_PATHS=/docs/guides/does-not-exist-xyz
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs:a11y
   ```

Expect `Expected a successful response for
/docs/guides/does-not-exist-xyz, got 404`, and no axe assertion.

5. Confirm the link checker still passes alongside the a11y test.

   ```bash
DOCS_E2E_PAGE_PATHS=/docs/guides/auth/passwords
PLAYWRIGHT_BASE_URL=https://supabase.com pnpm e2e:docs
   ```

   Expect `3 passed`.

## Known gaps

- `/docs/reference/*` is not scanned. Those routes render client-side
into tens of thousands of elements, where axe exceeds its timeout and
results depend on whether the scan caught the page mid-render.
- Shared chrome is outside the article scope, so nav, sidebar, footer,
menus, and drawers are not covered.
- axe catches roughly 30-40% of WCAG issues. Keyboard navigation, focus
management, and screen reader behavior still need manual testing.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-08-07 10:41:23 -07:00
Miranda Limonczenko
52cb1c2600 feat(docs) Dynamically E2E test all docs-owned content (#48320) 
Closes DOCS-1203

## Problem

The docs E2E workflow only ever tested one hardcoded page: the Next.js
quickstart. All other docs content had no E2E coverage.

## Solution

This PR expands the initial scaffolding to generalize the Next.js
quickstart tests, page runs and checks local links, to all pages
affecting Docs content:


- Add `resolveDocsScope` (`e2e/docs/utils/resolve-docs-scope.ts`) to map
changed guide and troubleshooting `.mdx` files to their `/docs/...` page
paths, and to expand changed `_partials` to every page that includes
them (including transitively, through partials nested inside other
partials). Federated guide sections (`graphql`,
`database/extensions/wrappers`, `ai/python`, `deployment/terraform`,
`deployment/ci`) and reference docs stay out of scope, and resolution is
capped at 20 pages to keep runtime bounded.
- Replace the single `quickstarts.spec.ts` test with a generic
`docs-pages.spec.ts` that loads whatever pages are resolved, asserting
each renders with an `<h1>` and that its docs-owned links resolve.
- Add `run-e2e-docs.ts` so `pnpm e2e:docs` resolves scope locally (from
commits since `origin/master`, plus staged/unstaged changes) and skips
Playwright entirely when nothing in scope changed.
- Update `.github/workflows/docs-e2e.yml` to widen the trigger paths to
all guides/troubleshooting/partials, resolve scope in a dedicated step,
skip the rest of the job when scope is empty, and accept a `page_paths`
input for manual `workflow_dispatch` runs.
- Rewrite `e2e/docs/README.md` to document the new scoping behavior, the
override envs (`DOCS_E2E_PAGE_PATHS`, `DOCS_E2E_BASE_REF`), and how CI
uses the suite.
- `pnpm e2e:docs:all` is also added to run tests on every page locally.
Good for scoping issues but should not be included in CI.

## Manual testing

Walk through the following steps to verify this works:

- [x] `pnpm e2e:docs` from repo root resolves the expected pages for a
local guide edit and can run against local dev
**Note:** Challenges with testing on local in part because of the long
lag for first page load. Recommendation to use a hosted URL is added to
docs.
- [x] Editing a shared `_partials` file resolves to every page that
includes it (including through nested partials)
- [x] `pnpm e2e:docs` exits cleanly with no Playwright run when no
in-scope files changed
- [x] `git diff --name-only ... | pnpm -C e2e/docs resolve-docs-scope`
prints the expected page list for a sample diff
- [x] Workflow run on a PR that only touches `e2e/docs`/workflow files
skips the Playwright steps
- [x] Manual `workflow_dispatch` run with `page_paths` set tests only
those pages
- [x] Run `pnpm e2e:docs:all` to run the suite on all docs content,
which takes awhile

## Next steps

After this PR merges, we have the scaffolding to add more fun tests like
a11y 😁

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added scoped Docs E2E runs that target eligible doc pages based on
changes, plus manual page-targeted runs and an “all eligible pages”
mode.
* Introduced `DOCS_E2E_PAGE_PATHS` (and updated base ref/base URL
behavior) to control which pages are tested.
* **Bug Fixes**
* Automatically skips Playwright setup when no relevant pages are in
scope; Playwright reporting now uploads only on failure.
* **Documentation**
* Updated the Docs E2E README with new run/CI behavior, troubleshooting
notes, and commands to inspect the resolved page list.
* **Tests**
* Added a Docs-owned pages E2E suite; removed the Next.js quickstart E2E
spec.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
 2026-07-28 00:04:58 +00:00