mirror of
https://github.com/supabase/supabase.git
synced 2026-09-08 10:59:38 +08:00
d45e0cd3d5
Supersedes #48725, which GitHub closed when its head branch was renamed. Same commits, same diff. Fixes [DOCS-1270](https://linear.app/supabase/issue/DOCS-1270/fail-the-e2e-pipeline-if-the-docs-preview-never-loads). `Docs E2E` is a required check on `master`, so anything that turns it red blocks a merge. It had three ways of going red that had nothing to do with whether the author's docs were correct. ## Problem **1. Every troubleshooting page could fail, with nothing actionable.** Troubleshooting entries were selected by `article.prose`. That class is not unique — `apps/docs/app/not-found.tsx` renders `<article className="prose …">` too — and nothing guaranteed it matched the entry's article at all. When it missed, the link test failed with `Page article should be present` and the a11y test failed inside axe with `No elements found for include in page Context` plus a stack trace. Neither tells the author what to do. This is what DOCS-1270 actually was. The ticket describes tests running "against a preview build that was never created", but the [failing run](https://github.com/supabase/supabase/actions/runs/30949924515/job/92132543658) for #48719 shows the preview resolved fine and `response.ok()` passed — it broke at the article assertion. **Blocked:** anyone adding or editing a troubleshooting entry. **2. Fork pull requests failed for being forks.** Fork runs get no `VERCEL_TOKEN`, so no preview URL resolves, and the base-URL step fell back to `https://supabase.com`. The page paths under test can include pages the pull request *adds*, which do not exist on production, so they 404. **Blocked:** every external contributor adding a docs page, unconditionally, with no action available to them. **3. A Vercel problem failed the docs check.** `waitForVercelDocsPreview.js` throws when Vercel reports a failed deployment, omits a `target_url`, or does not post a status within 900s. The step had no `continue-on-error`, so any of those turned `Docs E2E` red. **Blocked:** any author whose pull request coincided with a Vercel incident. This is live right now — two Vercel checks on this very pull request are failing with "unable to fetch required git information", a git-integration auth error that happens before any build runs. ## Solution **1. Select on a stable, purpose-named attribute.** Add `id="sb-docs-troubleshooting-main-article"` on the troubleshooting article, mirroring `#sb-docs-guide-main-article` on guides, and select on that instead of the class. Per review feedback, a plain id doesn't say it's a test hook, so both articles also get `data-testid` with the same value — matching the convention `apps/studio` already uses with Playwright's `getByTestId` — and the e2e selectors target that attribute instead. Guides keep their `id` — `GuidesMdx.client.tsx` and `GuidesSidebar.tsx` both query it directly for the table of contents and the "copy article" fallback — and gain `data-testid` alongside it. **2 and 3. Resolve a preview or skip — never substitute production, never fail on Vercel.** The production fallback is gone. `continue-on-error: true` on the preview wait means a Vercel failure resolves no URL instead of failing the job, which lands in the same path as a fork: `should_test=false`, so Playwright is skipped and the check passes. Both cases emit a `::warning::` and a job summary with the exact `gh workflow run` command to test the preview by hand, and manual runs against a non-production base URL now send the protection bypass so that command actually works. Skipping does not let a broken preview through: `Vercel – docs` is itself a required check on `master`, so a genuine preview failure still blocks the merge — via the check that describes the real problem. ## Manual test **1. The selector matches the markup, and it needs this pull request's preview.** `data-testid` isn't deployed anywhere yet — not on production, not on any other branch — so this is the one claim in this PR that production cannot confirm. Verified directly against this branch's own Vercel preview: ```bash curl -s https://docs-git-docs-e2e-stop-false-blocks-supabase.vercel.app/docs/guides/database/overview \ | grep -o 'data-testid="[^"]*"' curl -s https://docs-git-docs-e2e-stop-false-blocks-supabase.vercel.app/docs/guides/troubleshooting/42501--permission-denied-for-table-httprequestqueue-KnozmQ \ | grep -o 'data-testid="[^"]*"' ``` Expect `data-testid="sb-docs-guide-main-article"` and `data-testid="sb-docs-troubleshooting-main-article"` respectively. Then run the suite against that same preview — expect all page/link/a11y checks to pass: ```bash PLAYWRIGHT_BASE_URL=https://docs-git-docs-e2e-stop-false-blocks-supabase.vercel.app \ DOCS_E2E_PAGE_PATHS=/docs/guides/database/overview,/docs/guides/troubleshooting/42501--permission-denied-for-table-httprequestqueue-KnozmQ \ pnpm -C e2e/docs exec playwright test --reporter=list ``` Running the same command with `PLAYWRIGHT_BASE_URL=https://supabase.com` fails both pages right now — expected until this merges, not a regression. Once merged, exercise it through the real pipeline: ```bash gh workflow run docs-e2e.yml --ref docs-e2e/stop-false-blocks \ -f base_url=<preview-url> \ -f page_paths=/docs/guides/troubleshooting/42501--permission-denied-for-table-httprequestqueue-KnozmQ ``` **2. No preview means skip, not a run against production.** Exercise the base-URL step's three paths from the repository root: ```bash export GITHUB_OUTPUT=$(mktemp) GITHUB_STEP_SUMMARY=$(mktemp) PAGE_PATHS=/docs/guides/a script=$(python3 -c "import yaml;print([s for s in yaml.safe_load(open('.github/workflows/docs-e2e.yml'))['jobs']['e2e']['steps'] if s.get('name')=='Resolve base URL'][0]['run'])") for c in "workflow_dispatch|https://supabase.com|" "pull_request||https://docs-abc.vercel.app" "pull_request||"; do IFS='|' read -r ev url dep <<< "$c" : > "$GITHUB_OUTPUT" EVENT_NAME="$ev" BASE_URL_INPUT="${url:-https://supabase.com}" DEPLOYMENT_URL="$dep" bash -c "$script" >/dev/null 2>&1 echo "$ev deployment=[${dep:-none}] -> $(tr '\n' ' ' < "$GITHUB_OUTPUT")" done tail -4 "$GITHUB_STEP_SUMMARY" ``` Expected: ``` workflow_dispatch deployment=[none] -> url=https://supabase.com use_bypass=false should_test=true pull_request deployment=[https://docs-abc.vercel.app] -> url=https://docs-abc.vercel.app use_bypass=true should_test=true pull_request deployment=[none] -> url= use_bypass=false should_test=false ``` followed by a runnable `gh workflow run docs-e2e.yml` command in the job summary. The third line covers both the fork case and the Vercel-failure case: no base URL, no test, no block. **3. A Vercel failure no longer fails the job.** `continue-on-error: true` on the wait step is what routes a throw into that third line: ```bash python3 -c " import yaml s=[x for x in yaml.safe_load(open('.github/workflows/docs-e2e.yml'))['jobs']['e2e']['steps'] if x.get('name')=='Wait for Vercel docs preview'][0] print('continue-on-error:', s.get('continue-on-error')) for n in ('Install dependencies','Install Playwright Chromium','Run docs E2E'): print(n, '->', [x for x in yaml.safe_load(open('.github/workflows/docs-e2e.yml'))['jobs']['e2e']['steps'] if x.get('name')==n][0]['if']) " ``` Expect `continue-on-error: True` and all three run steps gated on `steps.base-url.outputs.should_test == 'true'`. **Note on this pull request's own check.** The scope resolver only maps `apps/docs/content/**` to pages, and this pull request changes none, so `Docs E2E` resolves zero pages and skips — which is correct, and why the dispatch above is the real test. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved documentation preview checks so unavailable or delayed previews no longer cause unnecessary workflow failures. * Added clearer handling for manual documentation checks and missing preview deployments. * **Tests** * Improved end-to-end documentation testing reliability across preview and production environments. * Added stable targeting for the troubleshooting article to reduce test failures caused by page structure changes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>