2 Commits

Author	SHA1	Message	Date
Pamela Chia	5db8a0e960	feat(studio): instrument sign-in attempts and failures (#49853) ... The /sign-in page emitted only a pageview on entry and the success-side `sign_in` event on exit: failed or abandoned attempts were invisible, so "never interacted" and "tried and failed silently" could not be told apart in the sign-in funnel. I added an unsampled `sign_in_submitted` event at every initiation point and classified failure capture via `dashboard_error_created` with a new `signin` origin. **Changed:** - **Submit attempts observable**: `sign_in_submitted` (method: `email`, provider id, `sso`, or partner) fires from the DOM submit handler on the password and SSO forms (so submits that fail client-side validation still count), and from the OAuth, custom-provider, and partner initiation handlers. - **Failures classified**: each sign-in error path feeds the existing funnel-error pipe with origin `signin` and a controlled reason slug (`invalid_credentials`, `email_not_confirmed`, `captcha_failed`, `sso_provider_not_found`, ...). GoTrue auth errors now classify via their numeric `status`, guarded so transport failures (`status: 0`) stay `network_error`. - **Attempt events survive the OAuth redirect**: the telemetry event POST sends with `keepalive` (scoped to `sign_in_submitted`, since keepalive requests share a per-page in-flight body quota), so a dispatched request is no longer aborted by the provider navigation; send rejections are caught centrally instead of surfacing as unhandled rejections. The fetch still dispatches after an async token lookup, so preview testing verifies the GitHub-path event actually lands on the wire. - **Captcha rejection is no longer silent**: a rejected hCaptcha challenge resolves the stuck loading toast with an error message, emits `captcha_challenge_failed` (distinct from `captcha_failed`, which stays reserved for the auth server rejecting a submitted token), reports to error monitoring, and resets the captcha widget (previously: unhandled promise rejection and a spinner that never resolved). - **Partner method validated**: the partner sign-in page resolves the URL-hash value against the provider registry and forwards the canonical provider id into `method` on both `sign_in_submitted` and `sign_in`; anything unregistered records as `unregistered_partner`, so a crafted link can't poison the breakdown on either event. **Note:** failure events stay on the shared 10% `dashboard_error_created` sampling rate (a per-origin carve-out would break cross-source volume comparability); the unsampled attempt event carries the tried-vs-never-interacted signal at full volume. ## To test Tested on Vercel preview (studio-staging, wire-level network capture + staging ingestion check): - [x] On `/sign-in`, submit a bogus email + password: expect a `POST */platform/telemetry/event` request with `action: sign_in_submitted`, `method: email` in the network tab, plus an error toast. Observed: 201, auth returned 400 as expected. - [x] Submit with an empty password: expect `sign_in_submitted` to still fire (validation failures count as attempts). Observed: event fired with 201 and no auth call followed. - [x] Click "Continue with GitHub": expect `sign_in_submitted` with `method: github` on the wire before the provider redirect. Observed: the POST completed (201) before the browser landed on github.com, so the keepalive path holds. - [x] Negative case: fresh page load with no interaction fires no `sign_in_submitted`. - [x] Ingestion: all fired events (methods `email`, `github`, plus organic `sso` submits from a real login on the same preview) arrived in the staging project with the expected properties. - [x] Re-ran the email and GitHub paths on the scoped-keepalive build (`129bf8d`): both `sign_in_submitted` POSTs returned 201 (the GitHub one completed despite the provider redirect), and both events ingested into the staging project with the expected `method`/`category` properties. ## Linear - GROWTH-1165 (no `fixes` keyword on purpose: the evidence checks run on prod data post-deploy, and the issue closes manually after they pass) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Improved sign-in protection with more reliable invisible CAPTCHA handling. * Added sign-in submission tracking across password, SSO, partner, custom OAuth, and external-provider flows. * Added detailed classification for authentication, validation, CAPTCHA, provider, and network errors. * **Bug Fixes** * Sign-in now stops safely and resets CAPTCHA when verification fails. * Improved error reporting for failed sign-in attempts, including redirects and OAuth flows. * Ensured sign-in telemetry is delivered reliably during OAuth redirects. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-02 19:02:37 +08:00
Pamela Chia	9858562b8b	fix(telemetry): dedupe funnel toast error events (#47802) ... ## Summary Since #47293, an API failure on a signup / org-creation / project-creation form emitted `dashboard_error_created` twice: `useTrackFunnelError` fired the origin-tagged event and the global `ToastErrorTracker` independently fired the legacy untagged `source:'toast'` event for the same toast, each behind its own 10% sampling draw. I verified the twin rate empirically at 8-11% of origin-tagged funnel toasts, exactly the floor for two independent 10% draws, meaning the twin co-fires for effectively every funnel error ([Hex thread](https://app.hex.tech/supabase/thread/019f3bc1-3a5c-7200-9122-8e3439bfbe8c)). Any consumer counting funnel errors without an `origin IS NOT NULL` filter saw ~2x inflation. The fix makes `ToastErrorTracker` the sole emitter of `source:'toast'` events, so the duplicate is unrepresentable rather than suppressed. Funnel call sites pass the id returned by `toast.error()` into `trackFunnelError`, which registers the funnel properties against that toast id instead of firing its own event – the tracker then emits a single `dashboard_error_created` enriched with `origin` / `errorCategory` / `errorReason` / `errorCode` for registered toasts, and the plain untagged event otherwise. The `'toast'` overload of `trackFunnelError` requires the toast id, so a missed pairing is a compile error rather than a silent double count. Registration is unconditional and there's only one sampling draw, so suppression can't lose a sampling race. `'form'`-sourced funnel events are unchanged. ## Changes - `lib/toast-errors.tsx`: toast-id → funnel-properties registry (`registerFunnelErrorToast`); `ToastErrorTracker` emits one (optionally enriched) event per error toast under a single 10% draw, deleting entries once consumed - `lib/telemetry/use-track-funnel-error.ts`: overloaded signature – `'toast'` requires the id returned by `toast.error()` (type-enforced), `'form'` keeps direct emission with its own sampling - Update the 7 funnel `toast.error` call sites in `NewOrgForm`, `SignUpForm`, and `pages/new/[slug]` to pass the toast id - Component tests for the tracker (previously uncovered), including an end-to-end test through `useTrackFunnelError` - Code hygiene (also flagged by CodeRabbit): all four `dashboard_error_created` emitters (toast, form, `AlertError`, `ErrorMatcher`) independently encoded the 10% draw – downstream analysis assumes a uniform sampling multiplier across sources, so one site drifting would silently skew comparisons. The rate and the draw now live in one place (`isDashboardErrorSampled()` in `lib/telemetry/error-sampling.ts`). No behavior change. - Mount `ToastErrorTracker` in the TanStack root (`routes/__root.tsx`), mirroring `pages/_app.tsx`. The TanStack tree mounted `Toaster` but never the tracker, so untagged toast error telemetry has never fired in that flavour – and with the tracker now the sole emitter, the missing mount would have silently dropped funnel toast events there too. Side effect once the TanStack flavour ships: untagged `source:'toast'` volume from it goes from zero to normal. ## Testing Component-tested (`apps/studio/lib/toast-errors.test.tsx`): - [x] Unregistered error toast fires exactly one untagged `dashboard_error_created {source:'toast'}` - [x] Registered funnel toast fires exactly one event, enriched with `origin`/`errorCategory`/`errorReason`/`errorCode` - [x] `useTrackFunnelError` with a toast id routes through the tracker as a single enriched event - [x] Non-error toasts ignored; the 10% sampling gate still applies Full Studio unit suite passes (392 files / 4371 tests), plus typecheck and lint. Also verified end-to-end in a local browser (TanStack flavour, sample rate temporarily forced to 1): a failed signup produced exactly one `dashboard_error_created` with `{source:'toast', origin:'signup', errorCategory:'api', errorReason:'email_already_registered', errorCode:403}` and no untagged twin (two independent trials); an unregistered error toast produced exactly one plain `{source:'toast'}`; a client-side validation failure produced exactly one `{source:'form', origin:'signup', errorCategory:'validation', errorReason:'email_invalid'}`; success toasts produced nothing. Post-deploy I'll re-run the twin-rate query from the Hex thread; the untagged-twin rate on funnel pages should decay to ~0 as stale bundles reload over 2-3 days. ## Notes - Origin-tagged funnel toast events now ride the tracker's single 10% draw instead of their own independent draw – statistically identical volume, but the event fires on the tracker's next effect rather than synchronously at the call site (irrelevant for PostHog) - Registration must happen in the same synchronous block as `toast.error()` (documented on the `TrackFunnelError` type) – all current call sites comply - The invalid Postgres version toast in `pages/new/[slug].tsx` (~line 416) needs no special-casing: unregistered toasts keep the plain untagged event, so its telemetry is preserved - Heads-up for `dashboard_error_created` consumers: overall untagged `source:'toast'` volume will dip slightly after this deploys, since funnel-page twins disappear. A volume monitor seeing that drop is this fix landing, not a tracking regression (same class as the intended GROWTH-893 sampling-unification drop). ## Linear - fixes GROWTH-965 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Enhanced error telemetry for organization creation, sign-up, payment, and project-creation flows by associating failures with toast identifiers and enriched funnel context. * Standardized dashboard error sampling logic across error handling components for consistency. * **Tests** * Added comprehensive test coverage for toast error tracking, including funnel registration, deduplication, filtering, and sampling behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>	2026-07-10 17:35:27 +08:00