supabase

lywsvip/supabase

Fork 0

mirror of https://github.com/supabase/supabase.git synced 2026-09-06 18:11:51 +08:00

Commit Graph

Author	SHA1	Message	Date
Pamela Chia	5db8a0e960	feat(studio): instrument sign-in attempts and failures (#49853 ) The /sign-in page emitted only a pageview on entry and the success-side `sign_in` event on exit: failed or abandoned attempts were invisible, so "never interacted" and "tried and failed silently" could not be told apart in the sign-in funnel. I added an unsampled `sign_in_submitted` event at every initiation point and classified failure capture via `dashboard_error_created` with a new `signin` origin. Changed: - Submit attempts observable: `sign_in_submitted` (method: `email`, provider id, `sso`, or partner) fires from the DOM submit handler on the password and SSO forms (so submits that fail client-side validation still count), and from the OAuth, custom-provider, and partner initiation handlers. - Failures classified: each sign-in error path feeds the existing funnel-error pipe with origin `signin` and a controlled reason slug (`invalid_credentials`, `email_not_confirmed`, `captcha_failed`, `sso_provider_not_found`, ...). GoTrue auth errors now classify via their numeric `status`, guarded so transport failures (`status: 0`) stay `network_error`. - Attempt events survive the OAuth redirect: the telemetry event POST sends with `keepalive` (scoped to `sign_in_submitted`, since keepalive requests share a per-page in-flight body quota), so a dispatched request is no longer aborted by the provider navigation; send rejections are caught centrally instead of surfacing as unhandled rejections. The fetch still dispatches after an async token lookup, so preview testing verifies the GitHub-path event actually lands on the wire. - Captcha rejection is no longer silent: a rejected hCaptcha challenge resolves the stuck loading toast with an error message, emits `captcha_challenge_failed` (distinct from `captcha_failed`, which stays reserved for the auth server rejecting a submitted token), reports to error monitoring, and resets the captcha widget (previously: unhandled promise rejection and a spinner that never resolved). - Partner method validated: the partner sign-in page resolves the URL-hash value against the provider registry and forwards the canonical provider id into `method` on both `sign_in_submitted` and `sign_in`; anything unregistered records as `unregistered_partner`, so a crafted link can't poison the breakdown on either event. Note: failure events stay on the shared 10% `dashboard_error_created` sampling rate (a per-origin carve-out would break cross-source volume comparability); the unsampled attempt event carries the tried-vs-never-interacted signal at full volume. ## To test Tested on Vercel preview (studio-staging, wire-level network capture + staging ingestion check): - [x] On `/sign-in`, submit a bogus email + password: expect a `POST /platform/telemetry/event` request with `action: sign_in_submitted`, `method: email` in the network tab, plus an error toast. Observed: 201, auth returned 400 as expected. - [x] Submit with an empty password: expect `sign_in_submitted` to still fire (validation failures count as attempts). Observed: event fired with 201 and no auth call followed. - [x] Click "Continue with GitHub": expect `sign_in_submitted` with `method: github` on the wire before the provider redirect. Observed: the POST completed (201) before the browser landed on github.com, so the keepalive path holds. - [x] Negative case: fresh page load with no interaction fires no `sign_in_submitted`. - [x] Ingestion: all fired events (methods `email`, `github`, plus organic `sso` submits from a real login on the same preview) arrived in the staging project with the expected properties. - [x] Re-ran the email and GitHub paths on the scoped-keepalive build (`129bf8d`): both `sign_in_submitted` POSTs returned 201 (the GitHub one completed despite the provider redirect), and both events ingested into the staging project with the expected `method`/`category` properties. ## Linear - GROWTH-1165 (no `fixes` keyword on purpose: the evidence checks run on prod data post-deploy, and the issue closes manually after they pass) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit New Features * Improved sign-in protection with more reliable invisible CAPTCHA handling. * Added sign-in submission tracking across password, SSO, partner, custom OAuth, and external-provider flows. * Added detailed classification for authentication, validation, CAPTCHA, provider, and network errors. * Bug Fixes * Sign-in now stops safely and resets CAPTCHA when verification fails. * Improved error reporting for failed sign-in attempts, including redirects and OAuth flows. * Ensured sign-in telemetry is delivered reliably during OAuth redirects. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-02 19:02:37 +08:00
Pamela Chia	98cfe3307e	feat(telemetry): fix creation-funnel tracking gaps (#47386 ) ## Summary The creation-funnel instrumentation that shipped Jun 25 (#47291, #47293) had real gaps, surfaced by the weekly telemetry audit and confirmed against production PostHog data before I touched code. The two automated reports also contradicted each other on `errorReason`; I checked production (every value is a controlled slug) and the emit path (only `useTrackFunnelError` sets it, and it only accepts classified slugs), so I left the type as-is rather than add a cross-package abstraction for a risk that cannot occur today. ## Changes - Classify HTTP 401/403/404 API errors as `unauthorized` / `forbidden` / `not_found` instead of the catch-all `other`. In production the `org_creation` `other` bucket was ~96% 401s (~1,300 real over 4 days), invisible in reason breakdowns. The status-code fallback runs after the message-pattern match, so specific reasons still win and it only rescues errors that would otherwise be `other`. - Add a single `tier` property (`tier_free` / `tier_pro` / `tier_payg` / `tier_team`) to `organization_creation_completed`, which previously carried no properties. One canonical billing slug (matching `SubscriptionTier`) instead of two overlapping plan/tier fields, so the org-creation funnel segments cleanly by tier and joins against subscription data. `tier_payg` is uncapped PRO. - Freeze the submitted tier at submit time (snapshot in `createOrg`) rather than reading live form state in the success callback, so the event records the tier that was actually created even if the user edits the form during the async payment flow. - Emit `project_creation_form_exposed` with `surface: 'vercel'` on the integration deploy-button project-creation page (the enum value existed but was never fired). Gated on the URL `slug` so the impression is captured as soon as the form renders, matching the sibling exposure hook on that page. I also checked the confirm-modal error path flagged in the insights post: it already classifies via the shared `useProjectCreateMutation.onError`, so adding instrumentation there would double-count. No change made. ## Testing These are analytics events with no UI change, so correctness is in what lands in PostHog. Post-deploy validation I will run against production (project 34344): - `dashboard_error_created` where `origin='org_creation'` and `errorReason='other'` drops ~96%, with `unauthorized` / `not_found` appearing. - `organization_creation_completed.tier` populated on 100% of new events with one of the four tier slugs. - `project_creation_form_exposed` with `surface='vercel'` goes from 0 to greater than 0. ## Linear - fixes GROWTH-948 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * New Features * Added telemetry for organization creation completion that includes the selected billing tier. * Added one-time telemetry when the Vercel project creation form is exposed. * Bug Fixes * Improved API error classification to more accurately distinguish unauthorized, forbidden, and not found responses. * Documentation * Updated telemetry event definitions to require tier metadata for organization creation events. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-06-30 23:11:53 +08:00
Pamela Chia	2d0bcd4714	feat(telemetry): classify funnel creation errors (#47293 ) ## Summary The KPI-3 friction dashboard needs to know why users hit errors on the signup, project-creation, and org-creation funnels, not just that they did. The existing `dashboard_error_created` event already fires for these paths (10% sampled, with `$pathname`), but carries no reason: ~98.5% of events have no `errorType` and no property carries an error message. This adds PII-safe classification computed client-side from a controlled vocabulary, so raw error text never leaves the browser. Validation errors (previously invisible, since they are inline form errors that never raise a toast) are now captured on invalid submit. ## Changes - Extend `dashboard_error_created` with `origin`, `errorCategory`, `errorReason`, `errorCode`, and a `form` source value - Add a pure, unit-tested classifier (`funnel-errors.ts`) and a 10%-sampled tracking hook (`use-track-funnel-error.ts`); the classifier maps errors to stable slugs and emits only slugs + HTTP status, never raw message text - Classify signup errors (API failures + validation) in `SignUpForm` - Classify project-creation errors (API failures, OrioleDB guard, validation) in the new-project wizard - Classify org-creation errors (API failures, payment/card declines, confirm-subscription, validation) in `NewOrgForm` ## Testing 13 unit tests cover every classifier branch (validation / api / network / payment, status-code handling, message-pattern matching, and fallbacks). To verify on the Vercel preview (events are 10% sampled; set the sample rate to 1 locally to observe each fire): - Signup with a weak but non-empty password: `origin=signup, source=form, errorCategory=validation, errorReason=password_invalid` - Signup with an already-registered email: `origin=signup, source=toast, errorCategory=api, errorReason=email_already_registered` - New project with an empty name: `origin=project_creation, source=form, errorReason=project_name_invalid` - New org with an empty name: `origin=org_creation, source=form, errorReason=org_name_missing` - New org with a declined test card: `origin=org_creation, errorCategory=payment` PII: raw `error.message` is never sent; only controlled slugs and HTTP status. Dashboard consumers must filter `origin IS NOT NULL` so these do not collide with the generic toast events the global tracker still emits. ## Linear - fixes FE-3691 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * New Features * Added improved, categorized telemetry for signup, project creation, and organization creation errors, including payment, subscription-change, and validation failures. * Extended dashboard error events with optional structured diagnostics (origin, category, reason, and optional error code) and support for form-origin reporting. * Bug Fixes * Improved project-creation handling to record a validation telemetry event when an Oriole image is unavailable. * Ensured payment-related and subscription-change failures are captured consistently alongside existing user toasts. * Tests * Added unit tests covering API/network/validation/Stripe error classification and reason mapping. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-06-25 20:56:23 +08:00

Author

SHA1

Message

Date

Pamela Chia

5db8a0e960

feat(studio): instrument sign-in attempts and failures (#49853 )

The /sign-in page emitted only a pageview on entry and the success-side
`sign_in` event on exit: failed or abandoned attempts were invisible, so
"never interacted" and "tried and failed silently" could not be told
apart in the sign-in funnel. I added an unsampled `sign_in_submitted`
event at every initiation point and classified failure capture via
`dashboard_error_created` with a new `signin` origin.

**Changed:**
- **Submit attempts observable**: `sign_in_submitted` (method: `email`,
provider id, `sso`, or partner) fires from the DOM submit handler on the
password and SSO forms (so submits that fail client-side validation
still count), and from the OAuth, custom-provider, and partner
initiation handlers.
- **Failures classified**: each sign-in error path feeds the existing
funnel-error pipe with origin `signin` and a controlled reason slug
(`invalid_credentials`, `email_not_confirmed`, `captcha_failed`,
`sso_provider_not_found`, ...). GoTrue auth errors now classify via
their numeric `status`, guarded so transport failures (`status: 0`) stay
`network_error`.
- **Attempt events survive the OAuth redirect**: the telemetry event
POST sends with `keepalive` (scoped to `sign_in_submitted`, since
keepalive requests share a per-page in-flight body quota), so a
dispatched request is no longer aborted by the provider navigation; send
rejections are caught centrally instead of surfacing as unhandled
rejections. The fetch still dispatches after an async token lookup, so
preview testing verifies the GitHub-path event actually lands on the
wire.
- **Captcha rejection is no longer silent**: a rejected hCaptcha
challenge resolves the stuck loading toast with an error message, emits
`captcha_challenge_failed` (distinct from `captcha_failed`, which stays
reserved for the auth server rejecting a submitted token), reports to
error monitoring, and resets the captcha widget (previously: unhandled
promise rejection and a spinner that never resolved).
- **Partner method validated**: the partner sign-in page resolves the
URL-hash value against the provider registry and forwards the canonical
provider id into `method` on both `sign_in_submitted` and `sign_in`;
anything unregistered records as `unregistered_partner`, so a crafted
link can't poison the breakdown on either event.

**Note:** failure events stay on the shared 10%
`dashboard_error_created` sampling rate (a per-origin carve-out would
break cross-source volume comparability); the unsampled attempt event
carries the tried-vs-never-interacted signal at full volume.

## To test

Tested on Vercel preview (studio-staging, wire-level network capture +
staging ingestion check):
- [x] On `/sign-in`, submit a bogus email + password: expect a `POST
*/platform/telemetry/event` request with `action: sign_in_submitted`,
`method: email` in the network tab, plus an error toast. Observed: 201,
auth returned 400 as expected.
- [x] Submit with an empty password: expect `sign_in_submitted` to still
fire (validation failures count as attempts). Observed: event fired with
201 and no auth call followed.
- [x] Click "Continue with GitHub": expect `sign_in_submitted` with
`method: github` on the wire before the provider redirect. Observed: the
POST completed (201) before the browser landed on github.com, so the
keepalive path holds.
- [x] Negative case: fresh page load with no interaction fires no
`sign_in_submitted`.
- [x] Ingestion: all fired events (methods `email`, `github`, plus
organic `sso` submits from a real login on the same preview) arrived in
the staging project with the expected properties.
- [x] Re-ran the email and GitHub paths on the scoped-keepalive build
(`129bf8d`): both `sign_in_submitted` POSTs returned 201 (the GitHub one
completed despite the provider redirect), and both events ingested into
the staging project with the expected `method`/`category` properties.

## Linear
- GROWTH-1165 (no `fixes` keyword on purpose: the evidence checks run on
prod data post-deploy, and the issue closes manually after they pass)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Improved sign-in protection with more reliable invisible CAPTCHA
handling.
* Added sign-in submission tracking across password, SSO, partner,
custom OAuth, and external-provider flows.
* Added detailed classification for authentication, validation, CAPTCHA,
provider, and network errors.

* **Bug Fixes**
  * Sign-in now stops safely and resets CAPTCHA when verification fails.
* Improved error reporting for failed sign-in attempts, including
redirects and OAuth flows.
* Ensured sign-in telemetry is delivered reliably during OAuth
redirects.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-09-02 19:02:37 +08:00

Pamela Chia

98cfe3307e

feat(telemetry): fix creation-funnel tracking gaps (#47386 )

## Summary

The creation-funnel instrumentation that shipped Jun 25 (#47291, #47293)
had real gaps, surfaced by the weekly telemetry audit and confirmed
against production PostHog data before I touched code. The two automated
reports also contradicted each other on `errorReason`; I checked
production (every value is a controlled slug) and the emit path (only
`useTrackFunnelError` sets it, and it only accepts classified slugs), so
I left the type as-is rather than add a cross-package abstraction for a
risk that cannot occur today.

## Changes

- Classify HTTP 401/403/404 API errors as `unauthorized` / `forbidden` /
`not_found` instead of the catch-all `other`. In production the
`org_creation` `other` bucket was ~96% 401s (~1,300 real over 4 days),
invisible in reason breakdowns. The status-code fallback runs after the
message-pattern match, so specific reasons still win and it only rescues
errors that would otherwise be `other`.
- Add a single `tier` property (`tier_free` / `tier_pro` / `tier_payg` /
`tier_team`) to `organization_creation_completed`, which previously
carried no properties. One canonical billing slug (matching
`SubscriptionTier`) instead of two overlapping plan/tier fields, so the
org-creation funnel segments cleanly by tier and joins against
subscription data. `tier_payg` is uncapped PRO.
- Freeze the submitted tier at submit time (snapshot in `createOrg`)
rather than reading live form state in the success callback, so the
event records the tier that was actually created even if the user edits
the form during the async payment flow.
- Emit `project_creation_form_exposed` with `surface: 'vercel'` on the
integration deploy-button project-creation page (the enum value existed
but was never fired). Gated on the URL `slug` so the impression is
captured as soon as the form renders, matching the sibling exposure hook
on that page.

I also checked the confirm-modal error path flagged in the insights
post: it already classifies via the shared
`useProjectCreateMutation.onError`, so adding instrumentation there
would double-count. No change made.

## Testing

These are analytics events with no UI change, so correctness is in what
lands in PostHog. Post-deploy validation I will run against production
(project 34344):

- `dashboard_error_created` where `origin='org_creation'` and
`errorReason='other'` drops ~96%, with `unauthorized` / `not_found`
appearing.
- `organization_creation_completed.tier` populated on 100% of new events
with one of the four tier slugs.
- `project_creation_form_exposed` with `surface='vercel'` goes from 0 to
greater than 0.

## Linear

- fixes GROWTH-948


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added telemetry for organization creation completion that includes the
selected billing tier.
* Added one-time telemetry when the Vercel project creation form is
exposed.
* **Bug Fixes**
* Improved API error classification to more accurately distinguish
unauthorized, forbidden, and not found responses.
* **Documentation**
* Updated telemetry event definitions to require tier metadata for
organization creation events.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-06-30 23:11:53 +08:00

Pamela Chia

2d0bcd4714

feat(telemetry): classify funnel creation errors (#47293 )

## Summary

The KPI-3 friction dashboard needs to know *why* users hit errors on the
signup, project-creation, and org-creation funnels, not just that they
did. The existing `dashboard_error_created` event already fires for
these paths (10% sampled, with `$pathname`), but carries no reason:
~98.5% of events have no `errorType` and no property carries an error
message. This adds PII-safe classification computed client-side from a
controlled vocabulary, so raw error text never leaves the browser.
Validation errors (previously invisible, since they are inline form
errors that never raise a toast) are now captured on invalid submit.

## Changes

- Extend `dashboard_error_created` with `origin`, `errorCategory`,
`errorReason`, `errorCode`, and a `form` source value
- Add a pure, unit-tested classifier (`funnel-errors.ts`) and a
10%-sampled tracking hook (`use-track-funnel-error.ts`); the classifier
maps errors to stable slugs and emits only slugs + HTTP status, never
raw message text
- Classify signup errors (API failures + validation) in `SignUpForm`
- Classify project-creation errors (API failures, OrioleDB guard,
validation) in the new-project wizard
- Classify org-creation errors (API failures, payment/card declines,
confirm-subscription, validation) in `NewOrgForm`

## Testing

13 unit tests cover every classifier branch (validation / api / network
/ payment, status-code handling, message-pattern matching, and
fallbacks).

To verify on the Vercel preview (events are 10% sampled; set the sample
rate to 1 locally to observe each fire):
- Signup with a weak but non-empty password: `origin=signup,
source=form, errorCategory=validation, errorReason=password_invalid`
- Signup with an already-registered email: `origin=signup, source=toast,
errorCategory=api, errorReason=email_already_registered`
- New project with an empty name: `origin=project_creation, source=form,
errorReason=project_name_invalid`
- New org with an empty name: `origin=org_creation, source=form,
errorReason=org_name_missing`
- New org with a declined test card: `origin=org_creation,
errorCategory=payment`

PII: raw `error.message` is never sent; only controlled slugs and HTTP
status. Dashboard consumers must filter `origin IS NOT NULL` so these do
not collide with the generic toast events the global tracker still
emits.

## Linear

- fixes FE-3691


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added improved, categorized telemetry for signup, project creation,
and organization creation errors, including payment,
subscription-change, and validation failures.
* Extended dashboard error events with optional structured diagnostics
(origin, category, reason, and optional error code) and support for
form-origin reporting.

* **Bug Fixes**
* Improved project-creation handling to record a validation telemetry
event when an Oriole image is unavailable.
* Ensured payment-related and subscription-change failures are captured
consistently alongside existing user toasts.

* **Tests**
* Added unit tests covering API/network/validation/Stripe error
classification and reason mapping.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2026-06-25 20:56:23 +08:00

3 Commits