Files
supabase/apps/studio/lib/telemetry/funnel-errors.test.ts
Pamela Chia 5db8a0e960 feat(studio): instrument sign-in attempts and failures (#49853)
The /sign-in page emitted only a pageview on entry and the success-side
`sign_in` event on exit: failed or abandoned attempts were invisible, so
"never interacted" and "tried and failed silently" could not be told
apart in the sign-in funnel. I added an unsampled `sign_in_submitted`
event at every initiation point and classified failure capture via
`dashboard_error_created` with a new `signin` origin.

**Changed:**
- **Submit attempts observable**: `sign_in_submitted` (method: `email`,
provider id, `sso`, or partner) fires from the DOM submit handler on the
password and SSO forms (so submits that fail client-side validation
still count), and from the OAuth, custom-provider, and partner
initiation handlers.
- **Failures classified**: each sign-in error path feeds the existing
funnel-error pipe with origin `signin` and a controlled reason slug
(`invalid_credentials`, `email_not_confirmed`, `captcha_failed`,
`sso_provider_not_found`, ...). GoTrue auth errors now classify via
their numeric `status`, guarded so transport failures (`status: 0`) stay
`network_error`.
- **Attempt events survive the OAuth redirect**: the telemetry event
POST sends with `keepalive` (scoped to `sign_in_submitted`, since
keepalive requests share a per-page in-flight body quota), so a
dispatched request is no longer aborted by the provider navigation; send
rejections are caught centrally instead of surfacing as unhandled
rejections. The fetch still dispatches after an async token lookup, so
preview testing verifies the GitHub-path event actually lands on the
wire.
- **Captcha rejection is no longer silent**: a rejected hCaptcha
challenge resolves the stuck loading toast with an error message, emits
`captcha_challenge_failed` (distinct from `captcha_failed`, which stays
reserved for the auth server rejecting a submitted token), reports to
error monitoring, and resets the captcha widget (previously: unhandled
promise rejection and a spinner that never resolved).
- **Partner method validated**: the partner sign-in page resolves the
URL-hash value against the provider registry and forwards the canonical
provider id into `method` on both `sign_in_submitted` and `sign_in`;
anything unregistered records as `unregistered_partner`, so a crafted
link can't poison the breakdown on either event.

**Note:** failure events stay on the shared 10%
`dashboard_error_created` sampling rate (a per-origin carve-out would
break cross-source volume comparability); the unsampled attempt event
carries the tried-vs-never-interacted signal at full volume.

## To test

Tested on Vercel preview (studio-staging, wire-level network capture +
staging ingestion check):
- [x] On `/sign-in`, submit a bogus email + password: expect a `POST
*/platform/telemetry/event` request with `action: sign_in_submitted`,
`method: email` in the network tab, plus an error toast. Observed: 201,
auth returned 400 as expected.
- [x] Submit with an empty password: expect `sign_in_submitted` to still
fire (validation failures count as attempts). Observed: event fired with
201 and no auth call followed.
- [x] Click "Continue with GitHub": expect `sign_in_submitted` with
`method: github` on the wire before the provider redirect. Observed: the
POST completed (201) before the browser landed on github.com, so the
keepalive path holds.
- [x] Negative case: fresh page load with no interaction fires no
`sign_in_submitted`.
- [x] Ingestion: all fired events (methods `email`, `github`, plus
organic `sso` submits from a real login on the same preview) arrived in
the staging project with the expected properties.
- [x] Re-ran the email and GitHub paths on the scoped-keepalive build
(`129bf8d`): both `sign_in_submitted` POSTs returned 201 (the GitHub one
completed despite the provider redirect), and both events ingested into
the staging project with the expected `method`/`category` properties.

## Linear
- GROWTH-1165 (no `fixes` keyword on purpose: the evidence checks run on
prod data post-deploy, and the issue closes manually after they pass)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Improved sign-in protection with more reliable invisible CAPTCHA
handling.
* Added sign-in submission tracking across password, SSO, partner,
custom OAuth, and external-provider flows.
* Added detailed classification for authentication, validation, CAPTCHA,
provider, and network errors.

* **Bug Fixes**
  * Sign-in now stops safely and resets CAPTCHA when verification fails.
* Improved error reporting for failed sign-in attempts, including
redirects and OAuth flows.
* Ensured sign-in telemetry is delivered reliably during OAuth
redirects.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-02 19:02:37 +08:00

202 lines
6.5 KiB
TypeScript

import type { FieldErrors } from 'react-hook-form'
import { describe, expect, it } from 'vitest'
import { classifyApiError, classifyStripeError, classifyValidationError } from './funnel-errors'
describe('classifyApiError', () => {
it('classifies connection timeout as network', () => {
expect(classifyApiError('signup', { errorType: 'connection-timeout' })).toEqual({
errorCategory: 'network',
errorReason: 'connection_timeout',
})
})
it('classifies a missing status code as network_error', () => {
expect(classifyApiError('project_creation', { message: 'Failed to fetch' })).toEqual({
errorCategory: 'network',
errorReason: 'network_error',
})
})
it('classifies 429 as rate_limited regardless of message', () => {
expect(classifyApiError('signup', { code: 429, message: 'whatever' })).toEqual({
errorCategory: 'api',
errorReason: 'rate_limited',
errorCode: 429,
})
})
it('classifies 5xx as server_error', () => {
expect(classifyApiError('org_creation', { code: 500, message: 'boom' })).toEqual({
errorCategory: 'api',
errorReason: 'server_error',
errorCode: 500,
})
})
it('matches a known 4xx signup message to a reason slug', () => {
expect(classifyApiError('signup', { code: 400, message: 'User already registered' })).toEqual({
errorCategory: 'api',
errorReason: 'email_already_registered',
errorCode: 400,
})
})
it('matches a known 4xx project message to a reason slug', () => {
expect(
classifyApiError('project_creation', {
code: 403,
message: 'Your organization can only have 2 projects',
})
).toEqual({ errorCategory: 'api', errorReason: 'project_limit_reached', errorCode: 403 })
})
it('falls back to other for an unmapped 4xx message', () => {
expect(classifyApiError('signup', { code: 400, message: 'totally novel error' })).toEqual({
errorCategory: 'api',
errorReason: 'other',
errorCode: 400,
})
})
describe('signin', () => {
it('reads a GoTrue AuthError status as the code', () => {
expect(
classifyApiError('signin', { status: 400, message: 'Invalid login credentials' })
).toEqual({
errorCategory: 'api',
errorReason: 'invalid_credentials',
errorCode: 400,
})
})
it('classifies an unconfirmed email', () => {
expect(classifyApiError('signin', { status: 400, message: 'Email not confirmed' })).toEqual({
errorCategory: 'api',
errorReason: 'email_not_confirmed',
errorCode: 400,
})
})
it('classifies 429 via status as rate_limited', () => {
expect(classifyApiError('signin', { status: 429, message: 'Rate limit exceeded' })).toEqual({
errorCategory: 'api',
errorReason: 'rate_limited',
errorCode: 429,
})
})
it('classifies a captcha failure', () => {
expect(
classifyApiError('signin', { status: 400, message: 'captcha verification process failed' })
).toEqual({ errorCategory: 'api', errorReason: 'captcha_failed', errorCode: 400 })
})
it('matches the SSO pattern before the 404 status map', () => {
expect(
classifyApiError('signin', {
status: 404,
message: 'No SSO provider assigned for this domain',
})
).toEqual({ errorCategory: 'api', errorReason: 'sso_provider_not_found', errorCode: 404 })
})
it('classifies a redirect allow-list rejection', () => {
expect(classifyApiError('signin', { status: 400, message: 'Invalid redirect URL' })).toEqual({
errorCategory: 'api',
errorReason: 'redirect_not_allowed',
errorCode: 400,
})
})
it('classifies a disabled provider', () => {
expect(
classifyApiError('signin', {
status: 400,
message: 'Unsupported provider: provider is not enabled',
})
).toEqual({ errorCategory: 'api', errorReason: 'provider_not_enabled', errorCode: 400 })
})
it('classifies a GoTrue transport failure (status 0) as network_error, never api/other', () => {
expect(
classifyApiError('signin', {
name: 'AuthRetryableFetchError',
status: 0,
message: 'Failed to fetch',
})
).toEqual({ errorCategory: 'network', errorReason: 'network_error' })
})
it('classifies a retryable 5xx via status as server_error', () => {
expect(classifyApiError('signin', { status: 503, message: 'Service unavailable' })).toEqual({
errorCategory: 'api',
errorReason: 'server_error',
errorCode: 503,
})
})
})
})
describe('classifyValidationError', () => {
it('maps a signup password error to password_invalid', () => {
expect(
classifyValidationError('signup', { password: { type: 'too_small' } } as FieldErrors)
).toEqual({
errorCategory: 'validation',
errorReason: 'password_invalid',
})
})
it('respects field priority (email before password)', () => {
expect(
classifyValidationError('signup', {
email: { type: 'invalid' },
password: { type: 'too_small' },
} as FieldErrors)
).toEqual({ errorCategory: 'validation', errorReason: 'email_invalid' })
})
it('maps a signin password error to password_invalid', () => {
expect(
classifyValidationError('signin', { password: { type: 'too_small' } } as FieldErrors)
).toEqual({
errorCategory: 'validation',
errorReason: 'password_invalid',
})
})
it('maps an org name error to org_name_missing', () => {
expect(
classifyValidationError('org_creation', { name: { type: 'too_small' } } as FieldErrors)
).toEqual({
errorCategory: 'validation',
errorReason: 'org_name_missing',
})
})
it('falls back to other for an unmapped field', () => {
expect(
classifyValidationError('project_creation', { somethingNew: { type: 'x' } } as FieldErrors)
).toEqual({ errorCategory: 'validation', errorReason: 'other' })
})
})
describe('classifyStripeError', () => {
it('maps a decline_code to a card reason slug', () => {
expect(
classifyStripeError({ code: 'card_declined', decline_code: 'insufficient_funds' })
).toEqual({
errorCategory: 'payment',
errorReason: 'card_insufficient_funds',
})
})
it('falls back to payment_failed for an unknown code', () => {
expect(classifyStripeError({ code: 'mystery' })).toEqual({
errorCategory: 'payment',
errorReason: 'payment_failed',
})
})
})