Files
supabase/apps/studio/lib/toast-errors.test.tsx
Pamela Chia 5db8a0e960 feat(studio): instrument sign-in attempts and failures (#49853)
The /sign-in page emitted only a pageview on entry and the success-side
`sign_in` event on exit: failed or abandoned attempts were invisible, so
"never interacted" and "tried and failed silently" could not be told
apart in the sign-in funnel. I added an unsampled `sign_in_submitted`
event at every initiation point and classified failure capture via
`dashboard_error_created` with a new `signin` origin.

**Changed:**
- **Submit attempts observable**: `sign_in_submitted` (method: `email`,
provider id, `sso`, or partner) fires from the DOM submit handler on the
password and SSO forms (so submits that fail client-side validation
still count), and from the OAuth, custom-provider, and partner
initiation handlers.
- **Failures classified**: each sign-in error path feeds the existing
funnel-error pipe with origin `signin` and a controlled reason slug
(`invalid_credentials`, `email_not_confirmed`, `captcha_failed`,
`sso_provider_not_found`, ...). GoTrue auth errors now classify via
their numeric `status`, guarded so transport failures (`status: 0`) stay
`network_error`.
- **Attempt events survive the OAuth redirect**: the telemetry event
POST sends with `keepalive` (scoped to `sign_in_submitted`, since
keepalive requests share a per-page in-flight body quota), so a
dispatched request is no longer aborted by the provider navigation; send
rejections are caught centrally instead of surfacing as unhandled
rejections. The fetch still dispatches after an async token lookup, so
preview testing verifies the GitHub-path event actually lands on the
wire.
- **Captcha rejection is no longer silent**: a rejected hCaptcha
challenge resolves the stuck loading toast with an error message, emits
`captcha_challenge_failed` (distinct from `captcha_failed`, which stays
reserved for the auth server rejecting a submitted token), reports to
error monitoring, and resets the captcha widget (previously: unhandled
promise rejection and a spinner that never resolved).
- **Partner method validated**: the partner sign-in page resolves the
URL-hash value against the provider registry and forwards the canonical
provider id into `method` on both `sign_in_submitted` and `sign_in`;
anything unregistered records as `unregistered_partner`, so a crafted
link can't poison the breakdown on either event.

**Note:** failure events stay on the shared 10%
`dashboard_error_created` sampling rate (a per-origin carve-out would
break cross-source volume comparability); the unsampled attempt event
carries the tried-vs-never-interacted signal at full volume.

## To test

Tested on Vercel preview (studio-staging, wire-level network capture +
staging ingestion check):
- [x] On `/sign-in`, submit a bogus email + password: expect a `POST
*/platform/telemetry/event` request with `action: sign_in_submitted`,
`method: email` in the network tab, plus an error toast. Observed: 201,
auth returned 400 as expected.
- [x] Submit with an empty password: expect `sign_in_submitted` to still
fire (validation failures count as attempts). Observed: event fired with
201 and no auth call followed.
- [x] Click "Continue with GitHub": expect `sign_in_submitted` with
`method: github` on the wire before the provider redirect. Observed: the
POST completed (201) before the browser landed on github.com, so the
keepalive path holds.
- [x] Negative case: fresh page load with no interaction fires no
`sign_in_submitted`.
- [x] Ingestion: all fired events (methods `email`, `github`, plus
organic `sso` submits from a real login on the same preview) arrived in
the staging project with the expected properties.
- [x] Re-ran the email and GitHub paths on the scoped-keepalive build
(`129bf8d`): both `sign_in_submitted` POSTs returned 201 (the GitHub one
completed despite the provider redirect), and both events ingested into
the staging project with the expected `method`/`category` properties.

## Linear
- GROWTH-1165 (no `fixes` keyword on purpose: the evidence checks run on
prod data post-deploy, and the issue closes manually after they pass)


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Improved sign-in protection with more reliable invisible CAPTCHA
handling.
* Added sign-in submission tracking across password, SSO, partner,
custom OAuth, and external-provider flows.
* Added detailed classification for authentication, validation, CAPTCHA,
provider, and network errors.

* **Bug Fixes**
  * Sign-in now stops safely and resets CAPTCHA when verification fails.
* Improved error reporting for failed sign-in attempts, including
redirects and OAuth flows.
* Ensured sign-in telemetry is delivered reliably during OAuth
redirects.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-09-02 19:02:37 +08:00

129 lines
3.8 KiB
TypeScript

import { act, render, renderHook, waitFor } from '@testing-library/react'
import { toast } from 'sonner'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { registerFunnelErrorToast, ToastErrorTracker } from './toast-errors'
import { useTrackFunnelError } from '@/lib/telemetry/use-track-funnel-error'
const { mockTrack } = vi.hoisted(() => ({ mockTrack: vi.fn() }))
vi.mock('@/lib/telemetry/track', () => ({ useTrack: () => mockTrack }))
describe('ToastErrorTracker', () => {
beforeEach(() => {
mockTrack.mockReset()
vi.spyOn(Math, 'random').mockReturnValue(0)
})
afterEach(() => {
vi.restoreAllMocks()
})
it('tracks an unregistered error toast without funnel properties', async () => {
render(<ToastErrorTracker />)
act(() => {
toast.error('request failed')
})
await waitFor(() =>
expect(mockTrack).toHaveBeenCalledWith('dashboard_error_created', { source: 'toast' })
)
expect(mockTrack).toHaveBeenCalledTimes(1)
})
it('fires a single enriched event for a registered funnel toast', async () => {
render(<ToastErrorTracker />)
act(() => {
registerFunnelErrorToast(toast.error('funnel error'), {
origin: 'signup',
errorCategory: 'api',
errorReason: 'other',
errorCode: 500,
})
})
await waitFor(() =>
expect(mockTrack).toHaveBeenCalledWith('dashboard_error_created', {
source: 'toast',
origin: 'signup',
errorCategory: 'api',
errorReason: 'other',
errorCode: 500,
})
)
expect(mockTrack).toHaveBeenCalledTimes(1)
})
it('routes toast-sourced funnel errors from useTrackFunnelError into one enriched event', async () => {
render(<ToastErrorTracker />)
const { result } = renderHook(() => useTrackFunnelError())
act(() => {
const toastId = toast.error('funnel failure')
result.current(
'project_creation',
{ errorCategory: 'api', errorReason: 'rate_limited', errorCode: 429 },
'toast',
toastId
)
})
await waitFor(() =>
expect(mockTrack).toHaveBeenCalledWith('dashboard_error_created', {
source: 'toast',
origin: 'project_creation',
errorCategory: 'api',
errorReason: 'rate_limited',
errorCode: 429,
})
)
expect(mockTrack).toHaveBeenCalledTimes(1)
})
it('tracks a loading toast updated to an error exactly once (sign-in reuses the loading toast id)', async () => {
render(<ToastErrorTracker />)
let toastId: string | number
act(() => {
toastId = toast.loading('Signing in...')
})
act(() => {
toast.error('Invalid login credentials', { id: toastId })
registerFunnelErrorToast(toastId, {
origin: 'signin',
errorCategory: 'api',
errorReason: 'invalid_credentials',
errorCode: 400,
})
})
await waitFor(() =>
expect(mockTrack).toHaveBeenCalledWith('dashboard_error_created', {
source: 'toast',
origin: 'signin',
errorCategory: 'api',
errorReason: 'invalid_credentials',
errorCode: 400,
})
)
expect(mockTrack).toHaveBeenCalledTimes(1)
})
it('ignores non-error toasts', async () => {
render(<ToastErrorTracker />)
act(() => {
toast.success('all good')
})
act(() => {
toast.error('unmarked sentinel')
})
await waitFor(() => expect(mockTrack).toHaveBeenCalledTimes(1))
})
it('skips error toasts that lose the sampling draw', async () => {
vi.spyOn(Math, 'random').mockReturnValueOnce(0.5)
render(<ToastErrorTracker />)
act(() => {
toast.error('sampled out')
})
act(() => {
toast.error('sampled in')
})
await waitFor(() => expect(mockTrack).toHaveBeenCalledTimes(1))
})
})