Files
supabase/apps/studio/lib/toast-errors.test.tsx
Pamela Chia 9858562b8b fix(telemetry): dedupe funnel toast error events (#47802)
## Summary
Since #47293, an API failure on a signup / org-creation /
project-creation form emitted `dashboard_error_created` twice:
`useTrackFunnelError` fired the origin-tagged event and the global
`ToastErrorTracker` independently fired the legacy untagged
`source:'toast'` event for the same toast, each behind its own 10%
sampling draw. I verified the twin rate empirically at 8-11% of
origin-tagged funnel toasts, exactly the floor for two independent 10%
draws, meaning the twin co-fires for effectively every funnel error
([Hex
thread](https://app.hex.tech/supabase/thread/019f3bc1-3a5c-7200-9122-8e3439bfbe8c)).
Any consumer counting funnel errors without an `origin IS NOT NULL`
filter saw ~2x inflation.

The fix makes `ToastErrorTracker` the sole emitter of `source:'toast'`
events, so the duplicate is unrepresentable rather than suppressed.
Funnel call sites pass the id returned by `toast.error()` into
`trackFunnelError`, which registers the funnel properties against that
toast id instead of firing its own event – the tracker then emits a
single `dashboard_error_created` enriched with `origin` /
`errorCategory` / `errorReason` / `errorCode` for registered toasts, and
the plain untagged event otherwise. The `'toast'` overload of
`trackFunnelError` requires the toast id, so a missed pairing is a
compile error rather than a silent double count. Registration is
unconditional and there's only one sampling draw, so suppression can't
lose a sampling race. `'form'`-sourced funnel events are unchanged.

## Changes
- `lib/toast-errors.tsx`: toast-id → funnel-properties registry
(`registerFunnelErrorToast`); `ToastErrorTracker` emits one (optionally
enriched) event per error toast under a single 10% draw, deleting
entries once consumed
- `lib/telemetry/use-track-funnel-error.ts`: overloaded signature –
`'toast'` requires the id returned by `toast.error()` (type-enforced),
`'form'` keeps direct emission with its own sampling
- Update the 7 funnel `toast.error` call sites in `NewOrgForm`,
`SignUpForm`, and `pages/new/[slug]` to pass the toast id
- Component tests for the tracker (previously uncovered), including an
end-to-end test through `useTrackFunnelError`
- Code hygiene (also flagged by CodeRabbit): all four
`dashboard_error_created` emitters (toast, form, `AlertError`,
`ErrorMatcher`) independently encoded the 10% draw – downstream analysis
assumes a uniform sampling multiplier across sources, so one site
drifting would silently skew comparisons. The rate and the draw now live
in one place (`isDashboardErrorSampled()` in
`lib/telemetry/error-sampling.ts`). No behavior change.
- Mount `ToastErrorTracker` in the TanStack root (`routes/__root.tsx`),
mirroring `pages/_app.tsx`. The TanStack tree mounted `Toaster` but
never the tracker, so untagged toast error telemetry has never fired in
that flavour – and with the tracker now the sole emitter, the missing
mount would have silently dropped funnel toast events there too. Side
effect once the TanStack flavour ships: untagged `source:'toast'` volume
from it goes from zero to normal.

## Testing
Component-tested (`apps/studio/lib/toast-errors.test.tsx`):
- [x] Unregistered error toast fires exactly one untagged
`dashboard_error_created {source:'toast'}`
- [x] Registered funnel toast fires exactly one event, enriched with
`origin`/`errorCategory`/`errorReason`/`errorCode`
- [x] `useTrackFunnelError` with a toast id routes through the tracker
as a single enriched event
- [x] Non-error toasts ignored; the 10% sampling gate still applies

Full Studio unit suite passes (392 files / 4371 tests), plus typecheck
and lint.

Also verified end-to-end in a local browser (TanStack flavour, sample
rate temporarily forced to 1): a failed signup produced exactly one
`dashboard_error_created` with `{source:'toast', origin:'signup',
errorCategory:'api', errorReason:'email_already_registered',
errorCode:403}` and no untagged twin (two independent trials); an
unregistered error toast produced exactly one plain `{source:'toast'}`;
a client-side validation failure produced exactly one `{source:'form',
origin:'signup', errorCategory:'validation',
errorReason:'email_invalid'}`; success toasts produced nothing.

Post-deploy I'll re-run the twin-rate query from the Hex thread; the
untagged-twin rate on funnel pages should decay to ~0 as stale bundles
reload over 2-3 days.

## Notes
- Origin-tagged funnel toast events now ride the tracker's single 10%
draw instead of their own independent draw – statistically identical
volume, but the event fires on the tracker's next effect rather than
synchronously at the call site (irrelevant for PostHog)
- Registration must happen in the same synchronous block as
`toast.error()` (documented on the `TrackFunnelError` type) – all
current call sites comply
- The invalid Postgres version toast in `pages/new/[slug].tsx` (~line
416) needs no special-casing: unregistered toasts keep the plain
untagged event, so its telemetry is preserved
- Heads-up for `dashboard_error_created` consumers: overall untagged
`source:'toast'` volume will dip slightly after this deploys, since
funnel-page twins disappear. A volume monitor seeing that drop is this
fix landing, not a tracking regression (same class as the intended
GROWTH-893 sampling-unification drop).

## Linear
- fixes GROWTH-965


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Enhanced error telemetry for organization creation, sign-up, payment,
and project-creation flows by associating failures with toast
identifiers and enriched funnel context.
* Standardized dashboard error sampling logic across error handling
components for consistency.

* **Tests**
* Added comprehensive test coverage for toast error tracking, including
funnel registration, deduplication, filtering, and sampling behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
2026-07-10 17:35:27 +08:00

102 lines
2.9 KiB
TypeScript

import { act, render, renderHook, waitFor } from '@testing-library/react'
import { toast } from 'sonner'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { registerFunnelErrorToast, ToastErrorTracker } from './toast-errors'
import { useTrackFunnelError } from '@/lib/telemetry/use-track-funnel-error'
const { mockTrack } = vi.hoisted(() => ({ mockTrack: vi.fn() }))
vi.mock('@/lib/telemetry/track', () => ({ useTrack: () => mockTrack }))
describe('ToastErrorTracker', () => {
beforeEach(() => {
mockTrack.mockReset()
vi.spyOn(Math, 'random').mockReturnValue(0)
})
afterEach(() => {
vi.restoreAllMocks()
})
it('tracks an unregistered error toast without funnel properties', async () => {
render(<ToastErrorTracker />)
act(() => {
toast.error('request failed')
})
await waitFor(() =>
expect(mockTrack).toHaveBeenCalledWith('dashboard_error_created', { source: 'toast' })
)
expect(mockTrack).toHaveBeenCalledTimes(1)
})
it('fires a single enriched event for a registered funnel toast', async () => {
render(<ToastErrorTracker />)
act(() => {
registerFunnelErrorToast(toast.error('funnel error'), {
origin: 'signup',
errorCategory: 'api',
errorReason: 'other',
errorCode: 500,
})
})
await waitFor(() =>
expect(mockTrack).toHaveBeenCalledWith('dashboard_error_created', {
source: 'toast',
origin: 'signup',
errorCategory: 'api',
errorReason: 'other',
errorCode: 500,
})
)
expect(mockTrack).toHaveBeenCalledTimes(1)
})
it('routes toast-sourced funnel errors from useTrackFunnelError into one enriched event', async () => {
render(<ToastErrorTracker />)
const { result } = renderHook(() => useTrackFunnelError())
act(() => {
const toastId = toast.error('funnel failure')
result.current(
'project_creation',
{ errorCategory: 'api', errorReason: 'rate_limited', errorCode: 429 },
'toast',
toastId
)
})
await waitFor(() =>
expect(mockTrack).toHaveBeenCalledWith('dashboard_error_created', {
source: 'toast',
origin: 'project_creation',
errorCategory: 'api',
errorReason: 'rate_limited',
errorCode: 429,
})
)
expect(mockTrack).toHaveBeenCalledTimes(1)
})
it('ignores non-error toasts', async () => {
render(<ToastErrorTracker />)
act(() => {
toast.success('all good')
})
act(() => {
toast.error('unmarked sentinel')
})
await waitFor(() => expect(mockTrack).toHaveBeenCalledTimes(1))
})
it('skips error toasts that lose the sampling draw', async () => {
vi.spyOn(Math, 'random').mockReturnValueOnce(0.5)
render(<ToastErrorTracker />)
act(() => {
toast.error('sampled out')
})
act(() => {
toast.error('sampled in')
})
await waitFor(() => expect(mockTrack).toHaveBeenCalledTimes(1))
})
})