Files
supabase/apps/studio/lib/telemetry/funnel-errors.ts
Pamela Chia 98cfe3307e feat(telemetry): fix creation-funnel tracking gaps (#47386)
## Summary

The creation-funnel instrumentation that shipped Jun 25 (#47291, #47293)
had real gaps, surfaced by the weekly telemetry audit and confirmed
against production PostHog data before I touched code. The two automated
reports also contradicted each other on `errorReason`; I checked
production (every value is a controlled slug) and the emit path (only
`useTrackFunnelError` sets it, and it only accepts classified slugs), so
I left the type as-is rather than add a cross-package abstraction for a
risk that cannot occur today.

## Changes

- Classify HTTP 401/403/404 API errors as `unauthorized` / `forbidden` /
`not_found` instead of the catch-all `other`. In production the
`org_creation` `other` bucket was ~96% 401s (~1,300 real over 4 days),
invisible in reason breakdowns. The status-code fallback runs after the
message-pattern match, so specific reasons still win and it only rescues
errors that would otherwise be `other`.
- Add a single `tier` property (`tier_free` / `tier_pro` / `tier_payg` /
`tier_team`) to `organization_creation_completed`, which previously
carried no properties. One canonical billing slug (matching
`SubscriptionTier`) instead of two overlapping plan/tier fields, so the
org-creation funnel segments cleanly by tier and joins against
subscription data. `tier_payg` is uncapped PRO.
- Freeze the submitted tier at submit time (snapshot in `createOrg`)
rather than reading live form state in the success callback, so the
event records the tier that was actually created even if the user edits
the form during the async payment flow.
- Emit `project_creation_form_exposed` with `surface: 'vercel'` on the
integration deploy-button project-creation page (the enum value existed
but was never fired). Gated on the URL `slug` so the impression is
captured as soon as the form renders, matching the sibling exposure hook
on that page.

I also checked the confirm-modal error path flagged in the insights
post: it already classifies via the shared
`useProjectCreateMutation.onError`, so adding instrumentation there
would double-count. No change made.

## Testing

These are analytics events with no UI change, so correctness is in what
lands in PostHog. Post-deploy validation I will run against production
(project 34344):

- `dashboard_error_created` where `origin='org_creation'` and
`errorReason='other'` drops ~96%, with `unauthorized` / `not_found`
appearing.
- `organization_creation_completed.tier` populated on 100% of new events
with one of the four tier slugs.
- `project_creation_form_exposed` with `surface='vercel'` goes from 0 to
greater than 0.

## Linear

- fixes GROWTH-948


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added telemetry for organization creation completion that includes the
selected billing tier.
* Added one-time telemetry when the Vercel project creation form is
exposed.
* **Bug Fixes**
* Improved API error classification to more accurately distinguish
unauthorized, forbidden, and not found responses.
* **Documentation**
* Updated telemetry event definitions to require tier metadata for
organization creation events.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 23:11:53 +08:00

156 lines
5.2 KiB
TypeScript

import type { FieldErrors } from 'react-hook-form'
export type FunnelOrigin = 'signup' | 'project_creation' | 'org_creation'
export type ErrorCategory = 'validation' | 'api' | 'network' | 'payment' | 'unknown'
export interface FunnelErrorClassification {
errorCategory: ErrorCategory
errorReason: FunnelErrorReason
errorCode?: number
}
const RATE_LIMIT_STATUS = 429
const API_REASON_PATTERNS = {
signup: [
[/already registered|already been registered|already exists/i, 'email_already_registered'],
[/rate limit|too many requests|after \d+ second/i, 'rate_limited'],
[/captcha/i, 'captcha_failed'],
[/password/i, 'password_rejected'],
[/valid email|invalid email|email address/i, 'email_invalid'],
],
project_creation: [
[/already exists/i, 'project_name_taken'],
[/free plan|free tier/i, 'free_tier_limit'],
[/limit|maximum number|can only have/i, 'project_limit_reached'],
[/payment|invoice|overdue|past due|billing/i, 'billing_issue'],
[/region/i, 'region_unavailable'],
[/db_pass|password/i, 'db_password_rejected'],
],
org_creation: [
[/already exists|name.*taken/i, 'org_name_taken'],
[/payment|card|invoice|billing/i, 'billing_issue'],
[/limit/i, 'org_limit_reached'],
],
} as const satisfies Record<FunnelOrigin, ReadonlyArray<readonly [RegExp, string]>>
const VALIDATION_FIELD_REASONS = {
signup: {
email: 'email_invalid',
password: 'password_invalid',
},
project_creation: {
organization: 'organization_missing',
projectName: 'project_name_invalid',
dbPass: 'db_password_weak',
dbPassStrength: 'db_password_weak',
dbRegion: 'region_missing',
cloudProvider: 'cloud_provider_invalid',
postgresVersion: 'postgres_version_missing',
highAvailability: 'incompatible_options',
useOrioleDb: 'incompatible_options',
},
org_creation: {
name: 'org_name_missing',
kind: 'org_kind_invalid',
size: 'org_size_invalid',
},
} as const satisfies Record<FunnelOrigin, Readonly<Record<string, string>>>
const STRIPE_DECLINE_REASONS = {
insufficient_funds: 'card_insufficient_funds',
card_declined: 'card_declined',
expired_card: 'card_expired',
incorrect_cvc: 'card_incorrect_cvc',
incorrect_number: 'card_incorrect_number',
processing_error: 'card_processing_error',
} as const satisfies Record<string, string>
const GENERIC_REASONS = [
'rate_limited',
'server_error',
'connection_timeout',
'network_error',
'payment_failed',
'payment_error',
'oriole_unavailable',
'unauthorized',
'forbidden',
'not_found',
'other',
] as const
type ValuesOf<T> = T extends Readonly<Record<string, infer V extends string>> ? V : never
export type FunnelErrorReason =
| (typeof API_REASON_PATTERNS)[FunnelOrigin][number][1]
| ValuesOf<(typeof VALIDATION_FIELD_REASONS)[FunnelOrigin]>
| ValuesOf<typeof STRIPE_DECLINE_REASONS>
| (typeof GENERIC_REASONS)[number]
const STATUS_REASONS: Readonly<Partial<Record<number, FunnelErrorReason>>> = {
401: 'unauthorized',
403: 'forbidden',
404: 'not_found',
}
export function classifyApiError(origin: FunnelOrigin, error: unknown): FunnelErrorClassification {
const err = error as { code?: unknown; errorType?: unknown; message?: unknown }
const code = typeof err?.code === 'number' ? err.code : undefined
const message = typeof err?.message === 'string' ? err.message : ''
if (err?.errorType === 'connection-timeout') {
return { errorCategory: 'network', errorReason: 'connection_timeout' }
}
if (code === undefined) {
return { errorCategory: 'network', errorReason: 'network_error' }
}
if (code === RATE_LIMIT_STATUS) {
return { errorCategory: 'api', errorReason: 'rate_limited', errorCode: code }
}
if (code >= 500) {
return { errorCategory: 'api', errorReason: 'server_error', errorCode: code }
}
for (const [pattern, reason] of API_REASON_PATTERNS[origin]) {
if (pattern.test(message)) {
return { errorCategory: 'api', errorReason: reason, errorCode: code }
}
}
const statusReason = STATUS_REASONS[code]
if (statusReason) {
return { errorCategory: 'api', errorReason: statusReason, errorCode: code }
}
return { errorCategory: 'api', errorReason: 'other', errorCode: code }
}
export function classifyValidationError(
origin: FunnelOrigin,
errors: FieldErrors
): FunnelErrorClassification {
const fieldErrors = errors as Record<string, unknown>
const reasons = VALIDATION_FIELD_REASONS[origin] as Readonly<Record<string, FunnelErrorReason>>
for (const field of Object.keys(reasons)) {
if (fieldErrors[field]) {
return { errorCategory: 'validation', errorReason: reasons[field] }
}
}
return { errorCategory: 'validation', errorReason: 'other' }
}
export function classifyStripeError(error: unknown): FunnelErrorClassification {
const err = error as { code?: unknown; decline_code?: unknown }
const key =
typeof err?.decline_code === 'string'
? err.decline_code
: typeof err?.code === 'string'
? err.code
: undefined
const reason = key
? (STRIPE_DECLINE_REASONS as Readonly<Record<string, FunnelErrorReason>>)[key]
: undefined
if (reason) {
return { errorCategory: 'payment', errorReason: reason }
}
return { errorCategory: 'payment', errorReason: 'payment_failed' }
}