mirror of
https://github.com/supabase/supabase.git
synced 2026-09-06 09:59:03 +08:00
## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Bug fix (performance), follow-up to #47894, plus regression-guard tests. ## What is the current behavior? #47894 scoped the Table Editor and entity-definition introspection queries, but four more `@supabase/pg-meta` query families still do O(catalog) work per request. On a production project with a very large catalog (hundreds of schemas, ~465K `pg_constraint` rows) they run 5 to 55 seconds each, trip the 58s `statement_timeout`, and spill sorts to temp files. During a recent "DB CPU > 85%" incident on such a project, 24 of 27 active backends were running these queries concurrently. 1. **`tables.retrieve()` (single-table lookup by name+schema or id)**: the `tables`/`columns` CTEs scan the whole catalog (`pg_class`, `pg_constraint`, `pg_index`, all of `pg_attribute`, per-table sizes) and the one-table predicate is applied only on the outer select. Same bug class #47894 fixed for the OID-based table editor query; this sibling path never got the treatment. It accounted for 94 of the 96 statement-timeout cancellations in the incident. 2. **Types listing**: the `t_enums` and `t_attributes` subqueries aggregate the entire `pg_enum` and every composite relation before the wrapper's schema filter applies. 3. **Table privileges**: `aclexplode` + double `pg_roles` join + GROUP BY over every relation in the database; schema/OID filters applied only after aggregation, in both `list()` and `retrieve()`. 4. **Row counts**: `getTableRowsCountSql` treats `reltuples = -1` (never-analyzed table) as "small table, run exact count(*)". A freshly bulk-loaded multi-million-row table times out on every Table Editor pagination render. Two Studio-side amplifiers turned one slow query into a sustained load storm: - `useTableQuery` (behind `tables.retrieve()`) mounts once per visible foreign-key grid cell via `ForeignKeyFormatter`, so a single Table Editor view fires ~20 concurrent copies against the FK target table. A timed-out query caches nothing, and TanStack retries errored no-data queries on every observer mount by default, so scrolling kept re-issuing the 58s scan. - `useTableApiAccessQuery` fetched table privileges for the entire database and filtered down to one schema client-side. ## What is the new behavior? **pg-meta (all behind the existing `pgMetaScopedIntrospection` flag, same rollout mechanism as #47894; `scoped: false` keeps serving the current SQL):** - `tables.retrieve()`: the identifier is resolved to a scalar `targetOid` init-plan and pushed into the base scan, primary-key, relationships (both FK directions kept: `conrelid` or `confrelid`) and columns CTEs. A materialized `target` CTE was deliberately avoided: it acts as an optimization barrier and forces the very seq scans being removed. - Types: filter `pg_type`/`pg_namespace` first, then compute enums/attributes per surviving row via correlated index-scan subqueries (`pg_enum(enumtypid, enumsortorder)`, `pg_attribute(attrelid, attnum)`). - Table privileges: schema/OID predicates injected into the base WHERE before `aclexplode`/GROUP BY for `list()` and `retrieve()`. - Row counts: `reltuples = -1` is treated as "unknown" and gated on physical size via `pg_relation_size` (a cheap stat call; `relpages` is equally stale pre-vacuum). At or below `THRESHOLD_ESTIMATE_BYTES` (~10MB, derived from `THRESHOLD_COUNT` at a conservative ~200 bytes/row) the exact count runs as before: fast by construction, and it avoids bogus estimates since Postgres floors never-vacuumed heaps at 10 pages, so an empty table would otherwise report ~2K estimated rows. Above the gate the count routes through the EXPLAIN-based `pg_temp.count_estimate`, or returns `-1`/`is_estimate = true` in read-only contexts where the temp function cannot be created. The scoped branch embeds the estimated select via `literal()` instead of legacy's apostrophe-only escaping, so it stays correct under `standard_conforming_strings = off`. `enforceExactCount` unchanged. **Studio:** - The flag decision is contained in the data layer instead of prop-drilled: a small imperative accessor (`apps/studio/data/scoped-introspection.ts`) is hydrated from `useFlag` via a one-line `useSyncScopedIntrospection()` call in `DefaultLayout`, and the query functions read it internally when building the pg-meta SQL. `DefaultLayout` is shared by both the Next and TanStack router trees; hydrating from `_app.tsx` alone would leave TanStack-served pages permanently unscoped since `routes/__root.tsx` mounts its own flag provider. Cold loads cannot race the flag: the query functions await a readiness promise that resolves only after the sync hook has hydrated the accessor with a loaded flag store (immediately on self-hosted where flags are disabled; a 5s safety net armed lazily on the first `ready()` call - not at module import, which would let the timer expire before a project page ever mounts - bounds genuine ConfigCat outages). No component threading, no query-key changes (remaining tradeoff, documented in the module: a mid-session flag flip can serve stale-keyed caches until refetch, fine for a session-stable rollout flag). #47894's existing threading is left as-is and gets deleted together with the flag in the cleanup PR. Also fixes the previously-missing `scoped` pass-through in `getTableRowsCount`. - Flag-independent hardening: `useTableQuery` now sets `retryOnMount: false`, `refetchOnWindowFocus: false` and `staleTime: 5min`. Errored (timed-out) queries no longer refire on every grid cell remount, while stale successful metadata still revalidates on mount after `staleTime`. - `useTableApiAccessQuery` now passes `includedSchemas: [schemaName]`; the client-side filter stays as a safety net. - The rows-count query is `enabled`-gated on the permission check settling, so a transiently-false `canSQLAdminWrite` can no longer cache a read-only `-1` count for a writable user (read replicas short-circuit synchronously as before). **Regression guards (extending the #47894 infrastructure):** - Execution-based scoped-vs-legacy equivalence tests for all four queries: both variants run against the test database and are compared with raw `toEqual` - no normalization, ids included (types across 6 option combos, privileges incl. multi-grantee + PUBLIC, `tables.retrieve` for both identifier branches, row counts for every case where the two paths must agree). Two documented exceptions where only the LEGACY side is sorted, because a de-normalized diagnostic run proved legacy emits genuinely plan-dependent order there (an adversarial-FK fixture shows it is neither oid, name, nor creation order): the `types.list` outer row order (scoped adds `order by t.oid`; legacy has no ORDER BY) and the `tables.retrieve` relationships array (scoped orders by `constraint_name` + column-name tie-breakers - a composite two-column FK expands to 4 entries sharing one constraint_name). Everything else (privileges via `aclexplode` over the same relacl, columns by `ordinal_position`, primary keys by `indkey` order, enums by `enumsortorder`) is byte-identical between the two paths with no test-side help. The one intentional value divergence, never-analyzed tables above the size gate where legacy's exact count is the timeout bug itself, is asserted explicitly as a divergence. - Plan-guard budgets for every scoped query against the stress catalog (extended with 200 enums + 200 composite types). Residual seq scans are justified in-budget: `pg_constraint` max 2 (no index on `confrelid`), `pg_attrdef` max 1, `pg_authid` max 2 (scales with role count, not schema count). - Legacy templates carry a FROZEN do-not-edit marker (they must keep matching production behavior until the flag cleanup deletes them); the ordinary test suite runs against the legacy default, so behavioral drift there fails regular tests. ### Validation - pg-meta: typecheck clean; the affected suites (types, table-privileges, tables, rows-count, catalog-plan-guard) pass in full. - Cross-version: the scoped-vs-legacy equivalence and rows-count behavioral suites were validated on PostgreSQL 14, 15, and 17 (identical results on all three). Two version-marginal planner choices surfaced on 17 (`pg_type` / `pg_class` seq scan vs full-index bitmap for per-schema listings, both structurally unavoidable without an index leading on the namespace column) and are carried as justified plan-guard budget entries. A full 468-test suite run sequentially: 452 passed, 16 failures verified environmental (13 timeouts in an untouched file that passes 27/27 in isolation on the marathon-run cluster, 3 cluster-global role collisions from container reuse). - Studio: `pnpm --filter studio typecheck` clean; 39/39 tests across the touched data hooks; eslint clean on touched files. ### Rollout Same staged ConfigCat rollout as #47894 via `pgMetaScopedIntrospection` (user-email targeting first, then percentage, then 100%). The `useTableQuery` hardening and the API-access schema scoping ship unflagged (behavior-safe). Gate before percentage rollout: functionally verify the FK popover/selector UX under the new `staleTime`/`retryOnMount` settings (a just-edited FK target must not look stale anywhere Studio does not already refetch on save). Once fully rolled out, the legacy templates and flag get deleted together with #47894's in one cleanup PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
157 lines
4.7 KiB
TypeScript
157 lines
4.7 KiB
TypeScript
import { getTableRowsCountSql } from '@supabase/pg-meta'
|
|
import { PermissionAction } from '@supabase/shared-types/out/constants'
|
|
import { QueryClient, useQuery, useQueryClient } from '@tanstack/react-query'
|
|
import { IS_PLATFORM, useFlag } from 'common'
|
|
|
|
import { tableRowKeys } from './keys'
|
|
import { formatFilterValue } from './utils'
|
|
import { parseSupaTable } from '@/components/grid/SupabaseGrid.utils'
|
|
import type { Filter, SupaTable } from '@/components/grid/types'
|
|
import { useConnectionStringForReadOps } from '@/data/read-replicas/replicas-query'
|
|
import { executeSql } from '@/data/sql/execute-sql-mutation'
|
|
import {
|
|
PG_META_SCOPED_INTROSPECTION_FLAG,
|
|
prefetchTableEditor,
|
|
} from '@/data/table-editor/table-editor-query'
|
|
import { useAsyncCheckPermissions } from '@/hooks/misc/useCheckPermissions'
|
|
import { RoleImpersonationState, wrapWithRoleImpersonation } from '@/lib/role-impersonation'
|
|
import { isRoleImpersonationEnabled } from '@/state/role-impersonation-state'
|
|
import { ResponseError, UseCustomQueryOptions } from '@/types'
|
|
|
|
export type GetTableRowsCountArgs = {
|
|
table?: SupaTable
|
|
filters?: Filter[]
|
|
enforceExactCount?: boolean
|
|
}
|
|
|
|
export type TableRowsCount = {
|
|
count?: number
|
|
is_estimate?: boolean
|
|
}
|
|
|
|
export type TableRowsCountVariables = Omit<GetTableRowsCountArgs, 'table'> & {
|
|
queryClient: QueryClient
|
|
tableId?: number
|
|
roleImpersonationState?: RoleImpersonationState
|
|
projectRef?: string
|
|
connectionString?: string | null
|
|
scoped?: boolean
|
|
}
|
|
|
|
export type TableRowsCountData = TableRowsCount
|
|
export type TableRowsCountError = ResponseError
|
|
|
|
export async function getTableRowsCount(
|
|
{
|
|
queryClient,
|
|
projectRef,
|
|
connectionString,
|
|
tableId,
|
|
filters,
|
|
roleImpersonationState,
|
|
enforceExactCount,
|
|
isReadOnlyContext = false,
|
|
scoped,
|
|
}: TableRowsCountVariables & { isReadOnlyContext?: boolean },
|
|
signal?: AbortSignal
|
|
) {
|
|
const entity = await prefetchTableEditor(queryClient, {
|
|
projectRef,
|
|
connectionString,
|
|
id: tableId,
|
|
scoped,
|
|
})
|
|
if (!entity) {
|
|
throw new Error('Table not found')
|
|
}
|
|
|
|
const table = parseSupaTable(entity)
|
|
|
|
const formattedFilters = filters?.map((x) => ({ ...x, value: formatFilterValue(table, x) }))
|
|
const sql = wrapWithRoleImpersonation(
|
|
getTableRowsCountSql({
|
|
table,
|
|
filters: formattedFilters,
|
|
enforceExactCount,
|
|
isReadOnlyContext,
|
|
scoped,
|
|
}),
|
|
roleImpersonationState
|
|
)
|
|
const { result } = await executeSql(
|
|
{
|
|
projectRef,
|
|
connectionString,
|
|
sql,
|
|
queryKey: ['table-rows-count', table.id],
|
|
isRoleImpersonationEnabled: isRoleImpersonationEnabled(roleImpersonationState?.role),
|
|
},
|
|
signal
|
|
)
|
|
|
|
return {
|
|
count: result?.[0]?.count,
|
|
is_estimate: result?.[0]?.is_estimate ?? false,
|
|
} as TableRowsCount
|
|
}
|
|
|
|
export const useTableRowsCountQuery = <TData = TableRowsCountData>(
|
|
{
|
|
projectRef,
|
|
tableId,
|
|
...args
|
|
}: Omit<TableRowsCountVariables, 'queryClient' | 'connectionString'>,
|
|
{
|
|
enabled = true,
|
|
...options
|
|
}: UseCustomQueryOptions<TableRowsCountData, TableRowsCountError, TData> = {}
|
|
) => {
|
|
const queryClient = useQueryClient()
|
|
const {
|
|
connectionString,
|
|
identifier: readReplicaIdentifier,
|
|
type,
|
|
} = useConnectionStringForReadOps()
|
|
const { can: canSQLAdminWrite, isLoading: isPermissionsLoading } = useAsyncCheckPermissions(
|
|
PermissionAction.TENANT_SQL_ADMIN_WRITE,
|
|
'tables'
|
|
)
|
|
const scoped = !!useFlag(PG_META_SCOPED_INTROSPECTION_FLAG)
|
|
|
|
return useQuery<TableRowsCountData, TableRowsCountError, TData>({
|
|
queryKey: tableRowKeys.tableRowsCount(projectRef, {
|
|
table: { id: tableId },
|
|
readReplicaIdentifier,
|
|
...args,
|
|
scoped,
|
|
}),
|
|
queryFn: ({ signal }) =>
|
|
getTableRowsCount(
|
|
{
|
|
queryClient,
|
|
projectRef,
|
|
connectionString,
|
|
tableId,
|
|
isReadOnlyContext: type === 'replica' || !canSQLAdminWrite,
|
|
...args,
|
|
scoped,
|
|
},
|
|
signal
|
|
),
|
|
enabled:
|
|
enabled &&
|
|
typeof projectRef !== 'undefined' &&
|
|
typeof tableId !== 'undefined' &&
|
|
(!IS_PLATFORM || typeof connectionString !== 'undefined') &&
|
|
// isReadOnlyContext resolves to `type === 'replica' || !canSQLAdminWrite`: for
|
|
// read replicas it's already known synchronously, but otherwise it depends on
|
|
// canSQLAdminWrite, which starts out `false` while permissions are loading.
|
|
// Firing while that's still in flight would cache a transient
|
|
// isReadOnlyContext:true (and, on a never-analyzed table, a scoped
|
|
// count:-1/is_estimate:true) for what may actually be a writable user. Wait
|
|
// for the permission check to settle before firing in that case.
|
|
(type === 'replica' || !isPermissionsLoading),
|
|
...options,
|
|
})
|
|
}
|