Files
supabase/apps/studio/data/table-rows/table-rows-count-query.ts
Andrew Valleteau 6c6a721cb7 fix(pg-meta): scope remaining O(catalog) introspection queries behind pgMetaScopedIntrospection (#48148)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Bug fix (performance), follow-up to #47894, plus regression-guard tests.

## What is the current behavior?

#47894 scoped the Table Editor and entity-definition introspection
queries, but four more `@supabase/pg-meta` query families still do
O(catalog) work per request. On a production project with a very large
catalog (hundreds of schemas, ~465K `pg_constraint` rows) they run 5 to
55 seconds each, trip the 58s `statement_timeout`, and spill sorts to
temp files. During a recent "DB CPU > 85%" incident on such a project,
24 of 27 active backends were running these queries concurrently.

1. **`tables.retrieve()` (single-table lookup by name+schema or id)**:
the `tables`/`columns` CTEs scan the whole catalog (`pg_class`,
`pg_constraint`, `pg_index`, all of `pg_attribute`, per-table sizes) and
the one-table predicate is applied only on the outer select. Same bug
class #47894 fixed for the OID-based table editor query; this sibling
path never got the treatment. It accounted for 94 of the 96
statement-timeout cancellations in the incident.
2. **Types listing**: the `t_enums` and `t_attributes` subqueries
aggregate the entire `pg_enum` and every composite relation before the
wrapper's schema filter applies.
3. **Table privileges**: `aclexplode` + double `pg_roles` join + GROUP
BY over every relation in the database; schema/OID filters applied only
after aggregation, in both `list()` and `retrieve()`.
4. **Row counts**: `getTableRowsCountSql` treats `reltuples = -1`
(never-analyzed table) as "small table, run exact count(*)". A freshly
bulk-loaded multi-million-row table times out on every Table Editor
pagination render.

Two Studio-side amplifiers turned one slow query into a sustained load
storm:

- `useTableQuery` (behind `tables.retrieve()`) mounts once per visible
foreign-key grid cell via `ForeignKeyFormatter`, so a single Table
Editor view fires ~20 concurrent copies against the FK target table. A
timed-out query caches nothing, and TanStack retries errored no-data
queries on every observer mount by default, so scrolling kept re-issuing
the 58s scan.
- `useTableApiAccessQuery` fetched table privileges for the entire
database and filtered down to one schema client-side.

## What is the new behavior?

**pg-meta (all behind the existing `pgMetaScopedIntrospection` flag,
same rollout mechanism as #47894; `scoped: false` keeps serving the
current SQL):**

- `tables.retrieve()`: the identifier is resolved to a scalar
`targetOid` init-plan and pushed into the base scan, primary-key,
relationships (both FK directions kept: `conrelid` or `confrelid`) and
columns CTEs. A materialized `target` CTE was deliberately avoided: it
acts as an optimization barrier and forces the very seq scans being
removed.
- Types: filter `pg_type`/`pg_namespace` first, then compute
enums/attributes per surviving row via correlated index-scan subqueries
(`pg_enum(enumtypid, enumsortorder)`, `pg_attribute(attrelid, attnum)`).
- Table privileges: schema/OID predicates injected into the base WHERE
before `aclexplode`/GROUP BY for `list()` and `retrieve()`.
- Row counts: `reltuples = -1` is treated as "unknown" and gated on
physical size via `pg_relation_size` (a cheap stat call; `relpages` is
equally stale pre-vacuum). At or below `THRESHOLD_ESTIMATE_BYTES`
(~10MB, derived from `THRESHOLD_COUNT` at a conservative ~200 bytes/row)
the exact count runs as before: fast by construction, and it avoids
bogus estimates since Postgres floors never-vacuumed heaps at 10 pages,
so an empty table would otherwise report ~2K estimated rows. Above the
gate the count routes through the EXPLAIN-based
`pg_temp.count_estimate`, or returns `-1`/`is_estimate = true` in
read-only contexts where the temp function cannot be created. The scoped
branch embeds the estimated select via `literal()` instead of legacy's
apostrophe-only escaping, so it stays correct under
`standard_conforming_strings = off`. `enforceExactCount` unchanged.

**Studio:**

- The flag decision is contained in the data layer instead of
prop-drilled: a small imperative accessor
(`apps/studio/data/scoped-introspection.ts`) is hydrated from `useFlag`
via a one-line `useSyncScopedIntrospection()` call in `DefaultLayout`,
and the query functions read it internally when building the pg-meta
SQL. `DefaultLayout` is shared by both the Next and TanStack router
trees; hydrating from `_app.tsx` alone would leave TanStack-served pages
permanently unscoped since `routes/__root.tsx` mounts its own flag
provider. Cold loads cannot race the flag: the query functions await a
readiness promise that resolves only after the sync hook has hydrated
the accessor with a loaded flag store (immediately on self-hosted where
flags are disabled; a 5s safety net armed lazily on the first `ready()`
call - not at module import, which would let the timer expire before a
project page ever mounts - bounds genuine ConfigCat outages). No
component threading, no query-key changes (remaining tradeoff,
documented in the module: a mid-session flag flip can serve stale-keyed
caches until refetch, fine for a session-stable rollout flag). #47894's
existing threading is left as-is and gets deleted together with the flag
in the cleanup PR. Also fixes the previously-missing `scoped`
pass-through in `getTableRowsCount`.
- Flag-independent hardening: `useTableQuery` now sets `retryOnMount:
false`, `refetchOnWindowFocus: false` and `staleTime: 5min`. Errored
(timed-out) queries no longer refire on every grid cell remount, while
stale successful metadata still revalidates on mount after `staleTime`.
- `useTableApiAccessQuery` now passes `includedSchemas: [schemaName]`;
the client-side filter stays as a safety net.
- The rows-count query is `enabled`-gated on the permission check
settling, so a transiently-false `canSQLAdminWrite` can no longer cache
a read-only `-1` count for a writable user (read replicas short-circuit
synchronously as before).

**Regression guards (extending the #47894 infrastructure):**

- Execution-based scoped-vs-legacy equivalence tests for all four
queries: both variants run against the test database and are compared
with raw `toEqual` - no normalization, ids included (types across 6
option combos, privileges incl. multi-grantee + PUBLIC,
`tables.retrieve` for both identifier branches, row counts for every
case where the two paths must agree). Two documented exceptions where
only the LEGACY side is sorted, because a de-normalized diagnostic run
proved legacy emits genuinely plan-dependent order there (an
adversarial-FK fixture shows it is neither oid, name, nor creation
order): the `types.list` outer row order (scoped adds `order by t.oid`;
legacy has no ORDER BY) and the `tables.retrieve` relationships array
(scoped orders by `constraint_name` + column-name tie-breakers - a
composite two-column FK expands to 4 entries sharing one
constraint_name). Everything else (privileges via `aclexplode` over the
same relacl, columns by `ordinal_position`, primary keys by `indkey`
order, enums by `enumsortorder`) is byte-identical between the two paths
with no test-side help. The one intentional value divergence,
never-analyzed tables above the size gate where legacy's exact count is
the timeout bug itself, is asserted explicitly as a divergence.
- Plan-guard budgets for every scoped query against the stress catalog
(extended with 200 enums + 200 composite types). Residual seq scans are
justified in-budget: `pg_constraint` max 2 (no index on `confrelid`),
`pg_attrdef` max 1, `pg_authid` max 2 (scales with role count, not
schema count).
- Legacy templates carry a FROZEN do-not-edit marker (they must keep
matching production behavior until the flag cleanup deletes them); the
ordinary test suite runs against the legacy default, so behavioral drift
there fails regular tests.

### Validation

- pg-meta: typecheck clean; the affected suites (types,
table-privileges, tables, rows-count, catalog-plan-guard) pass in full.
- Cross-version: the scoped-vs-legacy equivalence and rows-count
behavioral suites were validated on PostgreSQL 14, 15, and 17 (identical
results on all three). Two version-marginal planner choices surfaced on
17 (`pg_type` / `pg_class` seq scan vs full-index bitmap for per-schema
listings, both structurally unavoidable without an index leading on the
namespace column) and are carried as justified plan-guard budget
entries. A full 468-test suite run sequentially: 452 passed, 16 failures
verified environmental (13 timeouts in an untouched file that passes
27/27 in isolation on the marathon-run cluster, 3 cluster-global role
collisions from container reuse).
- Studio: `pnpm --filter studio typecheck` clean; 39/39 tests across the
touched data hooks; eslint clean on touched files.

### Rollout

Same staged ConfigCat rollout as #47894 via `pgMetaScopedIntrospection`
(user-email targeting first, then percentage, then 100%). The
`useTableQuery` hardening and the API-access schema scoping ship
unflagged (behavior-safe). Gate before percentage rollout: functionally
verify the FK popover/selector UX under the new
`staleTime`/`retryOnMount` settings (a just-edited FK target must not
look stale anywhere Studio does not already refetch on save). Once fully
rolled out, the legacy templates and flag get deleted together with
#47894's in one cleanup PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 08:07:08 +02:00

157 lines
4.7 KiB
TypeScript

import { getTableRowsCountSql } from '@supabase/pg-meta'
import { PermissionAction } from '@supabase/shared-types/out/constants'
import { QueryClient, useQuery, useQueryClient } from '@tanstack/react-query'
import { IS_PLATFORM, useFlag } from 'common'
import { tableRowKeys } from './keys'
import { formatFilterValue } from './utils'
import { parseSupaTable } from '@/components/grid/SupabaseGrid.utils'
import type { Filter, SupaTable } from '@/components/grid/types'
import { useConnectionStringForReadOps } from '@/data/read-replicas/replicas-query'
import { executeSql } from '@/data/sql/execute-sql-mutation'
import {
PG_META_SCOPED_INTROSPECTION_FLAG,
prefetchTableEditor,
} from '@/data/table-editor/table-editor-query'
import { useAsyncCheckPermissions } from '@/hooks/misc/useCheckPermissions'
import { RoleImpersonationState, wrapWithRoleImpersonation } from '@/lib/role-impersonation'
import { isRoleImpersonationEnabled } from '@/state/role-impersonation-state'
import { ResponseError, UseCustomQueryOptions } from '@/types'
export type GetTableRowsCountArgs = {
table?: SupaTable
filters?: Filter[]
enforceExactCount?: boolean
}
export type TableRowsCount = {
count?: number
is_estimate?: boolean
}
export type TableRowsCountVariables = Omit<GetTableRowsCountArgs, 'table'> & {
queryClient: QueryClient
tableId?: number
roleImpersonationState?: RoleImpersonationState
projectRef?: string
connectionString?: string | null
scoped?: boolean
}
export type TableRowsCountData = TableRowsCount
export type TableRowsCountError = ResponseError
export async function getTableRowsCount(
{
queryClient,
projectRef,
connectionString,
tableId,
filters,
roleImpersonationState,
enforceExactCount,
isReadOnlyContext = false,
scoped,
}: TableRowsCountVariables & { isReadOnlyContext?: boolean },
signal?: AbortSignal
) {
const entity = await prefetchTableEditor(queryClient, {
projectRef,
connectionString,
id: tableId,
scoped,
})
if (!entity) {
throw new Error('Table not found')
}
const table = parseSupaTable(entity)
const formattedFilters = filters?.map((x) => ({ ...x, value: formatFilterValue(table, x) }))
const sql = wrapWithRoleImpersonation(
getTableRowsCountSql({
table,
filters: formattedFilters,
enforceExactCount,
isReadOnlyContext,
scoped,
}),
roleImpersonationState
)
const { result } = await executeSql(
{
projectRef,
connectionString,
sql,
queryKey: ['table-rows-count', table.id],
isRoleImpersonationEnabled: isRoleImpersonationEnabled(roleImpersonationState?.role),
},
signal
)
return {
count: result?.[0]?.count,
is_estimate: result?.[0]?.is_estimate ?? false,
} as TableRowsCount
}
export const useTableRowsCountQuery = <TData = TableRowsCountData>(
{
projectRef,
tableId,
...args
}: Omit<TableRowsCountVariables, 'queryClient' | 'connectionString'>,
{
enabled = true,
...options
}: UseCustomQueryOptions<TableRowsCountData, TableRowsCountError, TData> = {}
) => {
const queryClient = useQueryClient()
const {
connectionString,
identifier: readReplicaIdentifier,
type,
} = useConnectionStringForReadOps()
const { can: canSQLAdminWrite, isLoading: isPermissionsLoading } = useAsyncCheckPermissions(
PermissionAction.TENANT_SQL_ADMIN_WRITE,
'tables'
)
const scoped = !!useFlag(PG_META_SCOPED_INTROSPECTION_FLAG)
return useQuery<TableRowsCountData, TableRowsCountError, TData>({
queryKey: tableRowKeys.tableRowsCount(projectRef, {
table: { id: tableId },
readReplicaIdentifier,
...args,
scoped,
}),
queryFn: ({ signal }) =>
getTableRowsCount(
{
queryClient,
projectRef,
connectionString,
tableId,
isReadOnlyContext: type === 'replica' || !canSQLAdminWrite,
...args,
scoped,
},
signal
),
enabled:
enabled &&
typeof projectRef !== 'undefined' &&
typeof tableId !== 'undefined' &&
(!IS_PLATFORM || typeof connectionString !== 'undefined') &&
// isReadOnlyContext resolves to `type === 'replica' || !canSQLAdminWrite`: for
// read replicas it's already known synchronously, but otherwise it depends on
// canSQLAdminWrite, which starts out `false` while permissions are loading.
// Firing while that's still in flight would cache a transient
// isReadOnlyContext:true (and, on a never-analyzed table, a scoped
// count:-1/is_estimate:true) for what may actually be a writable user. Wait
// for the permission check to settle before firing in that case.
(type === 'replica' || !isPermissionsLoading),
...options,
})
}