mirror of
https://github.com/supabase/supabase.git
synced 2026-09-06 18:11:51 +08:00
6c6a721cb7
## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Bug fix (performance), follow-up to #47894, plus regression-guard tests. ## What is the current behavior? #47894 scoped the Table Editor and entity-definition introspection queries, but four more `@supabase/pg-meta` query families still do O(catalog) work per request. On a production project with a very large catalog (hundreds of schemas, ~465K `pg_constraint` rows) they run 5 to 55 seconds each, trip the 58s `statement_timeout`, and spill sorts to temp files. During a recent "DB CPU > 85%" incident on such a project, 24 of 27 active backends were running these queries concurrently. 1. **`tables.retrieve()` (single-table lookup by name+schema or id)**: the `tables`/`columns` CTEs scan the whole catalog (`pg_class`, `pg_constraint`, `pg_index`, all of `pg_attribute`, per-table sizes) and the one-table predicate is applied only on the outer select. Same bug class #47894 fixed for the OID-based table editor query; this sibling path never got the treatment. It accounted for 94 of the 96 statement-timeout cancellations in the incident. 2. **Types listing**: the `t_enums` and `t_attributes` subqueries aggregate the entire `pg_enum` and every composite relation before the wrapper's schema filter applies. 3. **Table privileges**: `aclexplode` + double `pg_roles` join + GROUP BY over every relation in the database; schema/OID filters applied only after aggregation, in both `list()` and `retrieve()`. 4. **Row counts**: `getTableRowsCountSql` treats `reltuples = -1` (never-analyzed table) as "small table, run exact count(*)". A freshly bulk-loaded multi-million-row table times out on every Table Editor pagination render. Two Studio-side amplifiers turned one slow query into a sustained load storm: - `useTableQuery` (behind `tables.retrieve()`) mounts once per visible foreign-key grid cell via `ForeignKeyFormatter`, so a single Table Editor view fires ~20 concurrent copies against the FK target table. A timed-out query caches nothing, and TanStack retries errored no-data queries on every observer mount by default, so scrolling kept re-issuing the 58s scan. - `useTableApiAccessQuery` fetched table privileges for the entire database and filtered down to one schema client-side. ## What is the new behavior? **pg-meta (all behind the existing `pgMetaScopedIntrospection` flag, same rollout mechanism as #47894; `scoped: false` keeps serving the current SQL):** - `tables.retrieve()`: the identifier is resolved to a scalar `targetOid` init-plan and pushed into the base scan, primary-key, relationships (both FK directions kept: `conrelid` or `confrelid`) and columns CTEs. A materialized `target` CTE was deliberately avoided: it acts as an optimization barrier and forces the very seq scans being removed. - Types: filter `pg_type`/`pg_namespace` first, then compute enums/attributes per surviving row via correlated index-scan subqueries (`pg_enum(enumtypid, enumsortorder)`, `pg_attribute(attrelid, attnum)`). - Table privileges: schema/OID predicates injected into the base WHERE before `aclexplode`/GROUP BY for `list()` and `retrieve()`. - Row counts: `reltuples = -1` is treated as "unknown" and gated on physical size via `pg_relation_size` (a cheap stat call; `relpages` is equally stale pre-vacuum). At or below `THRESHOLD_ESTIMATE_BYTES` (~10MB, derived from `THRESHOLD_COUNT` at a conservative ~200 bytes/row) the exact count runs as before: fast by construction, and it avoids bogus estimates since Postgres floors never-vacuumed heaps at 10 pages, so an empty table would otherwise report ~2K estimated rows. Above the gate the count routes through the EXPLAIN-based `pg_temp.count_estimate`, or returns `-1`/`is_estimate = true` in read-only contexts where the temp function cannot be created. The scoped branch embeds the estimated select via `literal()` instead of legacy's apostrophe-only escaping, so it stays correct under `standard_conforming_strings = off`. `enforceExactCount` unchanged. **Studio:** - The flag decision is contained in the data layer instead of prop-drilled: a small imperative accessor (`apps/studio/data/scoped-introspection.ts`) is hydrated from `useFlag` via a one-line `useSyncScopedIntrospection()` call in `DefaultLayout`, and the query functions read it internally when building the pg-meta SQL. `DefaultLayout` is shared by both the Next and TanStack router trees; hydrating from `_app.tsx` alone would leave TanStack-served pages permanently unscoped since `routes/__root.tsx` mounts its own flag provider. Cold loads cannot race the flag: the query functions await a readiness promise that resolves only after the sync hook has hydrated the accessor with a loaded flag store (immediately on self-hosted where flags are disabled; a 5s safety net armed lazily on the first `ready()` call - not at module import, which would let the timer expire before a project page ever mounts - bounds genuine ConfigCat outages). No component threading, no query-key changes (remaining tradeoff, documented in the module: a mid-session flag flip can serve stale-keyed caches until refetch, fine for a session-stable rollout flag). #47894's existing threading is left as-is and gets deleted together with the flag in the cleanup PR. Also fixes the previously-missing `scoped` pass-through in `getTableRowsCount`. - Flag-independent hardening: `useTableQuery` now sets `retryOnMount: false`, `refetchOnWindowFocus: false` and `staleTime: 5min`. Errored (timed-out) queries no longer refire on every grid cell remount, while stale successful metadata still revalidates on mount after `staleTime`. - `useTableApiAccessQuery` now passes `includedSchemas: [schemaName]`; the client-side filter stays as a safety net. - The rows-count query is `enabled`-gated on the permission check settling, so a transiently-false `canSQLAdminWrite` can no longer cache a read-only `-1` count for a writable user (read replicas short-circuit synchronously as before). **Regression guards (extending the #47894 infrastructure):** - Execution-based scoped-vs-legacy equivalence tests for all four queries: both variants run against the test database and are compared with raw `toEqual` - no normalization, ids included (types across 6 option combos, privileges incl. multi-grantee + PUBLIC, `tables.retrieve` for both identifier branches, row counts for every case where the two paths must agree). Two documented exceptions where only the LEGACY side is sorted, because a de-normalized diagnostic run proved legacy emits genuinely plan-dependent order there (an adversarial-FK fixture shows it is neither oid, name, nor creation order): the `types.list` outer row order (scoped adds `order by t.oid`; legacy has no ORDER BY) and the `tables.retrieve` relationships array (scoped orders by `constraint_name` + column-name tie-breakers - a composite two-column FK expands to 4 entries sharing one constraint_name). Everything else (privileges via `aclexplode` over the same relacl, columns by `ordinal_position`, primary keys by `indkey` order, enums by `enumsortorder`) is byte-identical between the two paths with no test-side help. The one intentional value divergence, never-analyzed tables above the size gate where legacy's exact count is the timeout bug itself, is asserted explicitly as a divergence. - Plan-guard budgets for every scoped query against the stress catalog (extended with 200 enums + 200 composite types). Residual seq scans are justified in-budget: `pg_constraint` max 2 (no index on `confrelid`), `pg_attrdef` max 1, `pg_authid` max 2 (scales with role count, not schema count). - Legacy templates carry a FROZEN do-not-edit marker (they must keep matching production behavior until the flag cleanup deletes them); the ordinary test suite runs against the legacy default, so behavioral drift there fails regular tests. ### Validation - pg-meta: typecheck clean; the affected suites (types, table-privileges, tables, rows-count, catalog-plan-guard) pass in full. - Cross-version: the scoped-vs-legacy equivalence and rows-count behavioral suites were validated on PostgreSQL 14, 15, and 17 (identical results on all three). Two version-marginal planner choices surfaced on 17 (`pg_type` / `pg_class` seq scan vs full-index bitmap for per-schema listings, both structurally unavoidable without an index leading on the namespace column) and are carried as justified plan-guard budget entries. A full 468-test suite run sequentially: 452 passed, 16 failures verified environmental (13 timeouts in an untouched file that passes 27/27 in isolation on the marathon-run cluster, 3 cluster-global role collisions from container reuse). - Studio: `pnpm --filter studio typecheck` clean; 39/39 tests across the touched data hooks; eslint clean on touched files. ### Rollout Same staged ConfigCat rollout as #47894 via `pgMetaScopedIntrospection` (user-email targeting first, then percentage, then 100%). The `useTableQuery` hardening and the API-access schema scoping ship unflagged (behavior-safe). Gate before percentage rollout: functionally verify the FK popover/selector UX under the new `staleTime`/`retryOnMount` settings (a just-edited FK target must not look stale anywhere Studio does not already refetch on save). Once fully rolled out, the legacy templates and flag get deleted together with #47894's in one cleanup PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>