mirror of
https://github.com/supabase/supabase.git
synced 2026-09-06 18:11:51 +08:00
create-pull-request/patch
32 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Charis
|
4dee589735 |
fix(studio): stop assistant fabricating database_identifier for notebooks (#49558)
## Summary * Fixes [FE-4275](https://linear.app/supabase/issue/FE-4275/assistant-always-creates-notebooks-with-wrong-identifier-first-try): the assistant always created database notebook cells with a fabricated `database_identifier` (`"primary"`, later observed as `""` / `"_primary"` under different prompt wording) instead of omitting the key for the project's primary database, which tripped the tool's reject-and-retry validation on the very first attempt. * Prompt wording alone wasn't reliable — live eval runs against the real model kept substituting a new placeholder every time the prompt was tightened further. * Normalizes an empty-string `database_identifier` to absent at the schema level (`databaseIdentifierSchema` in `notebook-schema.ts`), which is inherited by every schema built from it — the AI SDK's `inputSchema` for `create_notebook`/`update_notebook`, and the write-boundary `writableNotebookSchema` used right before the PUT to the backend. * Adds an eval case (`evals/dataset.ts`) reproducing the original bug, plus unit tests covering schema-level and write-boundary normalization. ## Test plan - [X] `pnpm --filter studio exec tsc --noEmit` passes - [X] `pnpm exec prettier --check` passes on touched files - [X] Unit tests pass: `notebook-schema.test.ts`, `notebook-upsert-mutation.test.ts`, `notebook-tools.test.ts` (104 tests) - [X] Ran the new eval case against the real model 3x before the code fix (0% correctness, fabricated `""`/`"_primary"`) and 3x after (100% correctness) ## Summary by CodeRabbit * **Bug Fixes** * Improved notebook handling of empty database identifiers by treating them as absent. * Ensured notebook requests omit unused database identifier fields. * Added validation guidance for read-replica database identifiers. |
||
|
Charis
|
89b4f1aca4 |
feat(studio): add delete_notebook tool to AI assistant (#49413)
## Summary * Adds a `delete_notebook` AI assistant tool (`needsApproval: true`) that lets the assistant delete a notebook with explicit user approval, mirroring the existing `create_notebook`/`update_notebook` tools. * Wires up a destructive-styled approval card in the AI Assistant Panel (fetches the notebook to show its name, warns the deletion is permanent) using the same `Confirm`/tool-approval plumbing as the other notebook tools. * Updates `tool-filter.ts` opt-in gating, the assistant system prompt, the eval-harness mock tools, and the eval dataset with `delete_notebook` coverage. * Adds test coverage in `notebook-tools.test.ts`, `mock-tools.test.ts`, and `NotebookProposalRenderer.test.tsx`. Closes [FE-4242](https://linear.app/supabase/issue/FE-4242/assistant-delete-notebook-tool). ## Test plan - [X] `pnpm typecheck --filter=studio` passes - [X] `pnpm --filter studio exec vitest run` for the touched files (notebook-tools, mock-tools, NotebookProposalRenderer, [Message.Parts](<http://Message.Parts>), and existing consumers of `content-delete-mutation`) — all passing - [X] `eslint` and `prettier --check` clean on all touched files - [X] Manual verification of the approval UI in a running Studio instance (not done in this session) ## Summary by CodeRabbit * **New Features** * Added AI-assisted notebook deletion with explicit confirmation and irreversible-action warnings. * Added safeguards to distinguish deleting an entire notebook from removing individual panels. * Completed deletions now display the deleted notebook’s name without an option to reopen it. * **Bug Fixes** * Improved handling of missing notebooks and invalid deletion requests. * **Tests** * Added coverage for deletion approval, denial, errors, and successful completion. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added AI-assisted notebook deletion with explicit approval and irreversible-action warnings. * Added confirmation, loading, error, and completion states for notebook deletion. * Prevented accidental full-notebook deletion when only a panel or section should be removed. * Improved notebook update results by showing applied changes when available. * **Bug Fixes** * Notebook deletion now uses the required API version. * Improved handling and validation of missing notebooks during deletion. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Joshen Lim <joshenlimek@gmail.com> |
||
|
Charis
|
c172195269 |
test(studio): add eval cases for list_databases-driven notebook creation (#49398)
## Summary - Adds three eval cases exercising the behavior this stack wires up: a happy path where the model calls `list_databases` before targeting a named read replica, a guard against fabricating an identifier when the user names a region/replica `list_databases` doesn't actually return, and a guard against targeting a non-primary database when the user never asked for one. Part 6/6 (final) of the stack for FE-4225 (expose valid database identifiers to the notebook AI agent). Stacked on #49334. ## Test plan - [x] Ran all three cases against the real model; inspected transcripts directly - [x] Re-verified reworded `correctAnswer` text against real outputs via the correctness evaluator - [x] `pnpm --filter studio exec tsc --noEmit` passes - [x] `prettier --check` passes <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added evaluation coverage for selecting the correct database replica when creating notebooks. * Verified primary-database defaults when no database is specified. * Added checks to prevent fabricated database identifiers when a requested replica is unavailable. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Charis
|
7a2e237892 |
Add update_notebook evals; fix prompt gaps they surfaced (#49324)
## Summary - Add `update_notebook` eval cases (insert/replace/delete/move, a combined delete+insert, and guard/safety cases) mirroring the existing `create_notebook` cases, targeting the notebooks already seeded in the mock tool harness. - Fix two behavior gaps in `NOTEBOOKS_PROMPT`/`LIMITATIONS_PROMPT` that these cases surfaced when run live: the assistant asking the user for a notebook id instead of resolving it via `list_notebooks`, and the destructive-operations warning rule not being connected to SQL written into notebook cells. - Soften the destructive-SQL case's `correctAnswer` to match `update_notebook`'s real approval-gated behavior — a warning accompanying the reported change is acceptable, not only one strictly preceding the tool call. ## Test plan - [x] `pnpm run typecheck` (apps/studio) — clean - [x] `pnpm exec prettier --check` on both changed files — clean - [x] `evals/scorer.test.ts`, `evals/transcript.test.ts`, `evals/trace-utils.test.ts` — 21/21 pass - [x] Ran the new eval cases live against OpenAI (bypassing the Braintrust proxy) via Braintrust MCP; confirmed via trace inspection that the prompt fix resolved the id-resolution gap (Tool Usage 0% → 100% across 3 trials) and that the assistant now includes an explicit irreversibility warning when destructive SQL is written into a notebook cell <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Improved notebook creation and editing support across SQL, query, chart, and time-range cells. - Added clearer handling for saved notebooks, recurring requests, and one-time SQL execution. - Enhanced validation for database cells and notebook configuration. - **Bug Fixes** - Improved safeguards and warnings for destructive queries, including saved notebook queries. - Better handling of missing tables and notebooks. - More precise notebook cell updates and tool usage validation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Saxon Fletcher
|
aa2897f712 |
feat(studio): teach assistant to query ClickHouse logs (#49292)
## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Feature and bug fix. ## What is the current behavior? The assistant can call `query_logs`, but it is not given the ClickHouse schema and query-writing guidance it needs. It also lacks a current UTC reference for producing the absolute timestamps required by the tool, which can lead to valid queries being run against the wrong time range and reported as returning zero rows. ## What is the new behavior? - Adds a dedicated `logs` knowledge topic backed by the shared ClickHouse schema and query guidance. - Requires the assistant to load that knowledge before using `query_logs`. - Includes the current UTC time in project context so relative requests can be converted to correct absolute tool parameters. - Covers the new knowledge flow and context with focused tests and updates the assistant eval expectation. ## How to test 1. Check out this PR and run Studio against a project that has recent logs. Generate some project activity first, such as an API request, if needed. 2. Open the AI Assistant and ask: `Show log counts by minute for the last 15 minutes and summarize any spikes.` 3. Expand the assistant's tool activity and verify it loads the `logs` knowledge topic before calling `query_logs`. 4. Inspect the `query_logs` input and verify: - `iso_timestamp_start` and `iso_timestamp_end` are absolute UTC timestamps ending in `Z`. - The timestamps cover approximately the requested 15-minute window. - The SQL uses ClickHouse syntax, includes a `LIMIT`, and does not put the time range in the SQL `WHERE` clause. 5. Verify the assistant's summary reflects the rows returned by `query_logs` instead of reporting zero rows when results are present. ## Additional context This is the bottom PR in stack #49294. The front-end visualization is added separately in #49293. Verified with 59 focused tests across assistant context, Studio/MCP tools, query display, and logs result parsing. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added AI-assisted project log querying through the `query_logs` tool. - Added logs knowledge guidance for time ranges, schema discovery, query limits, and concise result summaries. - Project context now includes the current UTC timestamp to improve relative time-range interpretation. - Improved notebook assistance with safer table verification and appropriate handling of log queries. - **Bug Fixes** - Prevented incorrect SQL timestamp filtering and enabled cross-service searches without requiring a source filter. - Added validation for supported knowledge topics. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Charis
|
e7c3cad8de |
feat(evals): add notebook eval cases and forbiddenTools scorer capability (#49104)
## Summary - Added ~11 new eval cases for Notebooks AI assistant evals, including: basic notebook cell creation, multi-cell composition (markdown+database+log), log cell time ranges, row_limit defaults, chart config, destructive SQL safety warnings, and hallucination guards for nonexistent tables - Extended `toolUsageScorer` with deterministic `forbiddenTools` field to score both required and forbidden tool usage, enabling eval cases to assert tool choice (e.g., `execute_sql` vs `create_notebook`) without LLM-as-judge - Extended SQL validators (`sqlSyntaxScorer`/`sqlIdentifierQuotingScorer`) to validate SQL inside `create_notebook` database cells (log cells deliberately excluded as they use ClickHouse dialect) - Fixed two real assistant issues in `NOTEBOOKS_PROMPT`: (a) reuse `CLICKHOUSE_LOGS_COMPLETION_INSTRUCTIONS` and schema section to prevent incorrect BigQuery-style SQL in logs queries, (b) require schema verification before writing `database_cell` to prevent queries against nonexistent tables Resolves FE-4087 ## Test plan - All 11 new eval cases run live against OpenAI via Braintrust; traces inspected and validated - Existing unit tests, lint, and typecheck pass <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for creating notebooks with database, Markdown, chart, and log cells. * Improved handling of reusable notebook requests versus one-off SQL queries. * Added guidance for modern ClickHouse SQL and absolute log time ranges. * **Bug Fixes** * Improved SQL validation, row-limit enforcement, and destructive-query safety. * Prevented invalid or nonexistent-table queries from being accepted. * Improved validation of notebook cell types and tool usage. * Improved handling of ClickHouse log queries and database-cell SQL. * Improved evaluation reliability by limiting concurrent test execution. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Charis
|
8c409e2df5 |
Fix eval scorer truncation via local transcript capture (#49151)
## Problem Scorers previously derived the assistant's final answer via Braintrust's `trace.getThread()`, which silently truncates long traces at the backend's preview-length cap (~10KB). The SDK never passes `preview_length` in its BTQL query and there's no supported override. This caused false-negative scores (Completeness, Correctness, Goal Completion, Safety collapsing to 0/null) specifically on multi-step tool-calling eval cases, since longer traces are more likely to have their tail (the final assistant message) truncated away. ## Solution Capture the assistant's full, untruncated final answer directly in the eval task's output in memory (via AI SDK's `result.steps`, already fully available once the stream is consumed) instead of round-tripping through Braintrust's truncating storage/query layer. Scorers now read `output.transcript` instead of calling `trace.getThread()`. ## Changes - **New**: `apps/studio/evals/transcript.ts` — `Transcript` type and `buildTranscript()` function - **New**: `apps/studio/evals/transcript.test.ts` — unit tests (5 passing) - **Modified**: `apps/studio/evals/assistant.eval.ts` — captures `result.steps` and returns transcript - **Modified**: `apps/studio/evals/scorer.ts` — migrated 7 scorers to read from local transcript - **Modified**: `apps/studio/evals/trace-utils.ts` — removed dead thread-serialization code - **Deleted**: `apps/studio/evals/trace-utils.test.ts` — superseded by transcript tests ## Test Plan - [x] `pnpm --filter studio typecheck` — clean - [x] `pnpm --filter studio lint` — clean - [x] `npx vitest run evals/transcript.test.ts` — 5/5 passing - [x] Full live eval run (35/35 cases) against Braintrust — [experiment](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/eval-scorer-transcript-capture-1786985352) shows Completeness/Correctness/Goal Completion/Safety scores comparable to baseline ## Known Residual Risk Other scorers that derive data from `trace.getSpans()` (toolUsageScorer, sqlSyntaxScorer, sqlIdentifierQuotingScorer, knowledgeUsageScorer, and docsFaithfulnessScorer's docs-content lookup) could theoretically hit the same truncation issue, but have not been observed to fail in practice. This is not addressed in this PR. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added transcript generation from assistant interaction steps, including text and tool-call inputs. * Evaluation results can now include complete transcripts for detailed conversation analysis. * Online evaluations can derive transcripts from recorded interaction traces when needed. * **Bug Fixes** * Improved scoring by selecting the appropriate conversation content for each evaluation. * Ensured offline transcripts take precedence when available, with trace-based fallback support. * **Tests** * Added coverage for multi-step interactions, tool calls, filtering, empty steps, and URL validation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Charis
|
794e45378c |
test(studio): add get_notebook eval cases (FE-4088) (#49067)
## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Feature (test coverage) — second PR of the notebooks-evals plan, covering FE-4088 (Evals: Assistant can read notebook). Follows #49010 (FE-4086, list_notebooks). ## What is the current behavior? `dataset.ts` has no coverage for `get_notebook`. Separately, `NOTEBOOKS_PROMPT` only covers *choosing* between `create_notebook`, `update_notebook`, and `execute_sql` — it says nothing about reading or describing an existing notebook. ## What is the new behavior? **Eval cases** (`evals/dataset.ts`), three of them: - Resolve a notebook by name via `list_notebooks`, then `get_notebook`, and report the queries it actually contains. - Summarize a smaller, single-log-cell notebook as a baseline. - Report a nonexistent notebook id as not found instead of hallucinating contents. The mock's `execute` throws, which the AI SDK surfaces to the model as a `tool-error` part rather than failing the eval task. No new scorers or tool changes — `toolUsageScorer` and `correctnessScorer` already cover these, and `get_notebook` has unit coverage in `notebook-tools.test.ts`. **Prompt change** (`lib/ai/prompts.ts`) — please review this one separately, it's the only production behavior change here: Running the first case surfaced a real gap. The assistant transcribed each cell's SQL correctly but was inconsistent about the configuration that changes what a cell returns — dropping the log cell's time range in some runs, miscounting markdown cells as queries in others. Adds one bullet to `NOTEBOOKS_PROMPT` telling it to report a query cell's configuration and not count markdown cells as queries. Gated behind the Explorer flag, same as the rest of that prompt. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added notebook evaluation scenarios for database and log queries, including concise summaries and nonexistent notebook handling. * Improved notebook descriptions by reporting query configurations that affect returned results. * Markdown cells are now excluded from query counts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Charis
|
8baaa517d0 |
test(studio): add list_notebooks eval cases (FE-4086) (#49010)
## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Feature (test coverage) — first PR of the notebooks-evals plan, covering FE-4086 (Evals: Assistant can list notebooks). ## What is the current behavior? `dataset.ts` has zero notebook eval cases. Separately, two small gaps block writing them: `assistant.eval.ts` never sets `isExplorerEnabled`, so `NOTEBOOKS_PROMPT` never loads into the eval task's system prompt; and `list_notebooks`' `inputSchema` has no sort parameter, even though `getContent` already supports one, so "most recent" isn't answerable. ## What is the new behavior? - `assistant.eval.ts` passes `isExplorerEnabled: true` so `NOTEBOOKS_PROMPT` loads during evals. - `list_notebooks` gains a `sort_by: 'name' | 'inserted_at'` param, forwarded to `getContent`'s `sort`. Only creation order is exposed, since the underlying API has no `updated_at` sort key. - 3 new dataset cases: basic notebook enumeration, sorting by creation time (`sort_by`/`limit` args), and a nonexistent-notebook case guarding against hallucinated results. - A unit test covering `sort_by` forwarding to the content API's query param. ## Additional context <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added notebook sorting options when listing notebooks, including by name or insertion date. * Added evaluation coverage for listing notebooks, finding the newest notebook, and avoiding fabricated results for nonexistent notebooks. * **Bug Fixes** * Ensured selected notebook sorting preferences are correctly applied when retrieving content. * Improved assistant evaluation coverage with Explorer mode enabled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Pedro Rodrigues
|
47595f8ac7 |
feat(self-hosted): implement queryLogs for the MCP debugging tools (#48900)
> [!IMPORTANT] > > Only merge this when (https://github.com/supabase/platform/pull/36804) is merged, as the AI assistant will not have access to the `query_logs` tool for the remote MCP server ## I have read the [CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md) file. YES ## What kind of change does this PR introduce? Feature (self-hosted / CLI Studio MCP server). ## What is the current behavior? Self-hosted `getDebuggingOperations` (`apps/studio/lib/api/self-hosted/mcp.ts`) implements only `getLogs`, so the MCP `debugging` group exposes `get_logs` — a fixed per-service log dump built by `getLogQuery`. Logs are served by Logflare, which speaks BigQuery SQL. ## What is the new behavior? Bumps `@supabase/mcp-server-supabase` to `^0.10.0` (adds `query_logs` + `logsDialect`, and hides `get_logs` wherever a platform declares `queryLogs`) and moves logs over to it. - **Self-hosted `query_logs`:** declares `logsDialect: 'bigquery'` and implements `queryLogs`, passing the model's SQL straight through to the same Logflare `logs.all` endpoint (arbitrary `sql` param) — no new endpoint, no dialect translation. - **Drops `get_logs` from self-hosted:** `getLogs` throws (the server hides it once `queryLogs` exists) and the per-service `getLogQuery` builder is deleted; the model now writes its own BigQuery SQL, guided by the dialect schema hint. - **Honors no-logs mode:** `query_logs` throws when `logs:all` is disabled — the self-hosted default, enabled via the `docker-compose.logs.yml` override. - **Assistant:** switches the dashboard assistant from `get_logs` to `query_logs` (allowlist, drift guard, prompt, mocks, evals). Refs AI-1046 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI debugging can query recent project logs using read-only SQL. * Log queries support optional time-range filters, filtering, aggregation, and joins. * Self-hosted debugging checks whether logging is enabled before running queries. * **Bug Fixes** * Updated debugging workflows and validation to consistently use the new log-query capability. * Removed reliance on legacy service-specific log filtering and query behavior. * **Documentation** * Updated MCP debugging tool guidance to describe SQL-based log queries. * **Tests** * Expanded coverage for enabled, disabled, and unsupported logging scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Pedro Rodrigues
|
c4c213ce3d |
feat(studio): switch dashboard assistant to remote MCP server (#47479)
## I have read the [CONTRIBUTING.md](<https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md>) file. YES ## What kind of change does this PR introduce? Feature / refactor. ## What is the current behavior? The dashboard assistant runs `@supabase/mcp-server-supabase` in-process over an in-memory transport (`lib/ai/supabase-mcp.ts`). ## What is the new behavior? The assistant connects to the **remote MCP server** over HTTP (`@ai-sdk/mcp`), forwarding the dashboard session token as a bearer. URL comes from `NEXT_PUBLIC_MCP_URL` with a local-dev fallback; platform-only, and Nimbus works via the same env var. * **Tool model unchanged:** UI-controlled `execute_sql` (with `needsApproval`) and `deploy_edge_function` still come from Studio; the allowlist (`TOOL_CATEGORY_MAP`) remains the gate keeping the remote's write tools away from the assistant (`read_only` is defense-in-depth). * **Attribution:** sends `x-source-name: supabase-studio` (+ `x-source-version`) → logged as `source_name`/`client_name`. * **Connection lifecycle:** the HTTP client is closed via the request's `AbortSignal` (tools execute later during streaming); `signal` is required on `getTools`/`getMcpTools`. * **Resilience:** a remote-MCP failure degrades to the remaining tools instead of failing the assistant. * **Drift protection:** relied-upon tools are typed against `keyof typeof supabaseMcpToolSchemas`, so a package bump that renames/removes one fails `pnpm typecheck`; a runtime check also warns if the deployed server returns fewer tools. * Adds unit tests for the above. ## Additional context * Verified end-to-end against a local remote MCP server with a dashboard token: `initialize` 200, tools listed, a tool executed, client closed cleanly. * The remote MCP (mgmt-api) already accepts dashboard session tokens (GoTrue-JWT auth path) — no backend change needed. `NEXT_PUBLIC_MCP_URL` must point at each env's `/mcp`. * `@supabase/mcp-server-supabase` is kept — still used by the self-hosted `/api/mcp` routes. Closes [AI-137](https://linear.app/supabase/issue/AI-137/switch-dashboard-assistant-to-remote-mcp) ## Rollout * **Rollout:** merges with `USE_REMOTE_MCP` off (in-process); flip it to `true` per environment (staging → prod → Nimbus) once each one's prerequisites land. * **Rollback:** unset `USE_REMOTE_MCP` and redeploy to fall back to the in-process client — no revert needed. ## Summary by CodeRabbit * **Bug Fixes** * Improved AI request handling so tool loading and generation clean up properly when a request is cancelled or the browser connection closes. * Added safer fallback behavior when remote tool loading fails, so AI features can continue with available tools instead of stopping entirely. * Updated remote tool access to use the current project reference and preserve the correct access headers. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * AI tools now connect more reliably to remote services and stop cleanly when requests end or are canceled. * Tool loading is more resilient, continuing with available tools if remote access is unavailable. * **Bug Fixes** * Improved cleanup to prevent lingering connections during SQL generation and policy workflows. * Added safer handling for remote tool changes and invalid responses. * **Tests** * Expanded automated coverage for remote tool setup, cancellation, and fallback behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Matt Rossman
|
65fab30935 |
feat(ai): judge tool inputs, add storage guidance and permissive RLS evals (#46168)
Adding broad RLS policies to public buckets can cause users to expose more than they expected, like the ability to list all profile pictures on an app. This patches Assistant with knowledge to follow our latest guidance on restrictive RLS policies for storage buckets https://github.com/supabase/supabase/pull/46172 **Changes** - Adds Storage bucket evals for public website assets and avatar access patterns to distinguish public vs private bucket use cases - Adds eval for overly permissive table policies - Adds `storage` knowledge so Assistant distinguishes public buckets, private buckets, object reads, and object listing. - Adds `includeToolCallInputs` option for scorer transcripts so LLM judges can evaluate proposed SQL/tool actions. - Bumps max step count to 10 since storage knowledge may incur another tool call (also 10 is recommended [here](https://vercel.com/academy/ai-sdk/multi-step-and-generative-ui#why-multi-step-is-required) for complex multi-tool scenarios) **References** - https://supabase.com/docs/guides/storage/buckets/fundamentals#public-buckets - https://supabase.com/docs/guides/storage/security/access-control - https://github.com/supabase/supabase/pull/46172 **Notes:** - These prompt tweaks are not meant to be exhaustive fixes, they are mainly hotfixes intended to hold us out until these cases can be addressed more deeply in skills/docs and tracked in a central evals Closes AI-676 Closes AI-756 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Storage knowledge resource for the assistant covering Supabase Storage access patterns and RLS guidance. * Added three evaluation cases: two for Storage (marketing assets, avatars) and one for RLS policy generation for user profiles. * **Improvements** * Evaluators now include tool call inputs when judging conversations. * Assistant prompts and generation enhanced with richer Storage/RLS guidance and extended streaming limits. * **Tests** * Added test ensuring tool call inputs are included in serialized thread context. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/46168?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
212ccf8135 |
fix(ai): contextualize cron schedule as SQL writes, score in "Tool Usage" (#45997)
When Assistant tries to schedule crons in read-only mode, it succeeds but creates the jobs under the `supabase_read_only_user`. This causes permission errors when user try to delete or unschedule them from the Cron dashboard. The root fix will be to enforce read-only transactions for that user. In the meantime, this PR steers Assistant to avoid the mistake. **Changes** - Prompts `execute_sql` to treat side-effecting function calls such as `cron.schedule()` as write queries. - Adds tool input assertions for "Tool Usage" scorer and a focused cron regression eval. - Updates eval mocks to show pg_cron extension as installed so it can call `cron.schedule()` **Verification** See [this trace](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=4a9e8c0e-83b7-4555-8502-365662c3ec8e&r=e041e69b-b70f-41d1-b88c-e8f7888c3de5&s=e041e69b-b70f-41d1-b88c-e8f7888c3de5) from Braintrust where the new eval passes "Tool Usage", correctly using `isWriteQuery` for the `cron.schedule()` Closes AI-737 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool evaluation now validates tool inputs (including exact and substring matches) in addition to tool presence. * **Tests** * Added a test confirming cron-scheduling behavior and that SQL scheduling/enqueue calls are treated as write operations. * **Chores** * Added pg_cron to mock extension data. * Clarified description that SQL calls with side effects should be treated as writes. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/45997?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
36152cb7fe |
docs(ai): assistant evals development workflow (#45840)
Adds `README.md` to `apps/studio/evals` explaining the development workflow for updating offline and online evals for Studio's AI Assistant. Resolves AI-681 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added comprehensive documentation for Studio Assistant Evals, covering evaluation setup, configuration of scoring methods, and deployment workflows for both offline and online evaluation processes. [](https://app.coderabbit.ai/change-stack/supabase/supabase/pull/45840) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
d143571586 |
feat(assistant): trace-level scorers + server-side tool execution with needsApproval (#45654)
## Motivation When Assistant runs a potentially destructive tool like `execute_sql`, it stops the LLM request and prompts for client-side approval and execution of the tool. After approval, a second request kicks off under a separate trace. This has made scoring and [Topics](https://www.braintrust.dev/blog/topics) classification challenging, as the generated `output` is split across stateless requests. The [span-level scoring](https://www.braintrust.dev/docs/evaluate/custom-code#score-spans) approach we've used thusfar (after the LLM call, we massage the result into an `output` payload that's stuck onto the root span) has been cumbersome and led to invalid scores / topics where only part of the assistant response is considered. It's also inefficient, as we're duplicating potentially large info (like the `search_docs` output) that already exists within the trace. An alternative to scoring spans is to [score traces](https://www.braintrust.dev/docs/evaluate/custom-code#score-traces). Braintrust [best practices](https://www.braintrust.dev/docs/evaluate/score-online#best-practices) advise: > Use span scope for evaluating individual operations or outputs. Use trace scope for evaluating multi-turn conversations, overall workflow completion, or when your scorer needs access to the full execution context. We've also received [direct guidance](https://supabase.slack.com/archives/C05QYJBLX89/p1777925770927149?thread_ts=1777905716.911979&cid=C05QYJBLX89) from their team to use this approach. ## Changes Migrates eval scorers from custom `AssistantEvalOutput` shape to trace-level scoring via `trace.getThread()` / `trace.getSpans()`, with thread parsing that scores the full latest Assistant turn and passes prior conversation separately where relevant. Moves `execute_sql` and `deploy_edge_function` from client-side execution after approval to AI SDK `needsApproval` + server-side `execute()`. SQL results returned to the model are gated by AI opt-in level, so row data is only included with `schema_and_log_and_data`; otherwise the tool returns the no-data-permissions sentinel. Adds `metadata.isFinalStep` to disambiguate multiple LLM requests within an "assistant" turn due to tool call requests/responses. For online evals, this means we should configure automations to only score traces with `metadata.isFinalStep = true` to ensure we're judging the complete generated response. Other minor kaizen changes: - Renamed `promptProviderOptions` to `systemProviderOptions` to clarify that this is associated with the "system" message and disambiguate from the root `providerOptions` - Adds `evals/trace-utils.ts` to handle Zod validation of the `unknown` span shapes from Braintrust, to more easily access typed inputs/output on tool spans. - Bumps AI SDK floor version `^6.0.116` → `^6.0.174` - Tweaked the "Conciseness" scorer to not unfairly dock points for the new `[called tool_name]` labels in serialized assistant response ## Verification In the studio staging build, I asked Assistant to create a todos table with 3 sample todos. I manually approved the `execute_sql` call and saw Assistant generate text before & after the call. In Braintrust I verified two traces were produced (see [filtered logs](https://www.braintrust.dev/app/supabase.io/p/Assistant/logs?v=Staging&tvt=trace&search={%22filter%22:[{%22text%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22label%22:%22metadata.environment%2520%253D%2520%27staging%27%22,%22originType%22:%22btql%22},{%22text%22:%22%2560Chat%2520ID%2560%2520%253D%2520%25221cb2ac45-e5e7-458c-9da4-3bf6863b8842%2522%22,%22label%22:%22Chat%2520ID%2520equals%25201cb2ac45-e5e7-458c-9da4-3bf6863b8842%22,%22originType%22:%22form%22}]})), the first with `metadata.isFinalStep = false` and the second with `metadata.isFinalStep = true`. In the Braintrust staging scorers, I ran the preview Completeness scorer on the second trace and verified it sees the complete Assistant response including markers for tool calls ([link to trace](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)/trace?object_type=project_logs&object_id=b5214b62-ad1e-4929-9d5b-40b1daebe948&r=0ed0a4f8-8aff-4a34-bb1d-1df1d88a5070&s=ff9015f8-6bf7-4ab3-83a9-ca4e69e27e82)) <img width="1193" height="960" alt="CleanShot 2026-05-07 at 11 27 10@2x" src="https://github.com/user-attachments/assets/509d4858-c3a1-4068-986d-3aa4d5617d1a" /> I also tested the `deploy_edge_function` workflow and verified it still prompts for permission and warns on deployment of existing functions. **References** - https://www.braintrust.dev/docs/evaluate/custom-code#score-traces - https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling#tool-execution-approval Supercedes https://github.com/supabase/supabase/pull/45556 and https://github.com/supabase/supabase/pull/45339 Closes AI-473 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Tool actions (SQL execution, edge-function deploy) now require explicit user Approve/Deny before proceeding. * **Improvements** * Assistant pauses for approval responses before sending follow-ups, giving clearer control over risky actions. * Deploy/replace flows show confirmation and clearer replace warnings. * Evaluation/scoring updated to use richer trace data for more accurate assistant performance signals. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Greg Richardson
|
538f9e3e82 |
fix: prevent AI assistant from soliciting sensitive creds (#45692)
Adds prompt guardrails and evals to prevent the AI assistant from asking users to share sensitive data (API keys, `.env` contents, etc.) and to warn when credentials are shared. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Stronger safety behavior: assistant now refuses requests to share full environment files, asks for variable names only, and directs users to secure secret-management tooling. * Immediate warning and guidance if credentials or other sensitive values are pasted in chat, without repeating exposed secrets. * **Behavior** * Clarified evaluation rules so responses more consistently follow the new safety guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Greg Richardson
|
5f8906a20e |
fix: add destructive operation guardrails to AI assistant (#45194)
Prevents the AI assistant from helping with local git/filesystem operations, and adds explicit warnings before irreversible database operations (DROP TABLE, DELETE without WHERE, etc.). Adds a `safetyScorer` and eval cases to cover these behaviours. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a Safety metric to evaluations so assistant responses are scored for safe handling of destructive or risky requests * Assistant guidance updated to refuse destructive local VCS/filesystem actions and require clear warnings for irreversible database operations * **Tests** * Added evaluation cases covering safe refusals, clear warnings, and correct handling of destructive or risky prompts * **Chores** * Enabled Safety metric in online evaluation manifests/handlers <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Greg Richardson
|
e38ba624bc |
feat(ai): update rls knowledge for 'secure by default' (#45072)
Updates the RLS knowledge loaded by the dashboard AI assistant to explain the new secure-by-default functionality. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Clarified PostgreSQL/RLS guidance in Studio: tables are now "secure by default"—SQL-created tables aren’t exposed via the Data API unless explicit grants are given to anon/authenticated/service_role and RLS is enabled; added an “Exposing a Table to the Data API” workflow, strengthened RLS prerequisites in best practices, and improved troubleshooting/error-recovery guidance. * **Tests / Evaluations** * Added an evaluation case validating guidance for non-RLS tables requiring explicit grants and RLS policies. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Ali Waseem <waseema393@gmail.com> |
||
|
Matt Rossman
|
a325e86845 |
fix: only prefix scorer slugs on PR builds, not master deploys (#43578)
Cleanup task following https://github.com/supabase/supabase/pull/43194 I noticed the run of `braintrust-scorers-deploy.yml` included the branch prefix on scorers in Assistant. This is unnecessary since there's only one copy of scorers in the "Assistant" project, unlike "Assistant (Staging Scorers)" which uses prefixes to disambiguate branches. <img width="502" height="262" alt="CleanShot 2026-03-09 at 15 45 19@2x" src="https://github.com/user-attachments/assets/214ec1e8-5f40-411f-8d2a-71cc4a5fc294" /> This is a small housekeeping correction so scorers in the main "Assistant" project don't include branch prefixes, whereas scorers from PRs deployed to "Assistant (Staging Scorers)" remain prefixed. https://docs.github.com/en/actions/reference/variables-reference <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated CI deployment configuration for scorer branch/prefix handling to optimize behavior across different GitHub event types (PR vs. push/dispatch events). <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Charis
|
205cbe7d26 | chore(studio}: enforce import order, remove bare import specifiers (#44585) | ||
|
Matt Rossman
|
82deff37de |
feat(assistant): lazy load topic knowledge via load_knowledge tool (#44296)
Moves knowledge (RLS, Edge Functions, PostgreSQL best practices, Realtime) out of the static system prompt and into a `load_knowledge` tool the model calls on demand, reducing prompt bloat. This is a temporary stopgap until the [standard Supabase agent-skills](https://github.com/supabase/agent-skills) are ready for integration in Assistant. - New always-available `load_knowledge` tool added to `rendering-tools.ts` - Updated `Message.Parts.tsx` so the "Ran load_knowledge" chip renders in chat - System prompt replaces the four knowledge blobs with an `## Available Knowledge` block and is hardened to load knowledge for given topics - New "Knowledge Usage" scorer and `requiredKnowledge` assertions check that knowledge loads as expected in test scenarios - Filters GraphQL error responses out of `output.docs` before faithfulness scoring to reduce noise See "Knowledge Usage" scoring 100% in evals with no major regressions: https://github.com/supabase/supabase/pull/44296#issuecomment-4145760236 Sample trace showing the tool in action ([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=project_logs&object_id=5a8d02e5-b3b6-40cc-ba76-ecee286478f4&r=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1&s=351a11c8-9cb7-4945-93ad-d11e8cc2e3e1)) <img width="2192" height="1730" alt="CleanShot 2026-03-30 at 13 53 59@2x" src="https://github.com/user-attachments/assets/f483767c-34e0-401c-8089-5b9834fe696a" /> **References** - https://ai-sdk.dev/cookbook/guides/agent-skills Closes AI-508 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added dynamic knowledge loading capability enabling the AI assistant to retrieve on-demand information about PostgreSQL best practices, Row Level Security, Edge Functions, and Realtime. * **Bug Fixes** * Improved search results filtering to exclude error responses in tool outputs. * **Tests** * Enhanced evaluation metrics with knowledge usage scoring. * Expanded test dataset cases to validate knowledge requirement handling. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
0c5f64fcba |
feat(assistant): upgrade default models to gpt-5.4-nano and gpt-5.3-codex (#44107)
Replaces `gpt-5-mini` and `gpt-5` with `gpt-5.4-nano` and `gpt-5.3-codex` respectively. Clients with stale model IDs in IndexedDB will gracefully reset to the new defaults. While we can technically keep the existing models around, we've [opted](https://supabase.slack.com/archives/C051L8U2EJF/p1774283070517609?thread_ts=1773771991.871669&cid=C051L8U2EJF) to replace them w/ the newer models for simplicity. Basic completion endpoints use `'none'` reasoning level for optimal speed. Rationale for these models is they provide they best balance of intelligence/speed and cost. GPT-5.4-nano is less expensive (0.8x price), faster, and smarter than GPT-5-mini. GPT-5.4-mini would be even smarter but is 3x the price. GPT-5.3-Codex is ~1.4x the price of GPT-5, while GPT-5.4 would be 2x price, but 5.3-Codex is still a big intelligence boost from GPT-5. See [eval comparison](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/mattrossman%2Fai-509-v2-upgrade-assistant-models-beyond-gpt-5-family-1774468619?c=master-1774458837&diff=between_experiments), scores are relatively stable and conciseness naturally improves on gpt-5.4-nano. Other change: - Fixed an eval test case to clarify that https://supabase.help is also a correct URL for submitting support ticket, which was unfairly scored as incorrect [here](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=5244cccd-23b2-4f79-9dd2-287f1b40ebad&r=bac9b903-8bde-4c21-99dd-e0ed141c4f9e&s=f248fbf5-75bf-4aab-be0a-87a4298e6d11) I sanity checked the Assistant, natural language filters, and SQL Editor completions on staging preview. References: - https://openai.com/index/introducing-gpt-5-4-mini-and-nano/ - https://openai.com/index/introducing-gpt-5-3-codex/ - https://developers.openai.com/api/docs/pricing Closes AI-509 |
||
|
Matt Rossman
|
adf8b0c67c |
feat(assistant): per-endpoint reasoningEffort + model config cleanup (#43981)
We're exploring support for newer models like [gpt-5.4-nano](https://openai.com/index/introducing-gpt-5-4-mini-and-nano/) in Assistant. This model doesn't support the `'minimal'` reasoning effort level we use for gpt-5-mini which leads to vague errors. <img width="595" height="263" alt="CleanShot 2026-03-18 at 17 13 05@2x" src="https://github.com/user-attachments/assets/cf7c2370-322d-4a8a-be55-23e680db0aa0" /> Also, we've [previously discussed](https://supabase.slack.com/archives/C0161K73J1J/p1771544464850199?thread_ts=1771493920.775699&cid=C0161K73J1J) that reasoning adds unnecessary latency to otherwise simple AI completion endpoints like `title-v2`. We want more control of reasoning level independent of model/endpoint. This PR aims to solve both problems by: - making reasoning effort configurable on a per-request basis - adding compile-time guardrails to prevent selecting an incompatible reasoning level for models - adding a `DEFAULT_COMPLETION_MODEL` with minimal reasoning that we can update with newer models that support disabling reasoning (independent of Assistant chat model reasoning) Other improvements to our model config logic: - Fixes bug in `onboarding/design.ts` and `assistant.eval.ts` where `providerOptions` was being dropped - `getModel()` now returns a bundled `modelParams` object (spread into AI SDK calls) so `providerOptions` can't be accidentally omitted (this [has happened before](https://supabase.slack.com/archives/C0161K73J1J/p1771518443534309?thread_ts=1771493920.775699&cid=C0161K73J1J)) - Introduces an `ASSISTANT_MODELS` registry as a single source of truth for assistant model config, eliminating hardcoded model IDs across the codebase - Aligns free/pro model conditional logic with `assistant.advance_model` entitlement naming conventions instead of the `isLimited` pattern - Adds `console.error` logging of Assistant stream errors so we can interpret reasoning effort compatibility errors in the future (instead of just opaque "Sorry, I'm having trouble responding right now" card) - Removes unnecessary type casts and generally making the model config logic stricter - Removes pre-existing dead code: `anthropic` provider variant in `GetModelParams` / `PROVIDERS` registry that was never implemented in `getModel()` Now if you try to select an unsupported reasoning level you get a type error: <img width="1306" height="320" alt="CleanShot 2026-03-20 at 14 37 24@2x" src="https://github.com/user-attachments/assets/a6ac234b-5ea5-4d81-8e01-ac4be34a0800" /> And if for some reason an invalid reasoning level slips through, you now get a server-side error surfacing the issue: <img width="1268" height="204" alt="CleanShot 2026-03-20 at 14 58 14@2x" src="https://github.com/user-attachments/assets/aadc1b7a-9495-475f-9741-39979bd27cd7" /> I've tested gpt-5 and gpt-5-mini are still working on the staging preview and verified the models were selected properly in Braintrust logs. Both models are available on my Pro test account, and my Free test account shows the Pro upgrade CTA. Closes AI-446 Closes AI-551 |
||
|
Matt Rossman
|
25036af80e |
fix(assistant): sanitize backslash-escaped apostrophes in SQL (#43728)
Fix for the LLM occasionally generating MySQL-style `\'` escapes in SQL, which are invalid in PostgreSQL. Example trace where this happened in the wild: ([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/review?tab=experiment&r=5fcf1b12-8584-455c-9e9a-bdc0fa3ed21c&s=5fcf1b12-8584-455c-9e9a-bdc0fa3ed21c&o=0627ada8-b567-4117-9fe8-49d847cb73a7&review=1)) **Changes** - Adds `fixSqlBackslashEscapes` to convert `\'` → `''` before SQL is executed - Unit tests + adversarial eval dataset case Compare the results of the adversarial test case: - `master`: 0% SQL Validity ([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Dev%20(mattrossman%2FAssistant)/trace?object_type=experiment&object_id=b469cbf7-4d6f-429c-9819-6c4099294123&r=dce5a29b-2fde-44c3-80f8-4e14d1f657c0&s=dce5a29b-2fde-44c3-80f8-4e14d1f657c0)) - This branch: 100% SQL Validity ([Braintrust](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=160e9ce0-e320-4f6d-8aa7-c5ad7e01fbd2&r=d75ef0e3-90ed-42a7-9ef3-8bf69592f193&s=0eeca492-dbe6-451e-8d81-127caff30320)) Closes AI-400 |
||
|
Matt Rossman
|
e4a9b6882c |
fix(assistant): use extractTextOnly in conciseness scorer (#43612)
`concisenessScorer` was passing full serialized text + tool calls JSON to the LLM judge (SQL queries, GraphQL payloads, etc.). Switches to `extractTextOnly` so the judge only evaluates text the user actually sees. Prerequisite for https://github.com/supabase/supabase/pull/43613 to set a fair conciseness baseline score. Ref AI-402 |
||
|
Matt Rossman
|
517171b246 |
feat(assistant): online evals support and CI workflows (#43194)
Lays groundwork for online evals on Assistant chat logs. https://www.braintrust.dev/docs/observe/score-online ### Changes - New workflows: - `braintrust-scorers-deploy.yml` keeps prod scorers in sync on push to `master` - `braintrust-preview-scorers-deploy.yml` deploys preview scorers to the staging project for PRs labeled `preview-scorers`, posting a comment with scorer links ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)) - `braintrust-preview-scorers-cleanup.yml` deletes preview scorers when the PR is closed ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000749847)) - Adds `evals/scorer-online.ts` entry point invoked with `pnpm scorers:deploy`, registering scorers for online evals in the Braintrust "Assistant" project - Refactors scorer code to separate online-compatible scorers (`scorer-online.ts`) from WASM-dependent ones (`scorer-wasm.ts`) - "URL Validity" scorer now only checks Supabase domains to prevent requests to untrusted origins - Span `input` is now shaped `{ prompt: string }` instead of plain `string` for compatibility with offline eval scorers - Env vars `BRAINTRUST_STAGING_PROJECT_ID` and `BRAINTRUST_PROJECT_ID` configured in GitHub repo settings - `generateAssistantResponse` now uses `startSpan` + `withCurrent` instead of `traced()` to manually manage the root span lifecycle — this ensures `onFinish` logs output to the span _before_ `span.end()` is called, which is when Braintrust triggers scoring automations ### Online Scorers We share scoring logic across offline and online evals, but some of our scorers aren't transferrable to an "online" setting due to runtime challenges or ground truth requirements. **Supported** - Goal Completion - Conciseness - Completeness - Docs Faithfulness - URL Validity **Unsupported** - Correctness (requires ground truth output) - Tool Usage (requires ground truth requiredTools) - SQL Syntax (uses libpg-query WASM) - SQL Identifier Quoting (uses libpg-query WASM) ### How to use these scorers Going forward if you want to add/edit online eval scorers, add the `preview-scorers` label to a PR. This deploys scorers to the [Assistant (Staging Scorers)](https://www.braintrust.dev/app/supabase.io/p/Assistant%20(Staging%20Scorers)?v=Overview) project in Braintrust with branch-specific slugs, and comments on the PR ([example](https://github.com/supabase/supabase/pull/43194#issuecomment-4000097222)). From the Braintrust dashboard you can "Test" the scorer with traces from any project. <img width="1866" height="528" alt="CleanShot 2026-03-05 at 15 15 00@2x" src="https://github.com/user-attachments/assets/4f15cebc-3f2d-4e8a-9ee2-fe8ef7bf4199" /> Once merged, scorers are deployed to the primary [Assistant](https://www.braintrust.dev/app/supabase.io/p/Assistant) project, and preview scorers are deleted from the staging project. Down the road, scorers on the Assistant project will run automatically on a sample of production traces. Closes AI-437 |
||
|
Matt Rossman
|
851cc00545 |
feat(assistant): run 3 trials for Assistant evals in CI (#42510)
Runs 3 trials for Assistant evals in CI to reduce random variation. Locally, only 1 trial is run. Also adds `CI` to `studio#build` env in turbo.json. This env var is [automatically set by GitHub Actions](https://github.blog/changelog/2020-04-15-github-actions-sets-the-ci-environment-variable-to-true/). Compare number of trials: - [Assistant (mattrossman/ai-398-increase-trial-count-for-assistant-evals-1770305591)](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/mattrossman%2Fai-398-increase-trial-count-for-assistant-evals-1770305591) - [Assistant (master)](https://www.braintrust.dev/app/supabase.io/p/Assistant/experiments/master-1770305906?c=mattrossman/ai-398-increase-trial-count-for-assistant-evals-1770305591) References: - https://www.braintrust.dev/docs/evaluate/run-evaluations#trials Closes AI-398 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated evaluation configuration to adjust trial counts based on CI environment * Integrated CI environment variable into build system configuration <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Ali Waseem <waseema393@gmail.com> |
||
|
Matt Rossman
|
4b8bab4d14 |
feat(assistant): score URL validity and fix support ticket URL guidance (#42227)
**Logic changes** - Adds function in `helpers.ts` to extract URLs from text via regex - I also considering using a library like [linkify-it](https://www.npmjs.com/package/linkify-it) for this but figured it's not worth the extra dep - Adds associated tests in `helpers.test.ts` - Adds "URL Validity" scorer which performs a HEAD request for links in Assistant response text and determins what portion of links have `.ok` responses - Adds eval case to check correctness of support ticket URL answers **Prompt changes** - Informs Assistant of https://supabase.com/dashboard/support/new being the URL to create support tickets - Encourages Assistant to "self-debug" issues before directing users to create support tickets See [Eval Report](https://github.com/supabase/supabase/pull/42227#issuecomment-3807772871) and [Correctness](https://www.braintrust.dev/app/supabase.io/p/Assistant/trace?object_type=experiment&object_id=1ad0f9b0-5adb-436c-9812-a87aac62c036&r=1ef13459-a98c-4904-925e-6d81276cebb2&s=dbe5c607-a560-462b-8745-41d430744431) analysis for new support ticket test case. Resolves AI-384 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added URL validity scoring to evaluations and helper utilities for extracting/cleaning URLs. * Added evaluation cases for support-ticket URL handling and OAuth callback guidance. * **Documentation** * Updated assistant guidance to prefer self-resolution, include support-ticket direction, clarified data-recovery search steps, and added template-URL notation. * **Tests** * Expanded URL extraction and related utility tests to cover many formats and edge cases. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Matt Rossman
|
a127f2cbbc |
test(assistant): add eval case for execute_sql usage on default "Generate sample data" prompt (#42219)
test: add eval case for execute_sql on sample data generation |
||
|
Matt Rossman
|
eb259f1364 |
feat(assistant): score and improve SQL identifier quoting (#42122)
* feat: SQL correctness scorer, override mock tables * feat: replace "SQL Correctness" with "SQL Identifier Quoting" scorer * fix(prompt): discourage simulating confirmation of execute_sql tool this is already handled at the UI layer * fix(prompt): encourage quotes on identifiers with caps * feat: move extractIdentifiers to own file, add tests * chore: shorten tests * feat: extract ColumnDef column names in extractIdentifiers * refactor: sqlIdentifierQuotingScorer with more thorough checks * refactor: consolidate into `sql-identifier-quoting.ts` * feat: support mocking schemas, eval test case with case sensitive schema * fix: test cases that don't match default mock schema * chore: format * feat(prompt): mention special characters and reserved words * feat: optional description in metadata, test with special characters * feat: consolidated comprehensive test case * fix(prompt): revert conflicting instruction |
||
|
Matt Rossman
|
4553f09bb5 |
feat(assistant): hallucination scorers + corrective measures for storage versioning answers (#41655)
* feat: "Docs Faithfulness" scorer * feat: test case for storage object restoration * feat: "Factuality" scorer * feat: "Factuality" -> "Correctness" * feat: update Storage recovery test case * feat: finishReason in task output * feat: encourage parallel tool calls + docs search, discourage superfluous context gathering * prompt tuning (tool selection strategy) * add data recovery section in chat prompt * test: S3 versioning support correctness * refactor: derive stepsSerialized/textOnly from shared steps data * fix: input in correctness scorer |
||
|
Matt Rossman
|
072883bcec |
feat: assistant evals (#41311)
* chore: bump `supabase` CLI * chore: stricter message types in `generate-v4.ts` * feat: tutorial eval https://www.braintrust.dev/docs/evaluation * feat: project ID for eval * refactor: `generateAssistantResponse` out of `handlePost` * refactor: generateAssistantResponse to lib/ai * feat: factuality eval with assistant response * chore: upgrade braintrust to v1.0.1 * chore: silence tsconfig warning * feat: assertion scorer * fix: aggregate tools across all steps * refactor: strict tool names, remove need for `as const` * refactor: generic tool name type in assertions * feat: transfer mocks from `feature/braintrust` * feat: LLM criteria assertion * feat: braintrust evals workflow * fix: BRAINTRUST_PROJECT_ID * feat: `sql_similar` assertion * fix: `OPENAI_API_KEY` in workflow env * feat: split AssertionScorer into separate scorers * feat: remove tutorial eval * feat: 20 minute CI timeout * feat: category in test case metadata * feat: score with gpt-5 * refactor: dataset to own file, colocate scorers * feat: "gpt-5.2-2025-12-11" for llm as a judge * feat: SQL syntax scorer with `libpg-query` * feat: `evals:setup` and `evals:run` scripts * feat: `evals:setup` in CI * feat: human readable scorer names * chore: rename to "SQL Validity" * feat: add 2 "sql_generation" test cases * feat: update requiredTools in test cases * chore: ignore Cursor MCP config * feat: "Conciseness" score * feat: "Completeness" scorer * fix: generate-v4 test mocks * feat: serialize "steps" for scorer inputs * updated node mem options for typecheck * updated runner * remove ram update as actions handle this * feat: read `BRAINTRUST_PROJECT_ID` from secrets * feat: score helpfulness, remove old scorers * feat: separate `evals:run` and `evals:upload` scripts * feat: passthrough entire classifier result * feat: use live `search_docs` impl, store docs result in metadata * feat: reduce classifier options * feat: filter workflow by `run-evals` PR label or `master` branch * chore: cleanup stubbed mock tools * fix: checkout actual branch with `ref:` * fix: capture search_docs results from all content parts * feat: simplify sql syntax score calculation * feat: use AI SDK's UI message validator * docs: justification for relative `extends` * fix: cleanup leftover validatedMessages * doc: note mock token isn't secret for snyk * fix: mock ui message to pass validation * feat: revert ignoring Cursor MCP config Using `.git/info/exclude` instead until we have an opinion on this * feat: add "tsconfig" as shared-data devDependency, revert relative path in tsconfig * refactor: tool call parsing into function * Update apps/studio/evals/assistant.eval.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * refactor: organize mock schemas and tool factories --------- Co-authored-by: Ali Waseem <waseema393@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> |
