Files
supabase/apps/studio/lib/ai/tools/index.test.ts
Saxon Fletcher dcac820571 feat(studio): add assistant notebook run tool (#49361)
## I have read the
[CONTRIBUTING.md](https://github.com/supabase/supabase/blob/master/CONTRIBUTING.md)
file.

YES

## What kind of change does this PR introduce?

Assistant feature and data-handling plumbing.

## Stack context

This stack is based on #49352 (`chore/assistant-tool-outcomes`) and
assumes #49350–#49352 merge first.

Review bottom to top:

1. #49361 — assistant notebook run tool
2. #49362 — assistant notebook run UI
3. #49364 — terminal-state polish

## What is the current behavior?

The Assistant can read and edit notebooks, but it cannot execute all
saved query cells as one approved operation.

## What is the new behavior?

- Adds a `run_notebook` tool with one approval gate for the complete
notebook.
- Executes database and log cells sequentially in notebook order.
- Rejects stale runs when the notebook changed after the Assistant read
it.
- Resolves primary and read-replica connections and forwards
authorization to log and replica requests.
- Shares rows with the model only when the organization's AI
data-sharing level permits it.
- Strictly validates and sanitizes persisted notebook-run output before
replaying message history.
- Registers the tool in prompts, filtering, mocks, and tool
construction.

## How to test manually

This is the tool/data layer; use the top-of-stack preview from #49364
for the complete UI while checking these behaviors.

1. In Explorer, create and save a notebook named **Assistant run smoke
test** with:
   - a markdown cell
   - a working database query
   - a working Logs query
- a database query that returns no rows, such as `select 1 where false`
2. Open the AI Assistant and ask: **Read the “Assistant run smoke test”
notebook and analyze it using its current results.**
3. Confirm the Assistant reads the notebook and requests one
`run_notebook` approval for all query cells, rather than requesting one
approval per cell.
4. Approve the run. Confirm database and Logs queries execute in
notebook order, the markdown cell is not executed, and the Assistant
responds only after the complete run finishes.
5. Start another run but do not approve it yet. In another tab, edit and
save the notebook. Return to the pending approval and approve it.
6. Confirm the stale run is rejected, the Assistant reads the latest
notebook version, and a new approval is required.
7. Optional privacy check: set the organization AI data-sharing level to
schema-only, run a query containing a recognizable value, and confirm
the value remains visible in the notebook result UI but is not repeated
in the Assistant's answer.

## Automated test

`mise exec node@22 -- pnpm --dir apps/studio exec vitest --run
lib/ai/tool-filter.test.ts lib/ai/tools/index.test.ts
lib/ai/tools/mock-tools.test.ts lib/ai/tools/notebook-tools.test.ts
lib/ai/tools/tool-sanitizer.test.ts`

80 tests pass at this stack boundary.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added AI-assisted notebook execution for database and log cells, with
approval, freshness checks, replica support, and per-cell error
handling.
- Added notebook deletion and database discovery and validation for
notebook management.
  - Added configurable privacy controls for notebook results.
  - Added request header support for analytics SQL execution.

- **Bug Fixes**
- Improved replica lookup handling so other notebook cells can continue
when one lookup fails.
- Prevented invalid, unauthorized, or overly detailed notebook execution
results from being exposed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Saxon Fletcher <SaxonF@users.noreply.github.com>
2026-08-25 18:16:33 +10:00

103 lines
4.0 KiB
TypeScript

import { beforeEach, describe, expect, it, vi } from 'vitest'
import { getTools } from './index'
import { getMcpTools } from './mcp-tools'
vi.mock('common', () => ({ IS_PLATFORM: true }))
vi.mock('./mcp-tools', () => ({ getMcpTools: vi.fn() }))
vi.mock('./studio-tools', () => ({ getStudioTools: vi.fn(() => ({ studio_tool: {} })) }))
vi.mock('./schema-tools', () => ({ getSchemaTools: vi.fn(() => ({ schema_tool: {} })) }))
vi.mock('./incident-tools', () => ({ getIncidentTools: vi.fn(() => ({ incident_tool: {} })) }))
vi.mock('./fallback-tools', () => ({ getFallbackTools: vi.fn(() => ({ fallback_tool: {} })) }))
// Identity filter so assertions can check the raw merged tool set
vi.mock('../tool-filter', () => ({ filterToolsByOptInLevel: vi.fn((tools) => tools) }))
const BASE_PARAMS = {
projectRef: 'abcdefghijklmnopqrst',
connectionString: 'postgresql://localhost',
authorization: 'Bearer token',
aiOptInLevel: 'schema_and_log_and_data' as const,
accessToken: 'access-token',
baseUrl: 'https://supabase.com/dashboard',
signal: new AbortController().signal,
}
describe('ai/tools getTools', () => {
beforeEach(async () => {
vi.clearAllMocks()
vi.mocked(getMcpTools).mockResolvedValue({ list_tables: {} } as any)
// Reset to platform each test; the self-hosted test overrides to false.
// Done here (not afterEach) so the spy can't leak across tests via order.
const common = await import('common')
vi.spyOn(common, 'IS_PLATFORM', 'get').mockReturnValue(true)
})
it('includes studio, MCP, schema and incident tools on platform', async () => {
const tools = await getTools(BASE_PARAMS)
expect(getMcpTools).toHaveBeenCalledWith({
accessToken: BASE_PARAMS.accessToken,
projectRef: BASE_PARAMS.projectRef,
aiOptInLevel: BASE_PARAMS.aiOptInLevel,
signal: BASE_PARAMS.signal,
})
expect(tools).toHaveProperty('studio_tool')
expect(tools).toHaveProperty('list_tables')
expect(tools).toHaveProperty('schema_tool')
expect(tools).toHaveProperty('incident_tool')
})
it('degrades gracefully to the remaining tools when remote MCP fetch fails', async () => {
const consoleSpy = vi.spyOn(console, 'error').mockImplementation(() => {})
vi.mocked(getMcpTools).mockRejectedValueOnce(new Error('remote MCP unreachable'))
const tools = await getTools(BASE_PARAMS)
// The assistant still works with the non-MCP tools instead of throwing
expect(tools).toHaveProperty('studio_tool')
expect(tools).toHaveProperty('schema_tool')
expect(tools).toHaveProperty('incident_tool')
expect(tools).not.toHaveProperty('list_tables')
expect(consoleSpy).toHaveBeenCalled()
consoleSpy.mockRestore()
})
it('does not fetch MCP tools when no access token is provided', async () => {
const tools = await getTools({ ...BASE_PARAMS, accessToken: undefined })
expect(getMcpTools).not.toHaveBeenCalled()
expect(tools).toHaveProperty('studio_tool')
expect(tools).not.toHaveProperty('list_tables')
})
it('uses fallback tools and skips MCP when self-hosted', async () => {
const common = await import('common')
vi.spyOn(common, 'IS_PLATFORM', 'get').mockReturnValue(false)
const tools = await getTools(BASE_PARAMS)
expect(getMcpTools).not.toHaveBeenCalled()
expect(tools).toHaveProperty('studio_tool')
expect(tools).toHaveProperty('fallback_tool')
expect(tools).not.toHaveProperty('list_tables')
})
it('excludes notebook tools when isExplorerEnabled is not set', async () => {
const tools = await getTools(BASE_PARAMS)
expect(tools).not.toHaveProperty('list_notebooks')
expect(tools).not.toHaveProperty('get_notebook')
expect(tools).not.toHaveProperty('run_notebook')
})
it('includes notebook tools only when isExplorerEnabled is true', async () => {
const tools = await getTools({ ...BASE_PARAMS, isExplorerEnabled: true })
expect(tools).toHaveProperty('list_notebooks')
expect(tools).toHaveProperty('get_notebook')
expect(tools).toHaveProperty('run_notebook')
})
})