Files
ironclaw/tests/layered_memory.rs
Illia Polosukhin af9b59a284 feat: unified tool dispatch + schema-validated workspace (#2049)
* feat(workspace): add JSON Schema validation to document metadata

Add a `schema` field to `DocumentMetadata` that enables automatic content
validation on workspace writes. When a document or its folder `.config`
carries a JSON Schema, all write operations (write, append, patch,
write_to_layer, append_to_layer) validate content against it before
persisting. This is the foundation for typed system state (settings,
extension configs, skill manifests) stored as workspace documents.

Builds on the metadata infrastructure from #1723 — schema is inherited
via the existing `.config` chain (folder → document → defaults).

Refs: #640, #1894, #1937

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(tools): add channel-agnostic ToolDispatcher with audit trail

Introduce `ToolDispatcher` — a universal entry point for executing tools
from any caller (gateway, CLI, routine engine, WASM channels). Creates
lightweight system jobs for FK integrity, records ActionRecords, and
returns ToolOutput. This is a third entry point alongside v1's
Worker::execute_tool() and v2's EffectBridgeAdapter::execute_action().

DispatchSource::Channel(String) is intentionally string-typed — channels
are interchangeable extensions that can appear at runtime.

Also adds JobContext::system() factory and create_system_job() to both
PostgreSQL and libSQL backends.

Refs: #640

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(workspace): settings-as-workspace-documents with dual-write adapter

Add WorkspaceSettingsAdapter that implements SettingsStore by reading/
writing workspace documents at _system/settings/{key}.json. During
migration, dual-writes to both the legacy settings table and workspace.
Reads prefer workspace, falling back to the legacy table.

Known setting keys (llm_backend, selected_model, tool_permissions.*, etc.)
get JSON Schemas stored in document metadata — writes are validated
automatically by Phase 0's schema validation.

Also adds settings_schemas.rs with compile-time schema registry and
settings_path() helper.

Refs: #640, #1937

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(gateway): wire ToolDispatcher into GatewayState

Add tool_dispatcher field to GatewayState with with_tool_dispatcher()
builder method. Create and wire the dispatcher in main.rs when both
tool_registry and database are available. All 16 GatewayState
construction sites updated.

Per-handler migration (routing mutations through ToolDispatcher instead
of direct DB calls) is deferred to follow-up PRs — each handler has
complex ownership checks, cache refresh, and response types.

Refs: #640

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(tools): add system introspection tools (tools_list, version)

Add SystemToolsListTool and SystemVersionTool as proper Tool
implementations that replace hardcoded /tools and /version commands.
Registered at startup via register_system_tools(). Available in both
v1 and v2 engines — no is_v1_only_tool filter to worry about.

Refs: #640

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat(workspace): extension and skill state schemas and path helpers

Add workspace path helpers and JSON Schemas for storing extension configs,
extension state, and skill manifests under _system/extensions/ and
_system/skills/. This establishes the workspace document structure that
ExtensionManager and SkillRegistry will use as a durable persistence
backend (read-through cache pattern).

Runtime state (active MCP connections, WASM runtimes) stays in memory.
Only durable config and activation state moves to workspace documents.

Refs: #640, #1741

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review feedback and CI failures

CI fixes:
- deny.toml: allow MIT-0 license required by jsonschema
- workspace/document.rs: #[allow(dead_code)] on system path constants
  pending follow-up phases that consume them
- workspace/settings_adapter.rs: remove unused chrono::Utc import
- workspace/settings_adapter.rs: collapse nested if into && form

Review fixes (gemini-code-assist):
- tools/dispatch.rs: await save_action directly instead of fire-and-forget
  tokio::spawn so short-lived CLI callers cannot drop audit records before
  they are persisted; surface errors via tracing::warn
- tools/dispatch.rs: remove DispatchSource::Agent variant — sequence_num=0
  with a reused job_id would violate UNIQUE(job_id, sequence_num). Agent
  callers must use Worker::execute_tool() which manages sequence numbers
  atomically against the agent's existing job
- workspace/settings_adapter.rs: validate content against the schema BEFORE
  the first workspace write so the initial document creation cannot bypass
  schema enforcement (subsequent writes are validated by the workspace
  resolved-metadata path established after the first write)

Refs: #2049

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: unify all machine state under .system/

Rename the workspace prefix from `_system/` to `.system/` (Unix dot-prefix
convention for hidden internal state) and migrate v2 engine state from
`engine/` to `.system/engine/` so all machine-managed state lives under
one root.

New layout:

  .system/
  ├── settings/         (per-user settings as workspace docs)
  ├── extensions/       (extension config + activation state)
  ├── skills/           (skill manifests)
  └── engine/
      ├── README.md     (auto-generated index)
      ├── knowledge/    (lessons, skills, summaries, specs, issues)
      ├── orchestrator/ (Python orchestrator versions, failures, overlays)
      ├── projects/     (project files + nested missions/)
      └── runtime/      (threads, steps, events, leases, conversations)

The inner `.runtime/` dot-prefix is dropped under `.system/engine/` since
`.system/` itself is the hidden marker; no double-hiding needed.

The `ENGINE_PREFIX` constant in `workspace::document::system_paths` is
declared as the canonical convention; bridge `store_adapter` continues
to define per-subdirectory constants below it for ergonomic interpolation.

No legacy migration code — pre-production rename.

Refs: #2049

* fix(pr-2049): security, correctness, and robustness fixes from review

Critical security:
- dispatch.rs: redact sensitive params before persisting ActionRecord
  (was leaking plaintext secrets into the audit log for tools with
  sensitive_params())
- settings_schemas.rs: validate settings keys against path traversal
  (reject /, \, .., leading ., empty, length > 128, non-alphanumeric);
  wire validation into all settings_adapter read/write/delete paths

Data correctness:
- history/store.rs + libsql/jobs.rs: write status as JobState::Completed
  .to_string() ('completed' snake_case) instead of 'Completed'; system
  jobs were round-tripping as Pending in parse_job_state()
- settings_adapter.rs: fix .system/.config metadata to set
  skip_versioning: false (was true) — descendants inherit this via
  find_nearest_config, so the previous value silently disabled
  versioning for ALL .system/** documents, contradicting the audit-
  trail intent
- workspace/mod.rs: add resolve_metadata_in_scope; use it in
  write_to_layer / append_to_layer so non-primary layer writes resolve
  schema/indexing/versioning from the target layer's .config chain
  instead of the primary user_id's. Also pass &scope (not &self.user_id)
  to maybe_save_version so versions are attributed to the correct scope

Pipeline parity:
- dispatch.rs: add SafetyLayer to ToolDispatcher; mirror Worker pipeline
  (prepare_tool_params -> validator -> redact -> timeout -> sanitize
  output) so dispatch path gets the same safety guarantees as the agent
  worker. Sanitized output is now stored in ActionRecord.output_sanitized
  instead of duplicating raw JSON

Robustness:
- settings_adapter.rs: propagate update_metadata errors in
  ensure_system_config and write_to_workspace (was silently ignored
  via let _ =, leaving schemas/skip_indexing unenforced)
- settings_adapter.rs: set_all_settings now collects the first workspace
  write error and returns it after the legacy write completes, so
  partial-migration state is observable
- settings_schemas.rs: rewrite llm_custom_providers schema to match
  CustomLlmProviderSettings (id/name/adapter/base_url/default_model/
  api_key/builtin instead of stale name/protocol/base_url/model)

Build:
- Cargo.toml: jsonschema with default-features = false to avoid pulling
  a second reqwest major version

Docs:
- db/mod.rs: docstring for create_system_job uses 'completed' snake_case
- workspace/document.rs: clarify .system/ versioning ("by default ARE
  versioned; individual files may opt out via skip_versioning")
- settings_adapter.rs: clarify per-key reads prefer workspace, aggregate
  reads stay on legacy during migration
- tools/builtin/system.rs: trim doc to match implemented scope
  (system_tools_list, system_version)
- channels/web/mod.rs: move stale 'sweep tasks managed by with_oauth'
  comment back to oauth_sweep_shutdown line

Refs: #2049

* docs+ci: enforce 'everything goes through tools' principle

Document the core design principle from #2049 in two places so future
contributors (human and AI) discover it during development:

- CLAUDE.md: new "Everything Goes Through Tools" section near the
  "Adding a New Channel" guide. Includes the rule, the rationale (audit
  trail, safety pipeline parity, channel-agnostic surface, agent
  parity), and a pointer to the detailed rule file.
- .claude/rules/tools.md: full pattern with required/forbidden examples,
  the list of layers that ARE exempt (Worker::execute_tool, v2
  EffectBridgeAdapter, tool implementations themselves, background
  engine jobs, read-aggregation queries), and how to annotate
  intentional exceptions. Also extends `paths` to cover
  src/channels/** and src/cli/** so it surfaces when those files are
  edited.

Enforce with a new pre-commit safety check (#7) in
scripts/pre-commit-safety.sh:

- Scans newly added lines under src/channels/web/handlers/*.rs and
  src/cli/*.rs for direct touches of state.{store, workspace,
  workspace_pool, extension_manager, skill_registry, session_manager}.
- Suppress with a trailing `// dispatch-exempt: <reason>` comment on
  the same line, matching the existing `// safety:` convention.
- Only checks added lines (`+` in the diff), so existing untouched
  handlers don't trip the check during incremental migration.

The check fires only for new code: handlers that haven't been migrated
yet (52 existing direct accesses across 12 handler files) won't break
unmodified, but any new line that bypasses the dispatcher will be
flagged at commit time.

Refs: #2049

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(pr-2049): address Copilot review on workspace schema layer

- workspace::extension_state: extension/skill path helpers now reuse the
  canonical name validators (`canonicalize_extension_name`,
  `validate_skill_name`) instead of a weak `replace('/', "_")`. Names
  containing `..`, `\`, NUL, or other escapes are now rejected at the
  helper boundary, eliminating a path-traversal foothold for callers.
  Helpers return `Result<String, PathError>`. Regression tests added.

- workspace::settings_adapter::ensure_system_config: now idempotent across
  upgrades. If `.system/.config` already exists with stale metadata
  (e.g. an older `skip_versioning: true` from before fix #3042846635),
  it is repaired to the expected inherited values instead of being left
  silently broken. Regression test added.

- workspace::settings_adapter::write_to_workspace: lazily seeds
  `.system/.config` via a `OnceCell`, so callers no longer need to
  remember to invoke `ensure_system_config()` at startup before any
  setting write. Regression test added.

- workspace::settings_adapter::delete_setting: workspace delete failures
  are now logged via `tracing::warn!` instead of being silently dropped.
  We still don't propagate the error — the legacy table is the source of
  truth during migration and a stale workspace doc is recoverable on the
  next write — but partial-delete state is now observable.

- workspace::schema: documented why we don't cache compiled validators
  yet (settings/extension/skill writes are not a hot path; revisit if
  schema validation moves into a frequent write path).

[skip-regression-check] schema.rs change is doc-only.

* fix(pr-2049): address 4 remaining review issues

1. tool_dispatcher dropped during gateway startup
   src/channels/web/mod.rs: rebuild_state was initializing
   tool_dispatcher to None, so every subsequent with_* call zeroed
   the dispatcher the first caller injected. Preserve it across
   rebuild_state like every other field. Regression test:
   tool_dispatcher_survives_subsequent_with_calls.

2. WorkspaceSettingsAdapter not wired into runtime
   src/app.rs: Build the adapter in build_all() when workspace+db
   are both present, eagerly call ensure_system_config(), expose
   on AppComponents as settings_store, and thread it into
   init_extensions(...) so register_permission_tools and
   upgrade_tool_list receive it instead of the raw db.
   src/main.rs: SIGHUP handler prefers the adapter over raw db.
   src/workspace/mod.rs: re-export WorkspaceSettingsAdapter.

3. changed_by regression on layered writes
   src/workspace/mod.rs: write_to_layer and append_to_layer were
   passing the target layer's scope as changed_by, so version
   history attributed layered edits to the layer name instead of
   the actor. Pass self.user_id while keeping metadata resolution
   in the target scope. Regression test:
   layered_writes_record_actor_in_changed_by.

4. Legacy engine/ paths invisible after upgrade
   src/bridge/store_adapter.rs: Add migrate_legacy_engine_paths(),
   called at the start of load_state_from_workspace(), which scans
   list_all() for engine/... documents and rewrites them to
   .system/engine/... Idempotent: skips rewrites when the new path
   already exists, deletes the legacy duplicate either way. Three
   regression tests in #[cfg(all(test, feature = "libsql"))]
   module.

Quality gate: cargo fmt, cargo clippy --all --all-features zero
warnings, cargo test --all-features --lib 4313 passed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(e2e): use PUT for settings write in ownership test

test_settings_written_and_readable was sending POST /api/settings/{key}
but the route has been PUT since #4 (Feb 2026) — the test was returning
405 Method Not Allowed. Switch to httpx.put() so it matches the current
route registration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(pr-2049): address second round of review feedback

Addresses the remaining unresolved PR #2049 review comments from
serrrfirat and ilblackdragon.

## Changes

### ToolDispatcher — integration coverage + log level
- src/tools/dispatch.rs: add two libsql-gated integration tests for
  the full dispatch pipeline: (a) persist an ActionRecord with
  sensitive params redacted in the audit row while the tool still
  sees the raw value, sanitized output populated; (b) honor the
  per-tool execution_timeout() and record a failure action.
- Tests use a raw-SQL helper to find system-category jobs since
  list_agent_jobs_for_user intentionally filters them out.
- Replace warn! with debug! on audit persistence failure — dispatch
  is reachable from interactive CLI/REPL sessions where warn!/info!
  output corrupts the terminal UI (CLAUDE.md Code Style → logging).

### WorkspaceSettingsAdapter — log level
- src/workspace/settings_adapter.rs: same warn! → debug! fix on the
  delete_setting workspace failure path, for the same REPL reason.

### Schema validation — surface all errors
- src/workspace/schema.rs: switch from jsonschema::validate to
  validator_for + iter_errors so users fixing a malformed setting
  see every violation in one round instead of playing whack-a-mole.
  Also distinguishes "invalid schema" from "invalid content" errors.
- Regression tests: multiple_errors_are_all_reported and
  invalid_schema_is_distinguished_from_invalid_content.

### create_system_job — started_at + row growth docs
- src/db/libsql/jobs.rs and src/history/store.rs: include started_at
  in the INSERT (set to the same instant as created_at/completed_at)
  so duration queries don't see NULL and "started but not completed"
  filters don't misclassify these rows. Fixed in both backends.
- Add doc comments on both impls warning about row growth per
  dispatch call. Deleting rows would violate "LLM data is never
  deleted" (CLAUDE.md); if listing-query performance becomes a
  concern, prefer a partial index (WHERE category != 'system') over
  deletion.

### Lib test repair
- src/channels/web/server.rs: extensions_setup_submit_handler Err
  branch now sets resp.activated = Some(false) so clients and the
  regression test see an explicit `false` rather than `null`. Also
  rename the test's fake channel to snake_case (test_failing_channel)
  so it matches the canonicalize-extension-names behavior from
  PR #2129 — previously the test was passing a dashed name and
  getting "Capabilities file not found" instead of the intended
  activation failure.

## Not addressed (false positive / deferred)
- dispatch.rs:177 output_raw/output_sanitized swap — verified against
  ActionRecord::succeed(Option<String>, Value, Duration) and the
  worker's call site at job.rs:704; argument order is correct.
- settings_adapter.rs:186 TOCTOU window — author self-classified as
  "Low / completeness" and no other code path writes to
  .system/settings/** without going through write_to_workspace.
- schema.rs recompilation caching — deferred per earlier review.

## Quality gate
- cargo fmt
- cargo clippy --all --benches --tests --examples --all-features
  zero warnings
- cargo test --all-features --lib: 4387 passed, 0 failed, 3 ignored

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(pr-2049): address third round of review feedback

Addresses unresolved comments from serrrfirat's "Paranoid Architect
Review" and Copilot's third pass on the engine-state migration.

## src/workspace/settings_adapter.rs

### HIGH — Cross-tenant data leak through owner-scoped Workspace

`Workspace` is constructed for a single user_id at AppBuilder time.
Without gating, `set_setting("user_B", key, val)` would dual-write into
the **owner's** workspace, and a subsequent `user_A.get_setting(...)`
would return user_B's value: a real cross-user data leak.

Fix:
- Add `gate_user_id` field set to `workspace.user_id()` at construction.
- All `SettingsStore` methods that touch the workspace now check
  `workspace_allowed_for(user_id)` first; non-owner callers fall through
  to the legacy table only — preserving their pre-#2049 behavior.
- This matches the long-term plan: per-user settings live in the legacy
  table until a per-user `WorkspaceSettingsAdapter` (one per
  WorkspacePool entry) is wired up; admin/global settings go through
  the workspace-backed path so they pick up schema validation.

Regression test: `workspace_settings_are_owner_gated_in_multi_tenant_mode`
asserts (a) owner's workspace doc is not overwritten by a non-owner write,
(b) each user reads back their own legacy value, and (c) a non-owner with
no legacy entry must NOT see the owner's workspace value bleeding through.

### MEDIUM — Dual-write order

Reverse `set_setting` and `set_all_settings` to write legacy first,
workspace second. The legacy table is the source of truth during
migration (it backs aggregate `list_settings` reads), so writing it
first guarantees those readers always see a consistent value even if
the workspace write fails. Failed workspace writes are self-healing on
the next per-key read-miss.

### MEDIUM — `ensure_system_config_lazy` double-execution race

Replace the manual `get()`/`set()` pattern with
`OnceCell::get_or_try_init`. Two concurrent first-callers no longer
both run `ensure_system_config()`. Functionally equivalent (idempotent
either way) but no longer wasteful.

## src/bridge/store_adapter.rs

### MEDIUM — Migration drops document metadata (S3)

`migrate_legacy_engine_paths` previously copied only `doc.content`,
silently dropping the `metadata` column. Now calls
`ws.update_metadata(new_doc.id, &doc.metadata)` after each write to
preserve schema/skip_indexing/hygiene flags. Logged-not-fatal: content
has already been moved, metadata loss is recoverable.

Regression test: `migration_preserves_document_metadata` seeds a doc
with custom metadata and asserts it survives the rewrite.

### MEDIUM — `ws.exists()` swallowed transient errors (Copilot)

`unwrap_or(false)` on the existence check could cause the migrator to
overwrite an existing `.system/engine/...` doc when storage hiccups.
Now propagates the error (counts as failed step + `continue`), per
Copilot's exact suggested patch.

### LOW — `list_all()` runs every startup (Copilot)

Add a cheap preflight: `ws.list("engine")` first; only fall through to
the recursive `list_all()` discovery when the directory listing returns
at least one entry. Steady-state startups (post-migration) skip the
full workspace scan entirely.

Regression test: `migration_preflight_skips_full_scan_when_no_legacy_paths`
asserts unrelated and already-migrated documents are untouched.

### MEDIUM — Counter undercount on `already_present` (S5)

When `already_present` is true the legacy duplicate is still deleted,
but the previous code skipped the `migrated += 1` increment, undercounting
in debug logs. Fixed: `migrated` now counts every successful path
migration including the already-present case.

### Documented — Version-history loss is acceptable scope (C1)

Read-write-delete pattern means `memory_document_versions.document_id
ON DELETE CASCADE` drops the legacy doc's version chain. Documented in
the function-level doc comment as intentional + bounded:
- v2 engine state is runtime state (rewritten on every mutation), not
  user-curated data
- v2 was newly introduced in this PR — no production deployment with
  pre-existing curated history at risk
- A path-preserving rename op would need new trait methods on both
  backends; out of scope for fix-forward. If a future caller needs
  history-preserving rename, it should be added to the storage layer
  properly, not bolted onto migration.

## Quality gate
- cargo fmt
- cargo clippy --all --benches --tests --examples --all-features
  zero warnings
- cargo test --all-features --lib: 4390 passed, 0 failed, 3 ignored
  (+3 new tests on top of round 2)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(pr-2049): address fourth round of review feedback

Two latent issues flagged by serrrfirat in the latest review pass:

1. **Null schema permanently locks documents** (`src/workspace/schema.rs`).
   `serde_json` deserializes a metadata field of `"schema": null` as
   `Some(Value::Null)`, not `None`, so the upstream
   `if let Some(schema) = &metadata.schema` check passes through to
   `validate_content_against_schema`. There, `validator_for(Value::Null)`
   errors out and every subsequent write to that document is blocked — a
   latent DoS. Added an explicit `schema.is_null()` early-return guard at
   the top of the validator, plus a regression test
   (`null_schema_is_treated_as_no_op`) that asserts even non-JSON content
   passes when the schema is null.

2. **System job titles were raw source labels** (`src/history/store.rs`,
   `src/db/libsql/jobs.rs`). `create_system_job` set `title = source`,
   so any UI rendering `agent_jobs.title` would display dispatched
   system jobs as `channel:gateway` / `system` / etc. instead of a
   human-readable label. Both PostgreSQL and libSQL backends now write
   `format!("System: {source}")`. Updated the two dispatch integration
   tests that pinned the old format.

Schema-recompilation comment (`schema.rs:47`) was acknowledged as
"acceptable for now" by the reviewer; existing NOTE in the source
already documents the caching trade-off and upgrade path, so no code
change.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(pr-2049): address fifth round of review feedback

Eight comments from Copilot + serrrfirat. Real fixes for the load-bearing
gaps; doc clarifications for the rest where the existing behavior is
intentional.

**Real code changes**

- `src/tools/dispatch.rs` — enforce `tool.parameters_schema()` (JSON
  Schema) in the dispatch path. Previously the SafetyLayer validator only
  checked for injection patterns; channel/CLI/routine callers could pass
  arbitrary shapes and only discover the mismatch (or worse, silently
  malformed behavior) inside the tool itself. Now we run
  `jsonschema::validate(&tool.parameters_schema(), &normalized_params)`
  after the injection check, with a permissive-empty-schema fast path so
  tools that haven't yet declared a schema aren't penalised. Regression
  test `dispatch_rejects_params_violating_tool_schema` asserts a
  required-field violation is rejected before the tool is invoked.

- `src/workspace/settings_adapter.rs` — `write_to_workspace` now calls
  `schema_for_key(key)` once and reuses the resolved schema for both
  pre-write validation and post-write metadata persistence (was called
  twice). Eliminates duplicate work and removes a theoretical
  divergence window if the schema registry ever became non-deterministic.

- `src/workspace/settings_adapter.rs` — `ensure_system_config` now also
  rewrites the `.config` document content when its metadata is repaired,
  not just the metadata column. The metadata column is the inheritance
  source of truth, but having the doc's content silently diverge from it
  confuses anyone reading the doc directly to understand which inherited
  flags are active.

- `src/error.rs` + `src/workspace/settings_schemas.rs` — new
  `WorkspaceError::InvalidPath { path, reason }` variant. Path/key
  rejection (path-traversal, character set, length) now surfaces as
  `InvalidPath`, not `SchemaValidation` — callers and downstream UIs can
  distinguish "your settings *key* has bad characters" from "your
  settings *value* failed JSON-Schema validation" without string-matching
  error messages. `validate_settings_key` returns the new variant; the
  one match site in `settings_adapter.rs::write_to_workspace` is updated.
  Regression test `validate_settings_key_returns_invalid_path_variant`.

**Documentation-only fixes**

- `src/tools/dispatch.rs` — clarify in the `dispatch()` doc-comment that
  `sanitize_tool_output` runs only against the persisted ActionRecord
  payload, NOT against the value returned to the caller. This mirrors
  `Worker::execute_tool` (the agent loop also receives the raw output so
  reasoning can be reproduced from history). Channels that forward
  dispatcher output to end users must run their own boundary
  sanitization at the channel edge.

- `src/history/store.rs` + `src/db/libsql/jobs.rs` —
  `create_system_job` doc updated to explicitly state that system job
  timestamps do NOT reflect tool execution time (the row is INSERTed
  before the tool runs, with all three timestamps pinned to "now").
  Consumers that need execution duration must read
  `job_actions.duration_ms` for the associated action rows. Restructuring
  to a two-phase INSERT+UPDATE was rejected: the audit row must be
  durable even if the dispatcher panics mid-tool, and the second write
  would double per-dispatch DB cost.

- `src/workspace/schema.rs` — added baseline regression test
  `moderately_complex_schema_compiles_within_budget` that pins schema
  compile + validate latency for a moderately deep nested schema at
  <500ms wall-clock. Guards against orders-of-magnitude regressions
  from a future `jsonschema` upgrade or accidentally pathological
  schema construction. Hard limits on schema complexity are deferred
  (the real defense today is keeping schema-bearing paths under
  `.system/`, which is system-controlled).

**Acknowledged, no change**

- libSQL `create_system_job` unbounded row growth — already documented
  as intentional in the existing comment block, with the mitigation path
  spelled out (partial index on `WHERE category != 'system'` for listing
  queries). Rate-limiting dispatch would silently drop user-initiated
  actions, which is worse than unbounded retention. The "LLM data is
  never deleted" rule (CLAUDE.md) explicitly applies.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 00:02:05 +09:00

414 lines
14 KiB
Rust

#![cfg(feature = "libsql")]
//! Integration tests for layered memory using file-backed libSQL.
use std::sync::Arc;
use ironclaw::db::Database;
use ironclaw::db::libsql::LibSqlBackend;
use ironclaw::workspace::Workspace;
use ironclaw::workspace::layer::{LayerSensitivity, MemoryLayer};
use ironclaw::workspace::privacy::PatternPrivacyClassifier;
async fn setup() -> (Arc<dyn Database>, tempfile::TempDir) {
let dir = tempfile::tempdir().expect("create temp dir");
let db_path = dir.path().join("test.db");
let backend = LibSqlBackend::new_local(&db_path).await.expect("create db");
backend.run_migrations().await.expect("run migrations");
let db: Arc<dyn Database> = Arc::new(backend);
(db, dir)
}
fn test_layers() -> Vec<MemoryLayer> {
vec![
MemoryLayer {
name: "private".into(),
scope: "alice".into(),
writable: true,
sensitivity: LayerSensitivity::Private,
},
MemoryLayer {
name: "shared".into(),
scope: "shared".into(),
writable: true,
sensitivity: LayerSensitivity::Shared,
},
MemoryLayer {
name: "reports".into(),
scope: "reports".into(),
writable: false,
sensitivity: LayerSensitivity::Shared,
},
]
}
#[tokio::test]
async fn write_to_private_layer() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
let result = ws
.write_to_layer("private", "notes/test.md", "Private note", false)
.await
.expect("write should succeed");
assert_eq!(result.document.content, "Private note");
assert!(!result.redirected);
assert_eq!(result.actual_layer, "private");
}
#[tokio::test]
async fn write_to_shared_layer() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
let result = ws
.write_to_layer("shared", "plans/dinner.md", "Dinner Saturday at 6", false)
.await
.expect("write should succeed");
assert_eq!(result.document.content, "Dinner Saturday at 6");
assert!(!result.redirected);
assert_eq!(result.actual_layer, "shared");
}
#[tokio::test]
async fn write_to_read_only_layer_fails() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
let result = ws
.write_to_layer("reports", "notes/budget.md", "Some budget note", false)
.await;
assert!(result.is_err());
}
#[tokio::test]
async fn write_to_unknown_layer_fails() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
let result = ws
.write_to_layer("nonexistent", "notes/test.md", "content", false)
.await;
assert!(result.is_err());
}
#[tokio::test]
async fn no_redirect_without_classifier() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
// Without a classifier, PII goes exactly where requested
let result = ws
.write_to_layer("shared", "notes/pii.md", "My SSN is 123-45-6789", false)
.await
.expect("write should succeed");
assert!(!result.redirected);
assert_eq!(result.actual_layer, "shared");
}
#[tokio::test]
async fn sensitive_content_redirected_to_private() {
let (db, _dir) = setup().await;
let db_clone = db.clone();
let ws = Workspace::new_with_db("alice", db)
.with_memory_layers(test_layers())
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
// Write content containing hard PII to shared layer -- should be redirected
let result = ws
.write_to_layer("shared", "notes/pii.md", "My SSN is 123-45-6789", false)
.await
.expect("write should succeed (redirected)");
// WriteResult should indicate redirect to private layer
assert!(result.redirected, "Should be redirected");
assert_eq!(result.actual_layer, "private");
assert_eq!(result.document.content, "My SSN is 123-45-6789");
// Content should be in the private scope (alice), not the shared scope
let private_doc = ws.read("notes/pii.md").await;
assert!(
private_doc.is_ok(),
"Should find content in private scope (alice)"
);
assert_eq!(private_doc.unwrap().content, "My SSN is 123-45-6789");
// Verify content is NOT in the shared scope (same DB, different user_id)
let ws_shared = Workspace::new_with_db("shared", db_clone);
let shared_doc = ws_shared.read("notes/pii.md").await;
assert!(
shared_doc.is_err(),
"Should NOT find content in shared scope"
);
}
#[tokio::test]
async fn default_write_still_works() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
// Regular write (no layer) should still work
let doc = ws
.write("notes/test.md", "Regular note")
.await
.expect("write should succeed");
assert_eq!(doc.content, "Regular note");
}
#[tokio::test]
async fn layered_writes_record_actor_in_changed_by() {
// Regression: `write_to_layer` / `append_to_layer` previously passed the
// target layer's scope as `changed_by`, so version history attributed
// every layered edit to the layer name (e.g. "shared") instead of the
// user who actually wrote it. The fix passes `self.user_id` while still
// resolving metadata in the target layer's scope.
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
// First write — creates the document and (per maybe_save_version) records
// a version of the prior empty content; subsequent writes record versions
// of the prior content. We do two writes so we definitely have a version
// row to inspect.
let first = ws
.write_to_layer("shared", "plans/v.md", "v1", false)
.await
.expect("first write");
ws.write_to_layer("shared", "plans/v.md", "v2", false)
.await
.expect("second write");
let versions = ws
.list_versions(first.document.id, 10)
.await
.expect("list versions");
assert!(!versions.is_empty(), "expected at least one version row");
for v in &versions {
assert_eq!(
v.changed_by.as_deref(),
Some("alice"),
"changed_by should be the actor (alice), not the layer scope"
);
}
// Same check for append_to_layer.
let appended = ws
.append_to_layer("shared", "plans/v.md", "v3", false)
.await
.expect("append");
let versions = ws
.list_versions(appended.document.id, 10)
.await
.expect("list versions after append");
for v in &versions {
assert_eq!(
v.changed_by.as_deref(),
Some("alice"),
"append_to_layer should also attribute changed_by to the actor"
);
}
}
#[tokio::test]
async fn append_to_layer_works() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
// Write initial content to a layer
ws.write_to_layer("private", "notes/log.md", "Entry one", false)
.await
.expect("initial write should succeed");
// Append to the same layer path
let result = ws
.append_to_layer("private", "notes/log.md", "Entry two", false)
.await
.expect("append should succeed");
// Content should be concatenated with double newline
assert!(
result.document.content.contains("Entry one"),
"Should contain first entry"
);
assert!(
result.document.content.contains("Entry two"),
"Should contain second entry"
);
}
#[tokio::test]
async fn sensitive_content_fails_without_private_layer() {
let (db, _dir) = setup().await;
// Workspace with classifier but only shared layers (no private layer for redirect)
let shared_only_layers = vec![MemoryLayer {
name: "shared".into(),
scope: "shared".into(),
writable: true,
sensitivity: LayerSensitivity::Shared,
}];
let ws = Workspace::new_with_db("alice", db)
.with_memory_layers(shared_only_layers)
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
// Writing PII content should fail (no private layer to redirect to)
let result = ws
.write_to_layer("shared", "notes/pii.md", "My SSN is 123-45-6789", false)
.await;
assert!(
result.is_err(),
"Should fail when no private layer available for redirect"
);
}
#[tokio::test]
async fn append_sensitive_to_shared_redirects() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db)
.with_memory_layers(test_layers())
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
// Append PII content to shared layer -- should be redirected
let result = ws
.append_to_layer(
"shared",
"notes/pii.md",
"Card number is 4111 1111 1111 1111",
false,
)
.await
.expect("append should succeed (redirected)");
assert!(result.redirected, "Should be redirected");
assert_eq!(result.actual_layer, "private");
assert!(result.document.content.contains("4111"));
}
#[tokio::test]
async fn force_skips_privacy_redirect() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db)
.with_memory_layers(test_layers())
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
// PII content with force=true should stay in shared layer
let result = ws
.write_to_layer("shared", "notes/pii.md", "My SSN is 123-45-6789", true)
.await
.expect("write should succeed without redirect");
assert!(
!result.redirected,
"Should NOT be redirected with force=true"
);
assert_eq!(result.actual_layer, "shared");
}
#[tokio::test]
async fn search_finds_private_layer_content() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
// Write to the private layer (scope = "alice" = user_id)
ws.write_to_layer(
"private",
"notes/private.md",
"My private thought about waffles",
false,
)
.await
.unwrap();
// Search should find content in the primary scope
let results = ws.search("waffles", 10).await.unwrap();
assert!(
!results.is_empty(),
"Should find results in the private layer"
);
}
#[tokio::test]
async fn write_to_private_invisible_from_shared_scope() {
let (db, _dir) = setup().await;
let db_clone = db.clone();
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
ws.write_to_layer("private", "notes/secret.md", "Private data", false)
.await
.expect("write should succeed");
let ws_shared = Workspace::new_with_db("shared", db_clone);
let result = ws_shared.read("notes/secret.md").await;
assert!(
result.is_err(),
"Shared scope must not read private layer content"
);
}
#[tokio::test]
async fn write_to_shared_invisible_from_private_scope() {
let (db, _dir) = setup().await;
let db_clone = db.clone();
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
ws.write_to_layer("shared", "plans/visible.md", "Shared plan", false)
.await
.expect("write should succeed");
let ws_alice = Workspace::new_with_db("alice", db_clone);
let result = ws_alice.read("plans/visible.md").await;
assert!(
result.is_err(),
"Private scope must not read shared layer content without multi-scope"
);
}
#[tokio::test]
async fn write_empty_path_to_layer() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
let result = ws.write_to_layer("private", "", "content", false).await;
// normalize_path("") returns "" — the write succeeds with an empty-string path
assert!(result.is_ok(), "write with empty path should succeed");
let write_result = result.unwrap();
assert_eq!(write_result.document.content, "content");
assert!(!write_result.redirected);
assert_eq!(write_result.actual_layer, "private");
}
#[tokio::test]
async fn overwrite_existing_content_in_layer() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
ws.write_to_layer("private", "notes/evolving.md", "Version 1", false)
.await
.expect("first write");
let result = ws
.write_to_layer("private", "notes/evolving.md", "Version 2", false)
.await
.expect("overwrite should succeed");
assert_eq!(result.document.content, "Version 2");
assert!(!result.redirected);
}
#[tokio::test]
async fn sensitive_write_to_private_layer_not_redirected() {
let (db, _dir) = setup().await;
let ws = Workspace::new_with_db("alice", db)
.with_memory_layers(test_layers())
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
let result = ws
.write_to_layer("private", "notes/pii.md", "My SSN is 123-45-6789", false)
.await
.expect("write to private should succeed");
assert!(
!result.redirected,
"Private layer writes should not redirect"
);
assert_eq!(result.actual_layer, "private");
}