mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-03 08:06:01 +08:00
* feat(workspace): add JSON Schema validation to document metadata Add a `schema` field to `DocumentMetadata` that enables automatic content validation on workspace writes. When a document or its folder `.config` carries a JSON Schema, all write operations (write, append, patch, write_to_layer, append_to_layer) validate content against it before persisting. This is the foundation for typed system state (settings, extension configs, skill manifests) stored as workspace documents. Builds on the metadata infrastructure from #1723 — schema is inherited via the existing `.config` chain (folder → document → defaults). Refs: #640, #1894, #1937 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tools): add channel-agnostic ToolDispatcher with audit trail Introduce `ToolDispatcher` — a universal entry point for executing tools from any caller (gateway, CLI, routine engine, WASM channels). Creates lightweight system jobs for FK integrity, records ActionRecords, and returns ToolOutput. This is a third entry point alongside v1's Worker::execute_tool() and v2's EffectBridgeAdapter::execute_action(). DispatchSource::Channel(String) is intentionally string-typed — channels are interchangeable extensions that can appear at runtime. Also adds JobContext::system() factory and create_system_job() to both PostgreSQL and libSQL backends. Refs: #640 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(workspace): settings-as-workspace-documents with dual-write adapter Add WorkspaceSettingsAdapter that implements SettingsStore by reading/ writing workspace documents at _system/settings/{key}.json. During migration, dual-writes to both the legacy settings table and workspace. Reads prefer workspace, falling back to the legacy table. Known setting keys (llm_backend, selected_model, tool_permissions.*, etc.) get JSON Schemas stored in document metadata — writes are validated automatically by Phase 0's schema validation. Also adds settings_schemas.rs with compile-time schema registry and settings_path() helper. Refs: #640, #1937 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(gateway): wire ToolDispatcher into GatewayState Add tool_dispatcher field to GatewayState with with_tool_dispatcher() builder method. Create and wire the dispatcher in main.rs when both tool_registry and database are available. All 16 GatewayState construction sites updated. Per-handler migration (routing mutations through ToolDispatcher instead of direct DB calls) is deferred to follow-up PRs — each handler has complex ownership checks, cache refresh, and response types. Refs: #640 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(tools): add system introspection tools (tools_list, version) Add SystemToolsListTool and SystemVersionTool as proper Tool implementations that replace hardcoded /tools and /version commands. Registered at startup via register_system_tools(). Available in both v1 and v2 engines — no is_v1_only_tool filter to worry about. Refs: #640 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(workspace): extension and skill state schemas and path helpers Add workspace path helpers and JSON Schemas for storing extension configs, extension state, and skill manifests under _system/extensions/ and _system/skills/. This establishes the workspace document structure that ExtensionManager and SkillRegistry will use as a durable persistence backend (read-through cache pattern). Runtime state (active MCP connections, WASM runtimes) stays in memory. Only durable config and activation state moves to workspace documents. Refs: #640, #1741 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review feedback and CI failures CI fixes: - deny.toml: allow MIT-0 license required by jsonschema - workspace/document.rs: #[allow(dead_code)] on system path constants pending follow-up phases that consume them - workspace/settings_adapter.rs: remove unused chrono::Utc import - workspace/settings_adapter.rs: collapse nested if into && form Review fixes (gemini-code-assist): - tools/dispatch.rs: await save_action directly instead of fire-and-forget tokio::spawn so short-lived CLI callers cannot drop audit records before they are persisted; surface errors via tracing::warn - tools/dispatch.rs: remove DispatchSource::Agent variant — sequence_num=0 with a reused job_id would violate UNIQUE(job_id, sequence_num). Agent callers must use Worker::execute_tool() which manages sequence numbers atomically against the agent's existing job - workspace/settings_adapter.rs: validate content against the schema BEFORE the first workspace write so the initial document creation cannot bypass schema enforcement (subsequent writes are validated by the workspace resolved-metadata path established after the first write) Refs: #2049 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: unify all machine state under .system/ Rename the workspace prefix from `_system/` to `.system/` (Unix dot-prefix convention for hidden internal state) and migrate v2 engine state from `engine/` to `.system/engine/` so all machine-managed state lives under one root. New layout: .system/ ├── settings/ (per-user settings as workspace docs) ├── extensions/ (extension config + activation state) ├── skills/ (skill manifests) └── engine/ ├── README.md (auto-generated index) ├── knowledge/ (lessons, skills, summaries, specs, issues) ├── orchestrator/ (Python orchestrator versions, failures, overlays) ├── projects/ (project files + nested missions/) └── runtime/ (threads, steps, events, leases, conversations) The inner `.runtime/` dot-prefix is dropped under `.system/engine/` since `.system/` itself is the hidden marker; no double-hiding needed. The `ENGINE_PREFIX` constant in `workspace::document::system_paths` is declared as the canonical convention; bridge `store_adapter` continues to define per-subdirectory constants below it for ergonomic interpolation. No legacy migration code — pre-production rename. Refs: #2049 * fix(pr-2049): security, correctness, and robustness fixes from review Critical security: - dispatch.rs: redact sensitive params before persisting ActionRecord (was leaking plaintext secrets into the audit log for tools with sensitive_params()) - settings_schemas.rs: validate settings keys against path traversal (reject /, \, .., leading ., empty, length > 128, non-alphanumeric); wire validation into all settings_adapter read/write/delete paths Data correctness: - history/store.rs + libsql/jobs.rs: write status as JobState::Completed .to_string() ('completed' snake_case) instead of 'Completed'; system jobs were round-tripping as Pending in parse_job_state() - settings_adapter.rs: fix .system/.config metadata to set skip_versioning: false (was true) — descendants inherit this via find_nearest_config, so the previous value silently disabled versioning for ALL .system/** documents, contradicting the audit- trail intent - workspace/mod.rs: add resolve_metadata_in_scope; use it in write_to_layer / append_to_layer so non-primary layer writes resolve schema/indexing/versioning from the target layer's .config chain instead of the primary user_id's. Also pass &scope (not &self.user_id) to maybe_save_version so versions are attributed to the correct scope Pipeline parity: - dispatch.rs: add SafetyLayer to ToolDispatcher; mirror Worker pipeline (prepare_tool_params -> validator -> redact -> timeout -> sanitize output) so dispatch path gets the same safety guarantees as the agent worker. Sanitized output is now stored in ActionRecord.output_sanitized instead of duplicating raw JSON Robustness: - settings_adapter.rs: propagate update_metadata errors in ensure_system_config and write_to_workspace (was silently ignored via let _ =, leaving schemas/skip_indexing unenforced) - settings_adapter.rs: set_all_settings now collects the first workspace write error and returns it after the legacy write completes, so partial-migration state is observable - settings_schemas.rs: rewrite llm_custom_providers schema to match CustomLlmProviderSettings (id/name/adapter/base_url/default_model/ api_key/builtin instead of stale name/protocol/base_url/model) Build: - Cargo.toml: jsonschema with default-features = false to avoid pulling a second reqwest major version Docs: - db/mod.rs: docstring for create_system_job uses 'completed' snake_case - workspace/document.rs: clarify .system/ versioning ("by default ARE versioned; individual files may opt out via skip_versioning") - settings_adapter.rs: clarify per-key reads prefer workspace, aggregate reads stay on legacy during migration - tools/builtin/system.rs: trim doc to match implemented scope (system_tools_list, system_version) - channels/web/mod.rs: move stale 'sweep tasks managed by with_oauth' comment back to oauth_sweep_shutdown line Refs: #2049 * docs+ci: enforce 'everything goes through tools' principle Document the core design principle from #2049 in two places so future contributors (human and AI) discover it during development: - CLAUDE.md: new "Everything Goes Through Tools" section near the "Adding a New Channel" guide. Includes the rule, the rationale (audit trail, safety pipeline parity, channel-agnostic surface, agent parity), and a pointer to the detailed rule file. - .claude/rules/tools.md: full pattern with required/forbidden examples, the list of layers that ARE exempt (Worker::execute_tool, v2 EffectBridgeAdapter, tool implementations themselves, background engine jobs, read-aggregation queries), and how to annotate intentional exceptions. Also extends `paths` to cover src/channels/** and src/cli/** so it surfaces when those files are edited. Enforce with a new pre-commit safety check (#7) in scripts/pre-commit-safety.sh: - Scans newly added lines under src/channels/web/handlers/*.rs and src/cli/*.rs for direct touches of state.{store, workspace, workspace_pool, extension_manager, skill_registry, session_manager}. - Suppress with a trailing `// dispatch-exempt: <reason>` comment on the same line, matching the existing `// safety:` convention. - Only checks added lines (`+` in the diff), so existing untouched handlers don't trip the check during incremental migration. The check fires only for new code: handlers that haven't been migrated yet (52 existing direct accesses across 12 handler files) won't break unmodified, but any new line that bypasses the dispatcher will be flagged at commit time. Refs: #2049 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(pr-2049): address Copilot review on workspace schema layer - workspace::extension_state: extension/skill path helpers now reuse the canonical name validators (`canonicalize_extension_name`, `validate_skill_name`) instead of a weak `replace('/', "_")`. Names containing `..`, `\`, NUL, or other escapes are now rejected at the helper boundary, eliminating a path-traversal foothold for callers. Helpers return `Result<String, PathError>`. Regression tests added. - workspace::settings_adapter::ensure_system_config: now idempotent across upgrades. If `.system/.config` already exists with stale metadata (e.g. an older `skip_versioning: true` from before fix #3042846635), it is repaired to the expected inherited values instead of being left silently broken. Regression test added. - workspace::settings_adapter::write_to_workspace: lazily seeds `.system/.config` via a `OnceCell`, so callers no longer need to remember to invoke `ensure_system_config()` at startup before any setting write. Regression test added. - workspace::settings_adapter::delete_setting: workspace delete failures are now logged via `tracing::warn!` instead of being silently dropped. We still don't propagate the error — the legacy table is the source of truth during migration and a stale workspace doc is recoverable on the next write — but partial-delete state is now observable. - workspace::schema: documented why we don't cache compiled validators yet (settings/extension/skill writes are not a hot path; revisit if schema validation moves into a frequent write path). [skip-regression-check] schema.rs change is doc-only. * fix(pr-2049): address 4 remaining review issues 1. tool_dispatcher dropped during gateway startup src/channels/web/mod.rs: rebuild_state was initializing tool_dispatcher to None, so every subsequent with_* call zeroed the dispatcher the first caller injected. Preserve it across rebuild_state like every other field. Regression test: tool_dispatcher_survives_subsequent_with_calls. 2. WorkspaceSettingsAdapter not wired into runtime src/app.rs: Build the adapter in build_all() when workspace+db are both present, eagerly call ensure_system_config(), expose on AppComponents as settings_store, and thread it into init_extensions(...) so register_permission_tools and upgrade_tool_list receive it instead of the raw db. src/main.rs: SIGHUP handler prefers the adapter over raw db. src/workspace/mod.rs: re-export WorkspaceSettingsAdapter. 3. changed_by regression on layered writes src/workspace/mod.rs: write_to_layer and append_to_layer were passing the target layer's scope as changed_by, so version history attributed layered edits to the layer name instead of the actor. Pass self.user_id while keeping metadata resolution in the target scope. Regression test: layered_writes_record_actor_in_changed_by. 4. Legacy engine/ paths invisible after upgrade src/bridge/store_adapter.rs: Add migrate_legacy_engine_paths(), called at the start of load_state_from_workspace(), which scans list_all() for engine/... documents and rewrites them to .system/engine/... Idempotent: skips rewrites when the new path already exists, deletes the legacy duplicate either way. Three regression tests in #[cfg(all(test, feature = "libsql"))] module. Quality gate: cargo fmt, cargo clippy --all --all-features zero warnings, cargo test --all-features --lib 4313 passed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(e2e): use PUT for settings write in ownership test test_settings_written_and_readable was sending POST /api/settings/{key} but the route has been PUT since #4 (Feb 2026) — the test was returning 405 Method Not Allowed. Switch to httpx.put() so it matches the current route registration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(pr-2049): address second round of review feedback Addresses the remaining unresolved PR #2049 review comments from serrrfirat and ilblackdragon. ## Changes ### ToolDispatcher — integration coverage + log level - src/tools/dispatch.rs: add two libsql-gated integration tests for the full dispatch pipeline: (a) persist an ActionRecord with sensitive params redacted in the audit row while the tool still sees the raw value, sanitized output populated; (b) honor the per-tool execution_timeout() and record a failure action. - Tests use a raw-SQL helper to find system-category jobs since list_agent_jobs_for_user intentionally filters them out. - Replace warn! with debug! on audit persistence failure — dispatch is reachable from interactive CLI/REPL sessions where warn!/info! output corrupts the terminal UI (CLAUDE.md Code Style → logging). ### WorkspaceSettingsAdapter — log level - src/workspace/settings_adapter.rs: same warn! → debug! fix on the delete_setting workspace failure path, for the same REPL reason. ### Schema validation — surface all errors - src/workspace/schema.rs: switch from jsonschema::validate to validator_for + iter_errors so users fixing a malformed setting see every violation in one round instead of playing whack-a-mole. Also distinguishes "invalid schema" from "invalid content" errors. - Regression tests: multiple_errors_are_all_reported and invalid_schema_is_distinguished_from_invalid_content. ### create_system_job — started_at + row growth docs - src/db/libsql/jobs.rs and src/history/store.rs: include started_at in the INSERT (set to the same instant as created_at/completed_at) so duration queries don't see NULL and "started but not completed" filters don't misclassify these rows. Fixed in both backends. - Add doc comments on both impls warning about row growth per dispatch call. Deleting rows would violate "LLM data is never deleted" (CLAUDE.md); if listing-query performance becomes a concern, prefer a partial index (WHERE category != 'system') over deletion. ### Lib test repair - src/channels/web/server.rs: extensions_setup_submit_handler Err branch now sets resp.activated = Some(false) so clients and the regression test see an explicit `false` rather than `null`. Also rename the test's fake channel to snake_case (test_failing_channel) so it matches the canonicalize-extension-names behavior from PR #2129 — previously the test was passing a dashed name and getting "Capabilities file not found" instead of the intended activation failure. ## Not addressed (false positive / deferred) - dispatch.rs:177 output_raw/output_sanitized swap — verified against ActionRecord::succeed(Option<String>, Value, Duration) and the worker's call site at job.rs:704; argument order is correct. - settings_adapter.rs:186 TOCTOU window — author self-classified as "Low / completeness" and no other code path writes to .system/settings/** without going through write_to_workspace. - schema.rs recompilation caching — deferred per earlier review. ## Quality gate - cargo fmt - cargo clippy --all --benches --tests --examples --all-features zero warnings - cargo test --all-features --lib: 4387 passed, 0 failed, 3 ignored Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(pr-2049): address third round of review feedback Addresses unresolved comments from serrrfirat's "Paranoid Architect Review" and Copilot's third pass on the engine-state migration. ## src/workspace/settings_adapter.rs ### HIGH — Cross-tenant data leak through owner-scoped Workspace `Workspace` is constructed for a single user_id at AppBuilder time. Without gating, `set_setting("user_B", key, val)` would dual-write into the **owner's** workspace, and a subsequent `user_A.get_setting(...)` would return user_B's value: a real cross-user data leak. Fix: - Add `gate_user_id` field set to `workspace.user_id()` at construction. - All `SettingsStore` methods that touch the workspace now check `workspace_allowed_for(user_id)` first; non-owner callers fall through to the legacy table only — preserving their pre-#2049 behavior. - This matches the long-term plan: per-user settings live in the legacy table until a per-user `WorkspaceSettingsAdapter` (one per WorkspacePool entry) is wired up; admin/global settings go through the workspace-backed path so they pick up schema validation. Regression test: `workspace_settings_are_owner_gated_in_multi_tenant_mode` asserts (a) owner's workspace doc is not overwritten by a non-owner write, (b) each user reads back their own legacy value, and (c) a non-owner with no legacy entry must NOT see the owner's workspace value bleeding through. ### MEDIUM — Dual-write order Reverse `set_setting` and `set_all_settings` to write legacy first, workspace second. The legacy table is the source of truth during migration (it backs aggregate `list_settings` reads), so writing it first guarantees those readers always see a consistent value even if the workspace write fails. Failed workspace writes are self-healing on the next per-key read-miss. ### MEDIUM — `ensure_system_config_lazy` double-execution race Replace the manual `get()`/`set()` pattern with `OnceCell::get_or_try_init`. Two concurrent first-callers no longer both run `ensure_system_config()`. Functionally equivalent (idempotent either way) but no longer wasteful. ## src/bridge/store_adapter.rs ### MEDIUM — Migration drops document metadata (S3) `migrate_legacy_engine_paths` previously copied only `doc.content`, silently dropping the `metadata` column. Now calls `ws.update_metadata(new_doc.id, &doc.metadata)` after each write to preserve schema/skip_indexing/hygiene flags. Logged-not-fatal: content has already been moved, metadata loss is recoverable. Regression test: `migration_preserves_document_metadata` seeds a doc with custom metadata and asserts it survives the rewrite. ### MEDIUM — `ws.exists()` swallowed transient errors (Copilot) `unwrap_or(false)` on the existence check could cause the migrator to overwrite an existing `.system/engine/...` doc when storage hiccups. Now propagates the error (counts as failed step + `continue`), per Copilot's exact suggested patch. ### LOW — `list_all()` runs every startup (Copilot) Add a cheap preflight: `ws.list("engine")` first; only fall through to the recursive `list_all()` discovery when the directory listing returns at least one entry. Steady-state startups (post-migration) skip the full workspace scan entirely. Regression test: `migration_preflight_skips_full_scan_when_no_legacy_paths` asserts unrelated and already-migrated documents are untouched. ### MEDIUM — Counter undercount on `already_present` (S5) When `already_present` is true the legacy duplicate is still deleted, but the previous code skipped the `migrated += 1` increment, undercounting in debug logs. Fixed: `migrated` now counts every successful path migration including the already-present case. ### Documented — Version-history loss is acceptable scope (C1) Read-write-delete pattern means `memory_document_versions.document_id ON DELETE CASCADE` drops the legacy doc's version chain. Documented in the function-level doc comment as intentional + bounded: - v2 engine state is runtime state (rewritten on every mutation), not user-curated data - v2 was newly introduced in this PR — no production deployment with pre-existing curated history at risk - A path-preserving rename op would need new trait methods on both backends; out of scope for fix-forward. If a future caller needs history-preserving rename, it should be added to the storage layer properly, not bolted onto migration. ## Quality gate - cargo fmt - cargo clippy --all --benches --tests --examples --all-features zero warnings - cargo test --all-features --lib: 4390 passed, 0 failed, 3 ignored (+3 new tests on top of round 2) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(pr-2049): address fourth round of review feedback Two latent issues flagged by serrrfirat in the latest review pass: 1. **Null schema permanently locks documents** (`src/workspace/schema.rs`). `serde_json` deserializes a metadata field of `"schema": null` as `Some(Value::Null)`, not `None`, so the upstream `if let Some(schema) = &metadata.schema` check passes through to `validate_content_against_schema`. There, `validator_for(Value::Null)` errors out and every subsequent write to that document is blocked — a latent DoS. Added an explicit `schema.is_null()` early-return guard at the top of the validator, plus a regression test (`null_schema_is_treated_as_no_op`) that asserts even non-JSON content passes when the schema is null. 2. **System job titles were raw source labels** (`src/history/store.rs`, `src/db/libsql/jobs.rs`). `create_system_job` set `title = source`, so any UI rendering `agent_jobs.title` would display dispatched system jobs as `channel:gateway` / `system` / etc. instead of a human-readable label. Both PostgreSQL and libSQL backends now write `format!("System: {source}")`. Updated the two dispatch integration tests that pinned the old format. Schema-recompilation comment (`schema.rs:47`) was acknowledged as "acceptable for now" by the reviewer; existing NOTE in the source already documents the caching trade-off and upgrade path, so no code change. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(pr-2049): address fifth round of review feedback Eight comments from Copilot + serrrfirat. Real fixes for the load-bearing gaps; doc clarifications for the rest where the existing behavior is intentional. **Real code changes** - `src/tools/dispatch.rs` — enforce `tool.parameters_schema()` (JSON Schema) in the dispatch path. Previously the SafetyLayer validator only checked for injection patterns; channel/CLI/routine callers could pass arbitrary shapes and only discover the mismatch (or worse, silently malformed behavior) inside the tool itself. Now we run `jsonschema::validate(&tool.parameters_schema(), &normalized_params)` after the injection check, with a permissive-empty-schema fast path so tools that haven't yet declared a schema aren't penalised. Regression test `dispatch_rejects_params_violating_tool_schema` asserts a required-field violation is rejected before the tool is invoked. - `src/workspace/settings_adapter.rs` — `write_to_workspace` now calls `schema_for_key(key)` once and reuses the resolved schema for both pre-write validation and post-write metadata persistence (was called twice). Eliminates duplicate work and removes a theoretical divergence window if the schema registry ever became non-deterministic. - `src/workspace/settings_adapter.rs` — `ensure_system_config` now also rewrites the `.config` document content when its metadata is repaired, not just the metadata column. The metadata column is the inheritance source of truth, but having the doc's content silently diverge from it confuses anyone reading the doc directly to understand which inherited flags are active. - `src/error.rs` + `src/workspace/settings_schemas.rs` — new `WorkspaceError::InvalidPath { path, reason }` variant. Path/key rejection (path-traversal, character set, length) now surfaces as `InvalidPath`, not `SchemaValidation` — callers and downstream UIs can distinguish "your settings *key* has bad characters" from "your settings *value* failed JSON-Schema validation" without string-matching error messages. `validate_settings_key` returns the new variant; the one match site in `settings_adapter.rs::write_to_workspace` is updated. Regression test `validate_settings_key_returns_invalid_path_variant`. **Documentation-only fixes** - `src/tools/dispatch.rs` — clarify in the `dispatch()` doc-comment that `sanitize_tool_output` runs only against the persisted ActionRecord payload, NOT against the value returned to the caller. This mirrors `Worker::execute_tool` (the agent loop also receives the raw output so reasoning can be reproduced from history). Channels that forward dispatcher output to end users must run their own boundary sanitization at the channel edge. - `src/history/store.rs` + `src/db/libsql/jobs.rs` — `create_system_job` doc updated to explicitly state that system job timestamps do NOT reflect tool execution time (the row is INSERTed before the tool runs, with all three timestamps pinned to "now"). Consumers that need execution duration must read `job_actions.duration_ms` for the associated action rows. Restructuring to a two-phase INSERT+UPDATE was rejected: the audit row must be durable even if the dispatcher panics mid-tool, and the second write would double per-dispatch DB cost. - `src/workspace/schema.rs` — added baseline regression test `moderately_complex_schema_compiles_within_budget` that pins schema compile + validate latency for a moderately deep nested schema at <500ms wall-clock. Guards against orders-of-magnitude regressions from a future `jsonschema` upgrade or accidentally pathological schema construction. Hard limits on schema complexity are deferred (the real defense today is keeping schema-bearing paths under `.system/`, which is system-controlled). **Acknowledged, no change** - libSQL `create_system_job` unbounded row growth — already documented as intentional in the existing comment block, with the mitigation path spelled out (partial index on `WHERE category != 'system'` for listing queries). Rate-limiting dispatch would silently drop user-initiated actions, which is worse than unbounded retention. The "LLM data is never deleted" rule (CLAUDE.md) explicitly applies. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
414 lines
14 KiB
Rust
414 lines
14 KiB
Rust
#![cfg(feature = "libsql")]
|
|
//! Integration tests for layered memory using file-backed libSQL.
|
|
|
|
use std::sync::Arc;
|
|
|
|
use ironclaw::db::Database;
|
|
use ironclaw::db::libsql::LibSqlBackend;
|
|
use ironclaw::workspace::Workspace;
|
|
use ironclaw::workspace::layer::{LayerSensitivity, MemoryLayer};
|
|
use ironclaw::workspace::privacy::PatternPrivacyClassifier;
|
|
|
|
async fn setup() -> (Arc<dyn Database>, tempfile::TempDir) {
|
|
let dir = tempfile::tempdir().expect("create temp dir");
|
|
let db_path = dir.path().join("test.db");
|
|
let backend = LibSqlBackend::new_local(&db_path).await.expect("create db");
|
|
backend.run_migrations().await.expect("run migrations");
|
|
let db: Arc<dyn Database> = Arc::new(backend);
|
|
(db, dir)
|
|
}
|
|
|
|
fn test_layers() -> Vec<MemoryLayer> {
|
|
vec![
|
|
MemoryLayer {
|
|
name: "private".into(),
|
|
scope: "alice".into(),
|
|
writable: true,
|
|
sensitivity: LayerSensitivity::Private,
|
|
},
|
|
MemoryLayer {
|
|
name: "shared".into(),
|
|
scope: "shared".into(),
|
|
writable: true,
|
|
sensitivity: LayerSensitivity::Shared,
|
|
},
|
|
MemoryLayer {
|
|
name: "reports".into(),
|
|
scope: "reports".into(),
|
|
writable: false,
|
|
sensitivity: LayerSensitivity::Shared,
|
|
},
|
|
]
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn write_to_private_layer() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
let result = ws
|
|
.write_to_layer("private", "notes/test.md", "Private note", false)
|
|
.await
|
|
.expect("write should succeed");
|
|
assert_eq!(result.document.content, "Private note");
|
|
assert!(!result.redirected);
|
|
assert_eq!(result.actual_layer, "private");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn write_to_shared_layer() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
let result = ws
|
|
.write_to_layer("shared", "plans/dinner.md", "Dinner Saturday at 6", false)
|
|
.await
|
|
.expect("write should succeed");
|
|
assert_eq!(result.document.content, "Dinner Saturday at 6");
|
|
assert!(!result.redirected);
|
|
assert_eq!(result.actual_layer, "shared");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn write_to_read_only_layer_fails() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
let result = ws
|
|
.write_to_layer("reports", "notes/budget.md", "Some budget note", false)
|
|
.await;
|
|
assert!(result.is_err());
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn write_to_unknown_layer_fails() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
let result = ws
|
|
.write_to_layer("nonexistent", "notes/test.md", "content", false)
|
|
.await;
|
|
assert!(result.is_err());
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn no_redirect_without_classifier() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
// Without a classifier, PII goes exactly where requested
|
|
let result = ws
|
|
.write_to_layer("shared", "notes/pii.md", "My SSN is 123-45-6789", false)
|
|
.await
|
|
.expect("write should succeed");
|
|
assert!(!result.redirected);
|
|
assert_eq!(result.actual_layer, "shared");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn sensitive_content_redirected_to_private() {
|
|
let (db, _dir) = setup().await;
|
|
let db_clone = db.clone();
|
|
let ws = Workspace::new_with_db("alice", db)
|
|
.with_memory_layers(test_layers())
|
|
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
|
|
|
|
// Write content containing hard PII to shared layer -- should be redirected
|
|
let result = ws
|
|
.write_to_layer("shared", "notes/pii.md", "My SSN is 123-45-6789", false)
|
|
.await
|
|
.expect("write should succeed (redirected)");
|
|
|
|
// WriteResult should indicate redirect to private layer
|
|
assert!(result.redirected, "Should be redirected");
|
|
assert_eq!(result.actual_layer, "private");
|
|
assert_eq!(result.document.content, "My SSN is 123-45-6789");
|
|
|
|
// Content should be in the private scope (alice), not the shared scope
|
|
let private_doc = ws.read("notes/pii.md").await;
|
|
assert!(
|
|
private_doc.is_ok(),
|
|
"Should find content in private scope (alice)"
|
|
);
|
|
assert_eq!(private_doc.unwrap().content, "My SSN is 123-45-6789");
|
|
|
|
// Verify content is NOT in the shared scope (same DB, different user_id)
|
|
let ws_shared = Workspace::new_with_db("shared", db_clone);
|
|
let shared_doc = ws_shared.read("notes/pii.md").await;
|
|
assert!(
|
|
shared_doc.is_err(),
|
|
"Should NOT find content in shared scope"
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn default_write_still_works() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
// Regular write (no layer) should still work
|
|
let doc = ws
|
|
.write("notes/test.md", "Regular note")
|
|
.await
|
|
.expect("write should succeed");
|
|
assert_eq!(doc.content, "Regular note");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn layered_writes_record_actor_in_changed_by() {
|
|
// Regression: `write_to_layer` / `append_to_layer` previously passed the
|
|
// target layer's scope as `changed_by`, so version history attributed
|
|
// every layered edit to the layer name (e.g. "shared") instead of the
|
|
// user who actually wrote it. The fix passes `self.user_id` while still
|
|
// resolving metadata in the target layer's scope.
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
// First write — creates the document and (per maybe_save_version) records
|
|
// a version of the prior empty content; subsequent writes record versions
|
|
// of the prior content. We do two writes so we definitely have a version
|
|
// row to inspect.
|
|
let first = ws
|
|
.write_to_layer("shared", "plans/v.md", "v1", false)
|
|
.await
|
|
.expect("first write");
|
|
ws.write_to_layer("shared", "plans/v.md", "v2", false)
|
|
.await
|
|
.expect("second write");
|
|
|
|
let versions = ws
|
|
.list_versions(first.document.id, 10)
|
|
.await
|
|
.expect("list versions");
|
|
assert!(!versions.is_empty(), "expected at least one version row");
|
|
for v in &versions {
|
|
assert_eq!(
|
|
v.changed_by.as_deref(),
|
|
Some("alice"),
|
|
"changed_by should be the actor (alice), not the layer scope"
|
|
);
|
|
}
|
|
|
|
// Same check for append_to_layer.
|
|
let appended = ws
|
|
.append_to_layer("shared", "plans/v.md", "v3", false)
|
|
.await
|
|
.expect("append");
|
|
let versions = ws
|
|
.list_versions(appended.document.id, 10)
|
|
.await
|
|
.expect("list versions after append");
|
|
for v in &versions {
|
|
assert_eq!(
|
|
v.changed_by.as_deref(),
|
|
Some("alice"),
|
|
"append_to_layer should also attribute changed_by to the actor"
|
|
);
|
|
}
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn append_to_layer_works() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
// Write initial content to a layer
|
|
ws.write_to_layer("private", "notes/log.md", "Entry one", false)
|
|
.await
|
|
.expect("initial write should succeed");
|
|
|
|
// Append to the same layer path
|
|
let result = ws
|
|
.append_to_layer("private", "notes/log.md", "Entry two", false)
|
|
.await
|
|
.expect("append should succeed");
|
|
|
|
// Content should be concatenated with double newline
|
|
assert!(
|
|
result.document.content.contains("Entry one"),
|
|
"Should contain first entry"
|
|
);
|
|
assert!(
|
|
result.document.content.contains("Entry two"),
|
|
"Should contain second entry"
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn sensitive_content_fails_without_private_layer() {
|
|
let (db, _dir) = setup().await;
|
|
|
|
// Workspace with classifier but only shared layers (no private layer for redirect)
|
|
let shared_only_layers = vec![MemoryLayer {
|
|
name: "shared".into(),
|
|
scope: "shared".into(),
|
|
writable: true,
|
|
sensitivity: LayerSensitivity::Shared,
|
|
}];
|
|
let ws = Workspace::new_with_db("alice", db)
|
|
.with_memory_layers(shared_only_layers)
|
|
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
|
|
|
|
// Writing PII content should fail (no private layer to redirect to)
|
|
let result = ws
|
|
.write_to_layer("shared", "notes/pii.md", "My SSN is 123-45-6789", false)
|
|
.await;
|
|
assert!(
|
|
result.is_err(),
|
|
"Should fail when no private layer available for redirect"
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn append_sensitive_to_shared_redirects() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db)
|
|
.with_memory_layers(test_layers())
|
|
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
|
|
|
|
// Append PII content to shared layer -- should be redirected
|
|
let result = ws
|
|
.append_to_layer(
|
|
"shared",
|
|
"notes/pii.md",
|
|
"Card number is 4111 1111 1111 1111",
|
|
false,
|
|
)
|
|
.await
|
|
.expect("append should succeed (redirected)");
|
|
|
|
assert!(result.redirected, "Should be redirected");
|
|
assert_eq!(result.actual_layer, "private");
|
|
assert!(result.document.content.contains("4111"));
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn force_skips_privacy_redirect() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db)
|
|
.with_memory_layers(test_layers())
|
|
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
|
|
|
|
// PII content with force=true should stay in shared layer
|
|
let result = ws
|
|
.write_to_layer("shared", "notes/pii.md", "My SSN is 123-45-6789", true)
|
|
.await
|
|
.expect("write should succeed without redirect");
|
|
|
|
assert!(
|
|
!result.redirected,
|
|
"Should NOT be redirected with force=true"
|
|
);
|
|
assert_eq!(result.actual_layer, "shared");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn search_finds_private_layer_content() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
// Write to the private layer (scope = "alice" = user_id)
|
|
ws.write_to_layer(
|
|
"private",
|
|
"notes/private.md",
|
|
"My private thought about waffles",
|
|
false,
|
|
)
|
|
.await
|
|
.unwrap();
|
|
|
|
// Search should find content in the primary scope
|
|
let results = ws.search("waffles", 10).await.unwrap();
|
|
assert!(
|
|
!results.is_empty(),
|
|
"Should find results in the private layer"
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn write_to_private_invisible_from_shared_scope() {
|
|
let (db, _dir) = setup().await;
|
|
let db_clone = db.clone();
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
ws.write_to_layer("private", "notes/secret.md", "Private data", false)
|
|
.await
|
|
.expect("write should succeed");
|
|
|
|
let ws_shared = Workspace::new_with_db("shared", db_clone);
|
|
let result = ws_shared.read("notes/secret.md").await;
|
|
assert!(
|
|
result.is_err(),
|
|
"Shared scope must not read private layer content"
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn write_to_shared_invisible_from_private_scope() {
|
|
let (db, _dir) = setup().await;
|
|
let db_clone = db.clone();
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
ws.write_to_layer("shared", "plans/visible.md", "Shared plan", false)
|
|
.await
|
|
.expect("write should succeed");
|
|
|
|
let ws_alice = Workspace::new_with_db("alice", db_clone);
|
|
let result = ws_alice.read("plans/visible.md").await;
|
|
assert!(
|
|
result.is_err(),
|
|
"Private scope must not read shared layer content without multi-scope"
|
|
);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn write_empty_path_to_layer() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
let result = ws.write_to_layer("private", "", "content", false).await;
|
|
// normalize_path("") returns "" — the write succeeds with an empty-string path
|
|
assert!(result.is_ok(), "write with empty path should succeed");
|
|
let write_result = result.unwrap();
|
|
assert_eq!(write_result.document.content, "content");
|
|
assert!(!write_result.redirected);
|
|
assert_eq!(write_result.actual_layer, "private");
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn overwrite_existing_content_in_layer() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db).with_memory_layers(test_layers());
|
|
|
|
ws.write_to_layer("private", "notes/evolving.md", "Version 1", false)
|
|
.await
|
|
.expect("first write");
|
|
|
|
let result = ws
|
|
.write_to_layer("private", "notes/evolving.md", "Version 2", false)
|
|
.await
|
|
.expect("overwrite should succeed");
|
|
|
|
assert_eq!(result.document.content, "Version 2");
|
|
assert!(!result.redirected);
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn sensitive_write_to_private_layer_not_redirected() {
|
|
let (db, _dir) = setup().await;
|
|
let ws = Workspace::new_with_db("alice", db)
|
|
.with_memory_layers(test_layers())
|
|
.with_privacy_classifier(Arc::new(PatternPrivacyClassifier::new().unwrap()));
|
|
|
|
let result = ws
|
|
.write_to_layer("private", "notes/pii.md", "My SSN is 123-45-6789", false)
|
|
.await
|
|
.expect("write to private should succeed");
|
|
|
|
assert!(
|
|
!result.redirected,
|
|
"Private layer writes should not redirect"
|
|
);
|
|
assert_eq!(result.actual_layer, "private");
|
|
}
|