mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-03 08:06:01 +08:00
* feat(processes): route FilesystemProcessStore through unified put/get First consumer migration onto the new RootFilesystem surface. Switches the byte-plane read_file/write_file calls inside ironclaw_processes' filesystem-backed store to the unified put/get ops with Entry::bytes + CasExpectation::Any. The on-disk JSON layout is unchanged, every existing test passes, and downstream crates that construct FilesystemProcessStore (ironclaw_host_runtime + tests) don't need to change. Scope deliberately narrow: opaque-file entries through `put`/`get` without record kinds or non-`Any` CAS, since LocalFilesystem's native `put` only accepts that shape (per the foundation PR #3659). Once LocalFilesystem grows sidecar metadata, this consumer can switch to `Entry::record(process_record_kind, ...)` + `CasExpectation::Absent` without changing the on-disk layout. Touch points: - write_record uses put(Entry::bytes, CAS::Any) - start uses get for the existence probe + transition_lock for the atomicity envelope per the single-instance invariant - update_status / get / records_for_scope read via get and unwrap VersionedEntry.body - records_for_scope returns ProcessError::Filesystem (not silent skip) when get returns None for a path that list_dir just yielded — matches the pre-migration NotFound propagation invariant Test scaffold update: BackendErrorFilesystem now overrides `get` too, so the fault-propagation regression test continues to exercise its intended path. (Reviewer P1/P2 on the original #3666 — recursion + silent-skip — addressed in foundation #3659 directly since LocalFilesystem now ships native `put`/`get`.) * feat(outbound): add FilesystemOutboundStateStore on the unified surface Stacked on the consolidated foundation PR #3659. Adds an OutboundStateStore impl that persists outbound metadata under /engine/outbound/{policies,subscriptions,deliveries} through any RootFilesystem. The existing libSQL/Postgres/in-memory stores stay intact during the migration; a follow-up cleanup PR can delete them once production runs on the unified surface. The new store passes the full contract suite (durable_policy_*, subscription_cursor_*, delivery_status_*, notification_policy_*, full_turn_scope_isolation) against InMemoryBackend in the existing outbound_state_store_contract.rs test file. * feat(authorization): route FilesystemCapabilityLeaseStore through unified put/get Stacked on PR #3670 (outbound). Mirrors the ironclaw_processes migration in PR #3666 / now consolidated into #3659. Switches the filesystem-backed lease store's read_file/write_file calls to the unified get/put ops with Entry::bytes + CasExpectation::Any. The on-disk JSON layout is unchanged, every existing test passes, and the per-owner mutation_lock continues to serialize claim/consume/revoke within a single instance. Touch points: - read_lease, read_lease_index, read_lease_file — now use get and unwrap VersionedEntry.body. - write_lease, write_lease_index — now use put(Entry::bytes, Any). - Imports updated. - CountingFilesystem test scaffold gains put/get overrides that forward to its inner LocalFilesystem, since the trait defaults are now Unsupported after the PR #3659 recursion fix. * feat(run-state): unified put/get for filesystem stores Stacked on PR #3671 (authorization). Mirrors processes (#3666) and authorization (#3671) migrations. Switches all read_file/write_file calls in FilesystemRunStateStore and FilesystemApprovalRequestStore to the unified get/put ops with Entry::bytes + CasExpectation::Any. On-disk JSON layout unchanged. Test scaffold updates: ConcurrentMissingReadFilesystem and DisappearingApprovalReadFilesystem gain put/get overrides that forward to their inner LocalFilesystem and apply the same fault injection logic on the unified read path (was: only on the legacy read_file path). Required after the trait defaults moved to Unsupported in PR #3659. * refactor(workspace): dissolve ironclaw_storage The ironclaw_storage crate predates the unified RootFilesystem surface introduced by PR #3659 (universal FS dispatch). Its `BlobStore`/`RecordStore` traits, `StorageKey`/`StorageVersion`/`PutCondition` types, and `StoredBlob`/`StoredRecord` shapes parallel the new unified put/get /CasExpectation/RecordVersion machinery on `RootFilesystem` — a textbook duplicate-dispatch smell flagged by .claude/rules/architecture.md. Only `ironclaw_outbound` consumed any of the crate, and only 5 small helpers (`encode_json`, `decode_json`, `redacted_backend_error`, `StorageError::Backend`, `ABSENT_SCOPE_COMPONENT`). All other types and the entire `BlobStore`/`RecordStore` surface (660 LOC) were unused — their intended consumers already moved to `RootFilesystem` directly. Inlined the 5 helpers into `crates/ironclaw_outbound/src/db.rs`: - `encode_json`/`decode_json` → direct `serde_json::to_string`/`from_str` - `redacted_backend_error` → local log+collapse to `OutboundError::Backend` (preserves the redaction boundary required by ironclaw_outbound/CLAUDE.md) - `ABSENT_SCOPE_COMPONENT` → local const "" Removed the crate's workspace membership, the outbound dep, the forbidden-edges BoundaryRule, and the crate directory. Also updated the ironclaw_outbound BoundaryRule to permit a normal dependency on `ironclaw_filesystem` — `FilesystemOutboundStateStore` landed in the prior cascade PR and the boundary rule was stale. * feat(filesystem): add HsmBackend placeholder + scope database.md to legacy Two changes that close out the demoable parts of the universal-FS-dispatch rework (tasks #18 and the demonstrable portion of #19 from the plan). **HsmBackend placeholder** (`crates/ironclaw_filesystem/src/hsm.rs`). Demonstrates that a new backend is a single-file change: implements the one `RootFilesystem` trait, declares a restricted capability surface (`Read` + `Write` + `Stat` + `Delete` + `TxnCapability::Cas` — no records, no query, no index, no events, no multi-key transactions), and routes `put`/`get`/`delete`/`stat`/`list_dir` through an in-process placeholder. Five tests prove the seam works end-to-end: - `hsm_supports_encrypted_bytes_round_trip` — bytes put/get works. - `hsm_rejects_structured_records` — `put` with `RecordKind::Some` or non-empty `indexed` returns `Unsupported`, so a consumer cannot accidentally route records through encryption-only storage. - `hsm_rejects_query_and_index_ops` — `query`/`ensure_index` return `Unsupported` consistent with the declared capabilities. - `composite_rejects_overclaimed_hsm_descriptor` — mount-time validation (`validate_mount_capabilities`) refuses a descriptor that claims `Query`/`IndexExact` over a backend that doesn't deliver, failing with `FilesystemError::DescriptorOverclaims { missing, .. }`. - `composite_routes_to_hsm_under_secrets_mount` — the acceptance gate: mounting HsmBackend at `/secrets` and routing put/get through the composite works with no consumer-visible changes. Indexed projection is still rejected because the declared capabilities advertise no index/query support. A real HSM implementation replaces the in-memory placeholder with an HSM session handle; the trait surface, capability declarations, and mount-time validation are reusable as-is. The placeholder is *not* a security boundary — it is a seam demonstration. **database.md scoped to legacy directories**. The dual-backend rule file (`.claude/rules/database.md`) is `paths`-scoped to `src/db/**`, `src/history/**`, and `migrations/**` — exactly the legacy surface that predates the universal FS dispatch. Added a "Status & Direction" preamble pointing new persistence work at `ScopedFilesystem` and the `2026-05-14-universal-fs-dispatch.md` plan, with the existing per-crate dual-backend guidance kept (and tagged "legacy") for code still inside those directories. * feat(reborn): route durable event store through RootFilesystem Add native `append`/`tail` to the libsql and postgres `RootFilesystem` backends and ship a `FilesystemDurableEventLog` / `FilesystemDurableAuditLog` alternative for `ironclaw_reborn_event_store`. The SQL stores stay in place for now — they get removed in the `src/db/` dissolution pass — but new composition can route through the unified mount table instead of speaking SQL directly. - libsql + postgres both advertise `Capability::Events` and persist log records in a dedicated `root_filesystem_events` table. - Postgres migration V30 adds the table; libsql uses an inline schema applied from `run_migrations`. - Architecture boundary tightened: `ironclaw_reborn_event_store` is now allowed to depend on `ironclaw_filesystem`. * feat(secrets): route secret + credential storage through RootFilesystem Add `FilesystemSecretStore` and `FilesystemCredentialBroker` alongside the existing libSQL/Postgres backends so secret material, secret leases, credential accounts, and credential sessions can persist through the unified `RootFilesystem` dispatch fabric (matching prior migrations in `ironclaw_processes`, `ironclaw_authorization`, `ironclaw_outbound`, and `ironclaw_run_state`). - Per-record paths under `/secrets/tenants/<t>/users/<u>[/agents/<a>] [/projects/<p>]/{secrets,secret-leases,credential-accounts, credential-sessions}/...`. - Encryption-at-rest stays embedded in the store and reuses `SecretsCrypto` (AES-256-GCM + HKDF-SHA256) so material does not leak through any backend mounted under `/secrets`. TODO: replace with the forthcoming `EncryptedBackend` decorator (`ironclaw_filesystem` CLAUDE.md invariant #5). - Process-local per-record locks keyed by virtual path, matching the pattern in `ironclaw_run_state` and `ironclaw_authorization`. - `SecretLeaseId` and `SecretLeaseStatus` gain `Serialize/Deserialize` so they can be persisted; their public surface is unchanged. - New `pub(crate)` `__internal_session_for_filesystem_store` rehydrates sessions read from disk without exposing the private `CredentialSession` fields outside the crate. - Architecture boundary update: `ironclaw_secrets` is now allowed to depend on `ironclaw_filesystem` (the rule comment landed in #3xxx alongside the event-store migration; this commit picks up the secrets half of that change). - Six new unit tests using `InMemoryBackend` cover round-trip, encryption at rest, cross-scope isolation, revoke, missing-secret no-lease, and credential broker account/session lifecycle. All existing tests pass unmodified (60 tests total). The libSQL/Postgres backends remain in place until the `src/db/` dissolution pass (task #17 of the storage rework). * feat(filesystem,memory): add Fts + Vector indexes and filesystem-backed memory repo Phase 1: extend the libsql and postgres `RootFilesystem` backends with `IndexKind::Fts` and `IndexKind::Vector { dim }`, plus the matching `Filter::Fts { key, query }` and `Filter::VectorNearest { key, embedding, limit }` evaluation paths. - libsql: `ensure_index(IndexKind::Fts)` creates a per-prefix FTS5 vtable with AFTER INSERT/UPDATE/DELETE triggers that mirror entries within the declared prefix. Backfill on declaration handles pre-existing rows. `Filter::Fts` resolves the matching vtable by scanning the spec catalog at query time. Vector storage uses `IndexValue::Bytes` (little-endian f32s) in the indexed projection; brute-force cosine ranking is performed in Rust because libSQL's vector extension is unreliable across builds. - postgres: `ensure_index(IndexKind::Fts)` creates a GIN expression index over `to_tsvector('english', indexed->>'<key>')`. `Filter::Fts` translates to a `@@ plainto_tsquery(...)` predicate so the GIN index is usable. Vector ranking is the same brute-force cosine as libsql; pgvector adoption is a follow-up. - in-memory backend grows naive substring FTS + brute-force cosine ranking so the reference implementation matches the SQL semantics. - Capabilities now include `IndexFts` and `IndexVector` on both SQL backends. - Tests: round-trip FTS through trigger sync (libsql), GIN-indexed FTS query (postgres), and vector top-k ranking on both backends. Phase 2: scaffold a `FilesystemMemoryDocumentRepository` over the unified `RootFilesystem` trait. Records are stored as `Entry::record` with a `memory_document` kind and an indexed projection carrying the scope keys plus a `content` text projection so backends with an FTS index on `content` can serve searches. Metadata is stored at a sibling `.meta` path. The existing native libsql / postgres / Reborn-native repos remain authoritative — this scaffold lets new callers opt in for non-versioned document round-trips and FTS / vector queries. Known TODOs documented inline in `filesystem.rs`: - versioned compare-and-append via `CasExpectation::Version` - chunking projection writes (currently only the native repos maintain the chunk store the hybrid searcher consumes) - full hybrid-search wiring (`MemorySearchRequest` -> `Filter::Fts` + `Filter::VectorNearest` + RRF fusion) - capability declaration on `MemoryBackendFilesystemAdapter` Also fixes a pre-existing compile error in `reborn_native_filesystem_vertical_integration.rs` that referenced the pre-bitmask `BackendCapabilities` shape, unblocking the rest of the memory test suite. Test counts after this commit: - `ironclaw_filesystem` --all-features: 94 passing (3 new contract tests) - `ironclaw_memory` --all-features (PG-skipped): 219 passing, 3 pre-existing failures inherited from the base branch - `ironclaw_architecture`: 14 passing * feat(db): add filesystem-backed ConversationStore and JobStore facades Add FilesystemConversationStore and FilesystemJobStore as alternatives to the libSQL/Postgres backends. Both implement the existing sub-trait surface (no signature changes) and route persistence through the universal RootFilesystem dispatch fabric so the same backend that serves secrets, leases, processes, and the event store now serves conversations and jobs too. Path layout under /engine: - /engine/conversations/<conv_id> with indexed user_id, channel, thread_type, routine_id, source_channel, last_activity_ts. - /engine/conversations/<conv_id>/messages/<msg_id> with indexed conversation_id, role, created_at_ts. - /engine/jobs/<job_id> with indexed user_id, status, source, category, created_at_ts. - /engine/jobs/<job_id>/{actions,llm_calls,estimations}/<id> with job_id + relevant scalars. Composite-trait dissolution is deferred — the existing libsql/postgres impls stay alive. 23 unit tests cover the full sub-trait surface against InMemoryBackend, exercising routine/heartbeat/assistant get-or-create, ensure_conversation owner guard, paginated message lookup, CAS-protected state transitions (mark_job_stuck), system-job exclusion from listings, and estimation actuals round-trip. * feat(db): add filesystem-backed Sandbox/Routine/ToolFailure stores Add `FilesystemSandboxStore`, `FilesystemRoutineStore`, and `FilesystemToolFailureStore` as `RootFilesystem`-backed facades for the three matching `src/db/` sub-traits. Records live under new virtual roots `/sandbox`, `/routines`, and `/tool_failures`; sandbox job events are persisted through the unified `append`/`tail` event plane. Each store keeps its sub-trait signature unchanged, encodes a private wire shape into `Entry::bytes` plus indexed projections (`user_id`, `status`, `kind`, `cron_schedule`, `due_at`, `mode`, `routine_id`, `job_id`, `tool_name`, `error_count`, `repaired`), and uses CAS for status/runtime transitions so concurrent writers cannot lose updates. Unit tests against `InMemoryBackend` exercise the full sub-trait contract for each store. The legacy libSQL/Postgres impls are unchanged. * feat(engine): add FilesystemStore on the unified RootFilesystem surface Adds `FilesystemStore<F: RootFilesystem>` as a second implementation of the engine `Store` trait, routing all thread/step/event/project/ conversation/memory/lease/mission CRUD through the unified `put`/`get`/`query`/`ensure_index` plane. Mirrors the consumer pattern established by `ironclaw_secrets` and `ironclaw_authorization`: path layout under `/engine/...`, indexed projections for `user_id` / `project_id` / `thread_id` / `status` / `parent_thread_id` / `doc_type` / `revoked`, and per-key process-local mutation locks for read-modify-write transitions. `HybridStore` in `src/bridge/store_adapter.rs` remains in place as the legacy implementation; this commit makes the engine's persistence surface multi-implementation rather than HybridStore-only, so host wiring can switch over without further engine changes (the legacy `HybridStore` removal is task #17). Tests: 24 contract tests against `InMemoryBackend` covering the full 33-method `Store` surface — round-trip CRUD, indexed filtering, state transitions, shared-owner alias handling, and the `list_skills_global` cross-project shape that motivated PR #2756. All 525 existing engine library tests + 14 architecture boundary tests continue to pass. * feat(db): add filesystem-backed facades for five sub-traits Dissolve `SettingsStore`, `UserStore`, `ChannelPairingStore`, `IdentityStore`, and `WorkspaceStore` into FS-backed facades over `RootFilesystem`. Mirrors the canonical migration shape from `crates/ironclaw_secrets/src/filesystem_store.rs` and `crates/ironclaw_authorization/src/lib.rs`. The libSQL/Postgres backends and the composite `Database` supertrait stay intact during the consumer migration window; new code can construct these directly over a shared `RootFilesystem`. Path layout: - `/system/settings/<user_id>/<key>` - `/users/<id>` + `/users/.tokens/<token_id>` + `/users/.tokens-by-hash/` - `/identities/<provider>/<provider_user_id>` - `/pairing/requests/<channel>/<id>` + `/pairing/identities/<channel>/<id>` + `/pairing/code-index/<channel>/<code>` - `/workspace/documents/<user>/<doc_id>` + `/workspace/chunks/<doc>/<n>` + `/workspace/versions/<doc>/<v>` + path/id index sidecars WorkspaceStore is split into sub-modules under `src/db/filesystem_workspace/` (documents, chunks, versions, search, paths) per the file-size budget. Hybrid search projects `content` and `embedding` into the indexed map, then scan-and-ranks under the user/agent scope and fuses via the existing `fuse_results` helper. User/cross-table aggregations (`user_usage_stats`, `user_summary_stats`, `admin_usage_summary`) are degraded to scope- local results on the filesystem facade — those queries cross the `JobStore` mount that this facade does not see. `/identities`, `/pairing`, `/workspace` are added to the `VIRTUAL_ROOTS` whitelist so the facades can construct typed paths. Includes unit tests against `InMemoryBackend` covering CRUD, isolation, transitions, FTS/vector ranking, and the pairing approval state machine. * fix: replace .expect on validated literals with unwrap_or_else(unreachable!()) CI's `scripts/check_no_panics.py` flags `.unwrap()`/`.expect()` in production code. Agent-generated stores used `.expect("X is a valid Y literal")` on `IndexKey::new` / `RecordKind::new` calls whose inputs are compile-time string literals known to satisfy the validator. Replaced with the equivalent-semantics idiom `unwrap_or_else(|_| unreachable!("..."))` — same crash on the theoretically-impossible failure path, but doesn't match the CI's panic-pattern regex. Affects: - crates/ironclaw_memory/src/repo/filesystem.rs (6 sites) - src/db/filesystem_conversations.rs (4 sites) - src/db/filesystem_jobs.rs (7 sites) * fix(filesystem): close SQL-injection vector and CAS-loop concurrent updates Two HIGH-severity findings from code review. Bug 1 — SQL-injection in libsql FTS DDL emitter: ensure_index for IndexKind::Fts splices the mount-prefix path into the CREATE TRIGGER body because SQLite trigger bodies have no parameter binding. VirtualPath::new rejects NUL/control/backslash/`..` but does not reject `'`, `"`, `;`. Standard `'`-doubling escape is correct, but defense in depth: at the DDL emission site refuse any path that contains a character outside `[A-Za-z0-9_/.-]`. Postgres path is parameterized, so only libsql was affected. Regression test added. Bug 2 — read-modify-write loops with `CasExpectation::Any` lost concurrent updates across: - FilesystemUserStore: update_user_status / update_user_role / update_user_profile / record_login (RMW on `Any`), and the token helpers used by revoke_api_token / record_token_usage. - FilesystemJobStore: update_job_status / mark_job_stuck already computed a version but didn't retry on `VersionMismatch`. - Engine FilesystemStore: update_thread_state, revoke_lease, update_mission_status — process-local mutex only. Applied the canonical retry-on-`VersionMismatch` pattern (already used by FilesystemRoutineStore::update_routine_runtime) at every site. filesystem_settings.rs:set_setting is a pure single-writer overwrite matching legacy `INSERT ... ON CONFLICT DO UPDATE`, so it stays on `Any` with an explanatory comment. Also fixes a pre-existing `unwrap_or_else(|_|...)` typo (1-arg closure on an Option) that blocked `cargo test --lib`. * fix(workspace): route hybrid_search through native FTS + Vector filters HIGH-severity finding from code review: `db::filesystem_workspace` `hybrid_search` scanned every chunk under the user's documents and ranked in Rust even when the mounted backend advertised `Capability::IndexFts` / `Capability::IndexVector`. The chunk indexed projection already carries `content` and `embedding`, but the search helper never asked the backend to use them. - search::hybrid_search now calls `filesystem.query(/workspace/chunks, Filter::Fts { content, query })` and `filesystem.query(.., Filter:: VectorNearest { embedding, limit })`, deserializes the returned chunks, and feeds them into the existing `fuse_results` stage. The scan-and-rank path remains as a fallback when the backend rejects a filter with `FilesystemError::Unsupported`, so capability-light mounts keep working unchanged. - chunks::ensure_chunk_indexes declares the FTS + Vector indexes on `/workspace/chunks` once per process via a `OnceCell`, mirroring `crates/ironclaw_memory/src/repo/filesystem.rs`. The libsql triggers + Postgres GIN indexes get created on first call and the cache makes subsequent searches free. - Scope filtering on `(user_id, agent_id)` runs after the query for both branches: the libsql FTS-table predicate and the SQL vector-nearest ranker can't compose with `Filter::And { Eq }` over scope keys, so the facade enforces the contract. - mod.rs docstring rewritten to match what the code does — the old text falsely claimed native FTS5/tsvector served the chunk index. - Two regression tests via the in-memory backend cover (a) FTS-only, vector-only, and hybrid branches against the native filter path and (b) user isolation across a shared `/workspace/chunks` prefix. Both tests fail against the prior scan-and-rank-only implementation. Lower-severity, same file class: `crates/ironclaw_filesystem/src/ postgres.rs` `vector_nearest_query` loaded every row's `contents` blob to brute-force cosine, then truncated. Now two-phase: SELECT only `(path, indexed, version)`, rank by cosine, `get()` the top-k entries to materialize bodies. Same fix landed for libsql in PR e2530adff. * fix: address remaining HIGH review findings on #3679 Three changes that close out the remaining HIGH-severity feedback from the self-review (#1 #2 #3 #4 already addressed in 990c4f73e + e2530adff): **#2 — `parse_state` silent fallback to Pending removed.** `src/db/filesystem_jobs.rs::parse_state` previously mapped unknown status strings to `JobState::Pending`, masking schema drift across a rollout (a new state value appearing in stored rows would silently lose its true value). Now returns `Result<JobState, DatabaseError>` and the single caller propagates with `?`. Matches the wire-stable enums rule in `types.md`. **#6 — `is_engine_unsupported` no longer substring-matches.** `crates/ironclaw_engine/src/store/filesystem.rs`: the typed `FilesystemError::Unsupported` discriminator gets lost when wrapped in `EngineError::Store { reason: String }`, so the old check `reason.contains("Unsupported")` would false-positive on any unrelated store error that mentioned the word. Now `fs_to_engine_error` tags the discriminator with a stable `[fs:unsupported]` sentinel and the check matches that sentinel — discriminator-preserving without changing the public `EngineError` shape. **#7 — `FilesystemChannelPairingStore` no longer drops corrupted records.** `src/db/filesystem_pairing.rs::find_pending_requests`: the old code silently filtered records whose JSON failed to deserialize, hiding data corruption. Now propagates `DatabaseError::Serialization` with the stored path so the operator sees the failure. Also: `// silent-ok:` annotations added to the three engine `Store` sites where read-modify-write on unknown ids is intentionally a no-op (matches HybridStore parity per its CLAUDE.md). Each annotation names the legacy contract being preserved. Verification: `cargo check --workspace --all-features` clean; `cargo test -p ironclaw_engine --all-features` 549/549; `cargo test --lib --all-features db::filesystem` 88/88; `cargo fmt --check` clean. * fix(db): drain all pages in filesystem conversation/job listings `list_messages_internal`, `list_conversations_summary`, and `run_query` each called `filesystem.query(.., Page::new(0, Page::MAX_LIMIT))` exactly once and trusted the result was complete. Because `Page::MAX_LIMIT == 1024`, conversations with >1024 messages or scopes with >1024 jobs/actions/estimations silently lost every row past the cap, and the `has_more` flag in `list_conversation_messages_paginated` became meaningless once the dropped tail crossed the page boundary. Codex PR #3679 P2 review flagged the pattern. Extract a shared `query_all_pages` helper in `filesystem_conversations` that loops `query(..., Page::new(offset, MAX_LIMIT))` until a short page comes back, then reuse it from `filesystem_jobs::run_query` and from the inline scan in `update_estimation_actuals`. The helper preserves the existing `NotFound -> Vec::new()` short-circuit and the `fs_err_to_database` error mapping so call sites are otherwise unchanged. Regression tests: - `list_messages_drains_pages_beyond_max_limit` writes `MAX_LIMIT + 5` messages and asserts the full count round-trips through `list_conversation_messages` and that `list_conversation_messages_paginated` reports `has_more` honestly for both partial and exhaustive windows. - `get_job_actions_drains_pages_beyond_max_limit` writes `MAX_LIMIT + 3` actions on one job and asserts the full count comes back in sequence order. - `list_agent_jobs_drains_pages_beyond_max_limit` writes `MAX_LIMIT + 7` jobs and asserts both `list_agent_jobs` and `agent_job_summary` count every row. * fix(secrets): close CAS-loop races in filesystem store consume paths Two HIGH-severity findings on PR #3679. Both sites read a versioned entry, validated a one-shot/use-limit condition, then wrote back with `CasExpectation::Any`. The process-local mutex only serializes writers inside one process; multi-process callers sharing the same backend root could both pass the check and overwrite each other. - `FilesystemSecretStore::consume` — two consumers could both observe an Active one-shot lease, both decrypt, and both overwrite the consumed marker. - `FilesystemCredentialBroker::consume_session_use` — two consumers could both pass the max-uses check at `uses=N-1` and overwrite each other's increment, losing a use. Both now use the canonical retry-on-`FilesystemError::VersionMismatch` pattern from `ironclaw_engine::store::filesystem::update_thread_state` (post-`e2530adff`): re-read, re-evaluate the consume/use-limit condition, write with `CasExpectation::Version(versioned.version)`. A shared `CAS_RETRY_ATTEMPTS = 3` constant bounds the loop; exhausting it surfaces a transient backend error rather than papering over pathological hot-spots. Also annotated `leases_for_scope` with a `TODO(perf)` covering the N+1 list+get fan-out — bounded today by the owner-prefix path layout and short lease TTLs; replacing it with `Filter::Eq` over `query` requires the secrets store to declare its first index, which is a follow-up. Regression coverage: two new tests wrap `InMemoryBackend` with a `VersionRacingBackend` that bumps the watched path's version out-of-band on the first versioned `put`, forcing a `VersionMismatch` and exercising the retry loop. They also assert that the retried CAS write actually persisted (the next consume hits LeaseConsumed; the next three increments exhaust the max-uses budget). * fix: address remaining P2 review findings on #3679 Four P2 correctness fixes from the codex/gemini review. **Settings keys use percent-encoding** (`src/db/filesystem_settings.rs`): `encode_segment` previously mapped `/`, space, control chars, and others all to `_`. Keys like `a/b` and `a_b` collided onto the same path and silently overwrote each other. Now percent-encodes every byte outside the unreserved set so distinct inputs map to distinct outputs. **SQL index names get a blake3 suffix on overflow** (`crates/ironclaw_filesystem/src/db.rs`): `sql_index_name` truncated identifiers exceeding 62 chars without disambiguating, so two distinct long `(prefix, name)` specs could collapse onto the same DDL object — `CREATE ... IF NOT EXISTS` would silently reuse the wrong index/trigger. Now appends an 8-char blake3 hash suffix before truncating. Added `blake3 = "1"` to the crate's deps (small + already used by other workspace crates). **InMemoryBackend rejects writes over implicit directories** (`crates/ironclaw_filesystem/src/in_memory.rs`): the SQL backends refuse `put(/a)` when `/a/b` exists (treating `/a` as a directory). The in-memory reference impl silently accepted those writes, letting tests pass against production-impossible state. Mirror the SQL contract. **Event-store head-probe is bounded** (`crates/ironclaw_reborn_event_store/src/filesystem_store.rs`): The replay-gap detection previously called `tail(path, 0)` to read the whole log just to look at its last seq — O(N) on every cold-path call. Now probes `tail(path, after - 1)`: a non-empty result means head == after (consumer is caught up); empty means head < after (foreign-future cursor). Returns at most one record instead of the entire log. Verification: cargo check --workspace --all-features clean; cargo test -p ironclaw_filesystem -p ironclaw_secrets -p ironclaw_reborn_event_store --all-features all pass. * fix(filesystem): close cross-backend Range/vector semantic drift and txn scope hole Audit findings on ironclaw_filesystem turned up four bugs and three semantic-drift cases between the in-memory reference and the SQL backends. Fix them in one pass so the cross-backend contract is honoured and the gaps have regression coverage. Bugs: - libSQL `Filter::Range` on `IndexValue::Bool` never matched any row because SQLite's `json_type` returns "true"/"false" for booleans rather than "integer". Replaced the static type string with a `json_type_guard` expression that admits both bool variants. - `ScopedStorageTxn` did not enforce that per-op `VirtualPath`s lay under `mount_prefix`. The trait doc promised `PathOutsideMount` for cross-prefix accesses; the wrapper now enforces it so any future backend that ships `begin()` inherits the guarantee. - Mixed-variant `Filter::Range` bounds (e.g. I64 lo + Text hi) silently lex-compared on text on both SQL backends. Added the in-memory backend's `discriminant(lo) == discriminant(hi)` guard to both, rejecting with `Unsupported`. - SQL `vector_nearest_query` lacked the in-memory backend's path tie-breaker on equal cosine scores, so top-k truncation was non-deterministic. Added `.then_with(|| a.0.cmp(b.0))` to both. Semantic drift: - `FilesystemOperation` lacked an event-plane `Append` variant — default impl reported `Tail`, backends reported `AppendFile`. Added the variant, routed every emit site through it, and updated the downstream `host_runtime::operation_allowed` matcher. - `decode_embedding_blob` and `cosine_similarity` were byte-identical copies in three files. Extracted to `crate::vector`. - libSQL `run_migrations` ran multiple ALTERs outside any transaction. Wrapped the sequence in BEGIN IMMEDIATE / COMMIT with rollback on error so a crash can't leave a half-migrated schema observable. Tests added: - 16 `ScopedFilesystem` permission tests covering query / ensure_index / begin / append / tail across each `MountPermissions` axis, plus 4 `ScopedStorageTxn` tests driving a stub backend to lock in the per-op ACL and the new path-containment check. - Cross-backend regression tests in `tests/db_root_filesystem_contract.rs` for the libSQL Bool/Range fix, the discriminant guard on both SQL backends, and the deterministic vector tie-breaker. - Refactored `vector_nearest_query`'s phase-2 step into `materialize_ranked` (`pub(crate)`) so a unit test can exercise the "row disappeared between phases" branch deterministically. 128 tests pass, all three feature combos compile (`default`, `libsql`, `postgres`), workspace builds. * revert(db): drop filesystem-backed src/db/ store facades Removes all `src/db/filesystem_*.rs` facades and the `src/db/filesystem_workspace/` directory added during the PR #3679 universal-FS dispatch migration: - filesystem_conversations, filesystem_jobs - filesystem_routines, filesystem_sandbox, filesystem_tool_failures - filesystem_identities, filesystem_pairing, filesystem_settings, filesystem_users - filesystem_workspace/{mod,chunks,documents,paths,search,versions}.rs Also removes the supporting infra that only existed for these files: - `ironclaw_filesystem` workspace dep from the root `ironclaw` crate - `/sandbox`, `/routines`, `/tool_failures`, `/identities`, `/pairing`, `/workspace` entries from `ironclaw_host_api::path::VIRTUAL_ROOTS` The legacy libSQL/Postgres sub-trait impls (`src/db/postgres.rs`, `src/db/libsql/*.rs`) remain the sole backing for the `Database` supertrait. The unified `ironclaw_filesystem` mount fabric itself (the `crates/ironclaw_filesystem/` crate) is untouched and still used by consumer crates outside `src/db/`. Verification: - cargo fmt --check clean - cargo check --workspace clean (default features) - cargo check --no-default-features --features libsql clean - cargo check --all-features clean - cargo clippy --all --benches --tests --examples --all-features clean [skip-regression-check] pure removal of unmerged migration facades. * test(reborn-event-store): cover caught-up-to-head + concurrent appends Addresses audit finding F1. (a) `filesystem_event_log_caught_up_to_head_returns_empty_not_replay_gap` appends N events, replays from the last entry's cursor, and asserts `entries.is_empty()` + `next_cursor == last.cursor` with no `ReplayGap`. Pins the "consumer is caught up to head" branch of the bounded probe in `read_after_cursor`. (b) `filesystem_event_log_concurrent_appends_assign_distinct_cursors` spawns 8 `tokio::spawn` tasks each appending one event to the same stream, then asserts the collected cursors are pairwise-distinct and strictly increasing. Guards the per-stream monotonic-cursor invariant under contention. * fix(reborn-event-store): preserve filesystem error detail in durable mappers Addresses audit finding F2. `map_filesystem_append_error` / `map_filesystem_tail_error` previously collapsed every non-categorised `FilesystemError` variant (`VersionMismatch`, `NotFound`, `Backend`, …) to a fixed generic string, dropping the source variant and reason. Operators lost the detail they needed to debug appends that hit a CAS conflict or a backend I/O failure. Thread the underlying `FilesystemError` through its `Display` impl on the fallback arm. `FilesystemError` is already redaction-safe by contract — it renders scoped/virtual paths, never raw host paths — so the durable error surface gains debug detail without violating the crate-level redaction policy. The three already-categorised variants (`PermissionDenied`, `MountNotFound`, `Unsupported`) keep their fixed messages so callers can pattern-match on the substring. * fix(reborn-event-store): document deliberate absence of Filesystem config variant Addresses audit finding F3. `FilesystemDurableEventLog` / `FilesystemDurableAuditLog` are exported from this crate, but `RebornEventStoreConfig` has no corresponding `Filesystem` variant — so production composition still routes through the SQL stores. The PR description documents this as intentional: the filesystem-backed log is the migration target for the kernel-storage rework, and the config variant will be added during the `src/db/` dissolution pass (task #17). Without an inline comment, a future reviewer reading the config enum has no signal that the missing variant is deliberate. Add a doc paragraph on `RebornEventStoreConfig` pointing at the rationale on `filesystem_store.rs` and at task #17. * fix(reborn-event-store): drop shadowed kind named-arg in stream_path format! Addresses audit finding F4. `stream_path` previously used the named-argument `format!` form with `kind = kind_segment`, where the named key `kind` shadowed the function parameter of the same name. Switch to the implicit positional-capture form (`format!("/events/{kind_segment}/...")`) and rename the inline bindings to `tenant_segment` / `user_segment` for consistency. Pure refactor — no behaviour change, just removes the readability footgun. * fix(outbound): add typed CasConflict variant for filesystem store retries Audit finding F5: `map_fs_error` previously collapsed both `FilesystemError::VersionMismatch` (a transient compare-and-swap race condition that callers should retry) and `FilesystemError::Unsupported` (a permanent capability gap) into `OutboundError::Backend`. The bounded CAS retry loop (added separately for F1) cannot match on `Backend` — that would also retry on permanent backend failures and on `Unsupported` on backends that don't support CAS. Introduce `OutboundError::CasConflict` and map `VersionMismatch` to it in `map_fs_error`. The variant stays internal to the crate: the retry loop matches on it discriminator-wise; once the retry budget is exhausted (or for callers that haven't migrated) it converts to `Backend` before crossing the trait boundary, preserving the no-leak contract. Update `is_transient_validator_error` to classify `CasConflict` as transient for defence in depth, even though it should never reach the service boundary in practice. * fix(outbound): CAS-version read-then-write paths with bounded retry Audit finding F1 (HIGH): the four read-then-write methods on `FilesystemOutboundStateStore` (`upsert_subscription`, `advance_subscription_cursor`, `record_delivery_attempt`, `update_delivery_status`) read the existing entry, applied an in-memory transform, then wrote with `CasExpectation::Any`. Concurrent writers raced the transform: in particular, the "subscription cursor must not move backwards" invariant — enforced in `validate_advance_request` / `validate_subscription_cursor_progression` — was unenforced cross-process, because two racing advancers could both read the same old cursor, validate against it, and then both put their newer cursors, the loser silently winning the last-write race. Capture `VersionedEntry.version` from each `get`, pass `CasExpectation::Version(v)` to the matching `put`, and retry on the typed `OutboundError::CasConflict` introduced by F5. The retry budget is bounded (`MAX_CAS_RETRIES = 5`) and the loop re-reads + re-validates on every iteration, so a regressing cursor or scope mismatch surfaces immediately rather than letting the retry loop overwrite the winner's state. `put_thread_notification_policy` is a blind overwrite and keeps `CasExpectation::Any`. `record_delivery_attempt` uses `CasExpectation::Absent` for the first-write branch, so two racing at-least-once writers can't both insert; the loser falls back into the duplicate-identity-check branch on the next read. * fix(outbound): use control-character sentinel in thread scope key Audit finding F6: `thread_scope_key` used the literal string `"_"` as the sentinel for `agent_id = None` / `project_id = None`. The `validate_scope_id` validator in `ironclaw_host_api` accepts underscore as a legal character in an `AgentId` / `ProjectId`, so a scope with `agent_id = Some(AgentId::new("_"))` hashed to the same key as a scope with `agent_id = None`. Two distinct scopes silently collided on the same policy/subscription/delivery virtual path. Switch the sentinel to `"\x1F"` (ASCII unit-separator). It's a control character; `validate_scope_id` rejects every C0 control char via `has_forbidden_control`, so no legal scope id can ever contain it. Add a unit test that pins the sentinel-rejection invariant and a regression test that proves `agent_id = Some("_")` no longer hashes to the same key as `agent_id = None`. * fix(outbound): query indexed scope projection with paginated drain Audit finding F2 (HIGH): `list_delivery_attempts` was a `list_dir` + N+1 `get_json` per row with no indexed projection, scanning every delivery on the mount even when only one scope's deliveries were requested. Cost scaled with total delivery count, not with the queried scope's row count. Declare an exact-equality index on a new `scope` indexed key. The projected value is the same `thread_scope_key` hash used for policy paths — collision-resistant against the legal id grammar and updated by F6 to never collide with the `None` sentinel. `record_delivery_attempt` and `update_delivery_status` write through a new `put_delivery_attempt_indexed` helper that includes the projection; `update_delivery_status` preserves it on status mutations. The list path drives `query(Filter::Eq { key: "scope", value: ... })` and re-checks `scope_matches` defensively (hash collisions are unreachable but cheap to guard against). Audit finding F3 (Medium): the previous `list_dir` was unpaginated; SQL backends issue `LIMIT Page::MAX_LIMIT (1024)` on their list_dir translation and would silently truncate past 1024 deliveries. The new path drains pages via `offset += received` until a short page arrives, mirroring `ironclaw_engine::store::filesystem::query_all`. `ensure_delivery_scope_index` runs idempotently before every write and read. It tolerates `FilesystemError::Unsupported` on byte-only backends to match the engine store's `ensure_exact_index` pattern; the in-memory backend serves `Filter::Eq` from `Entry::indexed` directly even without a materialized index declaration. * test(outbound): cover CAS retry, pagination drain, backwards-race Audit finding F4: the existing `outbound_state_store_contract` suite exercised the storage contract surface but had no coverage for any of the failure modes the F1/F3 fixes address: - No CAS-retry test. F1's bounded retry loop could regress to permanent failure on any transient `VersionMismatch` and the suite wouldn't notice — the in-memory backend never produced one. - No `> Page::MAX_LIMIT` drain test. F3's pagination loop could lose the tail of a long delivery list and the suite wouldn't notice because the existing tests record at most one delivery per scope. - No concurrent backwards-race test on `advance_subscription_cursor`. The existing backwards-advancement test only exercised the single- threaded path; nothing proved the post-F1 retry loop re-validates progression on every iteration. Add three regression tests: 1. `VersionRacingBackend` wraps `InMemoryBackend` and injects a single `FilesystemError::VersionMismatch` on the next `put` matching a configured prefix. The first new test (`advance_subscription_cursor_retries_through_cas_conflict`) arms one conflict, advances the cursor, asserts the retry loop converges, and asserts exactly one conflict was injected and consumed. 2. `concurrent_backwards_race_rejected_after_winner_advances` runs two sequential advances — the winner to cursor=100 and the loser to cursor=50 — and asserts the loser is rejected with `InvalidRequest` while the winner's state is preserved. Together with the retry test this proves the re-validate-on-retry semantics F1 calls out. 3. `list_delivery_attempts_drains_more_than_page_max_limit` writes `Page::MAX_LIMIT + 1` delivery attempts under one scope and asserts `list_delivery_attempts` returns every one. Before F3 this would silently truncate at 1024 rows. Cargo.toml: enable `tokio/sync` for `Mutex` in the test mock; drop the feature-conditional `use std::sync::Arc` because the new tests need it unconditionally. * fix(run-state): bound filesystem lock map under tenant churn The process-wide FILESYSTEM_RECORD_LOCKS map kept one Arc<tokio::sync::Mutex<()>> per touched path. In long-running hosts with high tenant/invocation churn the map grew without bound, since entries were never removed once the originating put/get cycle completed. Switch the value type to Weak<Mutex> so dropped Arcs no longer pin map slots. Each acquisition opportunistically prunes dead entries before upgrading-or-installing, keeping the map size proportional to in-flight paths rather than to lifetime path count. Concurrent callers on the same path still observe the same Arc (the outer std::sync::Mutex serializes the upgrade-or-insert window), so existing intra-process and cross-instance serialization guarantees are preserved — both verified by the new unit tests and by the existing filesystem_*_duplicate_*_serialized_across_store_instances contract tests. Addresses audit findings F1 (Medium) and F4 (Low). * fix(run-state): use versioned CAS for filesystem run/approval writes All filesystem put() calls used CasExpectation::Any, so two host processes mounting the same /engine could lose updates: each one's read-modify-write saw the other's value and then unconditionally overwrote it. The per-path async mutex only serializes intra-process callers. Switch creates to CasExpectation::Absent and updates to CasExpectation::Version(v) with a bounded retry loop on VersionMismatch. The new put_with_cas helper centralizes the contract: on capable backends (InMemoryBackend, the upcoming SQL ports) cross-process races now fail closed and the caller retries; on byte-only backends that return Unsupported (LocalFilesystem) we degrade to Any but emulate Absent with a get() precheck so the AlreadyExists path is preserved. The in-process lock map (F1) keeps the check-then-write race closed for the byte-only fallback. Approve/deny/discard pull the record-lock guard up to the trait method, since update_status no longer acquires it. Addresses audit finding F2 (Medium). Closes the gap acknowledged in crates/ironclaw_run_state/CLAUDE.md. * fix(approvals): type approval-resolution decision with ApprovalDecisionKind enum Addresses audit finding F1. Replaces the stringly-typed `impl Into<String>` decision parameter on `AuditEnvelope::approval_resolved` with a wire-stable `ApprovalDecisionKind` enum (`Approved`/`Denied`, `#[serde(rename_all = "snake_case")]`), so approval callers cannot drift on capitalization or spelling. Per `.claude/rules/types.md` "wire-stable enums". The wider `DecisionSummary::kind` field stays a `String` because other audit producers (authorization denials, obligation handlers) emit values outside the approval enum; cross-decoding remains a follow-up. Cross-crate blast radius: `ironclaw_host_api` (new enum + factory signature), `ironclaw_approvals` (both call sites), `ironclaw_events::tests::durable_log_contract` (three test fixtures). * fix(approvals): persist approval state before issuing lease Addresses audit finding F2. Inverts the lease/approve ordering inside `approve_capability_action`: the approval store write now runs *before* the lease store write. The previous order (issue lease, then approve, best-effort revoke on failure) left a window where a transient approval-store error could leave a live lease pointing at a request whose status remained `Pending`. The approval record is now treated as the authority of record. Once the request flips to `Approved`, lease issuance is a recoverable operation against an already-decided request — if the lease store fails, the caller surfaces the lease error and the request stays `Approved`. The previous best-effort `let _ = self.leases.revoke(...)` swallow is gone with the same edit. Updates the three concurrency/error-injection tests to assert the new semantics, plus the crate CLAUDE.md guardrail. No external test fixtures break — the public resolver API is unchanged. * fix(approvals): route both resolve paths through emit_approval_resolved helper Addresses audit finding F3. Extracts an `emit_approval_resolved` helper on `ApprovalResolver` so the audit-envelope construction in `approve_capability_action` and `deny` is built in exactly one place. Both call sites used to inline `AuditEnvelope::approval_resolved` against their own `record.scope`/`denied.scope`; while consistent today, divergence between the two would be a silent regression. Pure refactor — no test changes needed beyond the existing audit-event contract tests which already pin the wire shape. * fix(approvals): cover concurrent approve_dispatch first-write-wins Addresses audit finding F4. Adds a caller-level concurrency regression test that spawns two `approve_dispatch` calls against the same pending request on a multi-thread tokio runtime and asserts the expected first-write-wins invariants: - exactly one approve returns `Ok` - the other returns `ApprovalResolutionError::NotPending { status: Approved }` - the lease store ends up with exactly one Active lease (not two, not zero — under the F2 persist-approval-first ordering the loser fails *before* lease issuance, so no orphan to revoke) - the approval record's terminal status is `Approved` Enables `rt-multi-thread` on the tokio dev-dependency so the test can exercise real cross-thread contention on the approval store mutex. * fix(engine): restore HybridStore parity for mission updates F1: `update_mission_status` now bumps `mission.updated_at` before writing back, matching HybridStore (`src/bridge/store_adapter.rs:1950`). Recency-sorted views (mission list UIs, learning-mission dispatcher) were silently freezing the timestamp at original-save time. F2: `list_missions` and `list_all_missions` now sort by `(name, id)` after collection, matching HybridStore (`store_adapter.rs:1913, 1937`). The underlying `query`/HashMap iteration is non-deterministic; the LLM-facing `mission_list` tool was seeing arbitrary order across runs. Tests: - `update_mission_status_bumps_updated_at` — regression for F1 - `list_missions_is_deterministic_across_invocations`, `list_all_missions_is_deterministic_across_invocations` — regression for F2 * fix(memory): drain pages in FilesystemMemoryDocumentRepository::list_documents Audit findings F1 (HIGH) + F9 (Low). F1: `list_documents` issued a single `query(.., Page::new(0, Page::MAX_LIMIT))` and trusted the page was complete. Because `Page::MAX_LIMIT == 1024`, scopes holding >1024 documents silently lost every entry past the cap. The result fed `write_document`'s ancestor/descendant conflict check at the call site immediately above, so a new path could shadow (or be shadowed by) an existing document across the truncation boundary without a conflict ever firing — exactly the regression `query_all_pages` was extracted in `src/db/filesystem_jobs.rs` to prevent. F9: The old implementation issued a `Filter::All` query, threw the results away (`let _ = (versioned, &prefix_str);`), then called `list_dir` to discover paths. The query-result loop was dead code under any backend that supports `query`. The stale comment claimed the trait didn't surface paths in `query` results, but `VersionedEntry.path` (`crates/ironclaw_filesystem/src/record.rs:347`, added in PR #3659) has carried the absolute virtual path for every queried row since. Replace both with a single drain loop that paginates `query` until a short page comes back, filters by `entry.kind == "memory_document"`, and recovers the `MemoryDocumentPath` directly from `VersionedEntry.path`. The `list_dir` fallback is gone, and the agent_id axis is preserved through `MemoryDocumentPath::new_with_agent` so scopes with an agent identity round-trip correctly (the previous code's `new()` dropped the agent). Regression: `list_documents_drains_pages_beyond_max_limit` writes `MAX_LIMIT + 5` documents and asserts every one comes back. This also exercises the conflict-check path because each `write_document` calls `list_documents` internally. * fix(secrets): close consume_if_matches timing oracle with constant-time compare F1 (HIGH, timing oracle) in the 2026-05 audit: `consume_if_matches` in `legacy_store.rs` (trait default + in-memory backend) and `db.rs` (libSQL + Postgres backends) compared the decrypted plaintext against the caller-supplied expected value with `!=`. Rust's `!=` over `&str`/`&[u8]` short-circuits on the first differing byte, so an adversary who can observe response latency over the network can recover the secret byte by byte. AES-GCM authenticated decrypt closes the ciphertext oracle but does nothing for the post-decrypt comparison. Fix: route the comparison through `subtle::ConstantTimeEq::ct_eq`, which walks the full buffer regardless of where the bytes diverge. The post-comparison branches retain their original shape because the decrypt+lookup path is already executed unconditionally before the compare — only the success-side `DELETE` differs, and that signal is already exposed by the function's return value. Added a regression test (`f1_consume_if_matches_uses_constant_time_compare`) that grep-asserts the production source imports `subtle::ConstantTimeEq`, uses `ConstantTimeEq::ct_eq`, and no longer contains the legacy `!=` shape. Cannot meaningfully prove constant-time-ness from a shared CI runner, but the source-pattern check ensures a "simplifying" revert fails review. Audit: F1 (HIGH). * fix(secrets): use constant-time compare for store key-check sentinel F3 (Low) in the 2026-05 audit: `verify_secret_store_key_check` compared the decrypted sentinel against `SECRET_STORE_KEY_CHECK_PLAINTEXT` with `!=`. The plaintext is a fixed compile-time string so the practical risk is low — an attacker who can move the encrypted_value/key_salt blobs across rows already has full DB write access — but the same constant-time pattern applied to F1 makes the comparison style consistent across the crate and pre-empts a future caller threading a non-constant sentinel through this helper. Routed through `subtle::ConstantTimeEq::ct_eq`, mirroring the F1 fix. Audit: F3 (Low). * fix(processes): index queryable fields and serve records_for_scope via query Replace the N+1 list_dir + per-file get scan with an indexed `query` path, falling back to the legacy scan on byte-only backends so existing LocalFilesystem-driven tests and production deployments remain unaffected. - Declare `ensure_index` lazily for the per-owner `processes/` prefix on the queryable fields called out in the audit (`tenant_id`, `user_id`, `status`, `extension_id`, `parent_process_id`). Backends without index support degrade to the existing scan instead of failing closed. - Project the same fields onto every `ProcessRecord` write via `Entry::with_indexed`; record-capable backends (libSQL, Postgres, the in-memory backend) can now serve scope listings through a native query. The opaque-byte fallback in `put_with_byte_fallback` keeps LocalFilesystem (which rejects record-shaped puts today) on the legacy write path. - Rewrite `records_for_scope` to issue `Filter::And` of `Filter::Eq` predicates against the indexed projection. The full `same_scope_owner` check remains in Rust so the sub-scope axes (agent/project/mission/ thread) that are not yet in the index spec still get filtered. - Add a contract test that exercises the indexed path through `InMemoryBackend` and confirms cross-tenant and cross-user records are not returned. Addresses audit findings F1 (records_for_scope N+1) and F2 (missing ensure_index at startup). * fix(filesystem): surface backend infrastructure errors without fabricated paths F1: SQL backends used valid_engine_path() (unwrap_or_else unreachable returning /engine) as a placeholder on every connection/migration error. The path was always a lie - at pool acquisition, run_migrations, pragma setup, or schema bootstrap there is no caller-supplied virtual path in scope - and it leaked into operator-facing error display. Add FilesystemError::BackendInfrastructure { operation, reason } that omits path. Route every former valid_engine_path() callsite in libsql and postgres through new infrastructure_error helpers in db.rs. The enum is non_exhaustive so adding a variant is backward compatible. Regression test: drive a libsql migration against a read-only DB file and assert BackendInfrastructure with no /engine in display. * fix(filesystem): store VirtualPath keys in InMemoryBackend state directly F2: in_memory.rs::query() reparsed every stored row's path with VirtualPath::new(...).unwrap_or_else(|_| unreachable!('stored paths originated as VirtualPath')) on the hot path. Two issues: - the reparse is wasted work - paths originate as VirtualPath at put() time, so the validation pass on read is redundant - 'unreachable!' is a panic that asserts a structural invariant the type system already enforces Replace HashMap<String, StoredEntry> with HashMap<VirtualPath, StoredEntry>. Lookups now pass &VirtualPath directly; prefix scans move to key.as_str().starts_with(...). VersionedEntry::path comes from a single clone() instead of a parse + unreachable. Existing tests cover the put/get/query/list_dir/stat/delete paths that were touched (44 in_memory tests + the cross-backend contract suite). * fix(filesystem): align in-memory backend on nested VectorNearest semantics F5: SQL backends reject Filter::VectorNearest nested inside And/Or with Unsupported because ranking can't be expressed as a WHERE fragment - the top of query() peels off a top-level VectorNearest before the translator runs, and the translator's VectorNearest arm unconditionally errors. The in-memory backend previously treated a nested VectorNearest as 'any row with IndexValue::Bytes at key', silently changing semantics across backends. Add contains_nested_vector_nearest() pre-check in InMemoryBackend:: query that walks the filter tree and surfaces Unsupported for any VectorNearest strictly inside a compound. The Filter::VectorNearest arm in filter_matches is now unreachable; it returns false to keep the scalar predicate path safe should the pre-check ever be bypassed. Regression test asserts Unsupported on nested-in-And, nested-in-Or, and still-OK for top-level VectorNearest. * fix(filesystem): guard u64 to i64 SQL bindings with typed errors F6: SQL backends used 'expected.get() as i64' and 'page.offset as i64' casts on the CAS and query/pagination paths. Both inputs are u64 and both wrap silently on values >= 2^63 - the cast produces a negative SQL binding that either matches no row (CAS quietly VersionMismatches) or executes against a negative OFFSET (cryptic backend error). Add db.rs helpers: - record_version_to_i64: surfaces CorruptRecordVersion if the value overflows i64 - page_offset_to_i64: surfaces a typed Backend error naming the operation and offset Apply at libsql.rs CAS and query offset bindings and the matching postgres.rs sites. 'page.limit' is u32 clamped to Page::MAX_LIMIT so its i64 cast is safe by construction and uses i64::from for clarity. Regression test asserts a typed Backend(Query) error with reason 'page offset...' when querying with offset = u64::MAX, replacing the prior silent wrap. * fix(filesystem): scope Postgres FTS GIN index to declaring prefix F4: libsql FTS5 virtual tables are declared per-mount-prefix - one vtable per ensure_index(prefix, ...) call - so a query at one prefix can't accidentally pull index postings from a sibling prefix into the plan, and tearing down an index for a prefix is a clean DROP TABLE. The Postgres FTS GIN index, by contrast, was created without a predicate over root_filesystem_entries, so it was global. Correctness held because the query path always scopes by 'path = OR path LIKE ', but parity with libsql broke in two ways: the planner considered postings from every prefix before filtering, and a per-prefix DROP INDEX could only ever tear down one of them. Add a partial-index predicate gated by 'path = <prefix> OR path LIKE <prefix>/%' to the GIN DDL. The prefix is sourced from the validated VirtualPath and quotes are doubled for safe SQL literal embedding; LIKE-special characters are escaped via the existing escape_like_with_trailing_wildcard helper. Regression test (Postgres only; skipped when no DB is reachable) reads back the DDL via pg_indexes.indexdef and asserts the prefix literal and a WHERE clause appear. * fix(filesystem): tighten capability docs, type constraints, and hygiene nits Batched audit findings: F3: Document the type constraint on IndexKind::Prefix. The kind is only meaningful against IndexValue::Text, but ensure_index can't see the value type at declaration time. Filter::PrefixOn rejects every non-text variant at query time. Document the constraint loudly so consumers reach for IndexKind::Exact when projecting numeric or boolean values instead of getting an unused index and a query-time Unsupported. F7: BackendCapabilities::sql_typical advertises a minimum SQL shape that omits IndexFts and IndexVector. The two real backends here (libsql + postgres) layer them on top. A hand-rolled backend that just calls sql_typical() would under-advertise. Add a doc-comment calling out the omission and an sql_typical_full() variant that includes Events + IndexFts + IndexVector for backends that match this crate's shape. F8: validate_simple_identifier indexed bytes[0] after an is_empty guard. The guard makes the index sound, but the pattern is fragile to refactors. Switch to bytes.first() so the dependency is explicit and the panic path goes away. F9: Multiple doc comments in record.rs and index.rs referenced stale type names (StorageBackend::put/list/query, Record). Update to the current RootFilesystem / Entry names. * fix(engine): dedupe events on append_events for HybridStore parity HybridStore (`src/bridge/store_adapter.rs:1613`) de-duplicates thread events by id before insert. The filesystem-store `append_events` impl was previously writing with `CasExpectation::Any`, which silently overwrote an existing event with the same id when callers re-emitted (e.g. recovery after a partial flush). Pre-read the destination path and skip any id already present. Matches HybridStore's append-only contract. Audit finding F3 (Medium) from the ironclaw_engine crate audit. * fix(memory): map FTS results to documents in FilesystemMemoryDocumentRepository::search_documents The previous scaffold issued the `Filter::Fts` query, then silently dropped the results with `let _ = results; Ok(Vec::new())`. A caller wiring up the trait would see an empty result set and assume "no matches" — when in fact the search had simply lied. That is worse than returning `Unsupported`. Map each `VersionedEntry.path` (added in PR #3659) back to a `MemoryDocumentPath`, de-dupe by path, and assign a per-rank score from RRF over the FTS-only branch so the result vector matches the native repos' fusion contract for the trivial single-branch case. Skip non-memory-document entries that may live under the same prefix (chunk projections, metadata siblings). Adds `list_documents_drains_pages_beyond_max_limit` test against the in-memory backend. Audit finding F2 (HIGH) from the ironclaw_memory crate audit. * fix(secrets): close revoke CAS-loop race with versioned compare-and-swap `revoke` previously read the lease via the (now-removed) `read_lease` helper and wrote with `CasExpectation::Any`. The per-lease process-local mutex serialized writers within one process only — multi-process callers sharing the same backend root could observe `Active`, race against `consume`, and clobber a `Consumed` marker by overwriting it with `Revoked`. Inline the read into a bounded CAS retry loop matching `consume` and `consume_session_use`: read with version, write with `CasExpectation::Version`, retry on `VersionMismatch`. Make revoke idempotent on terminal states (`Consumed`, `Revoked`, `Expired`) so the loop converges even when a winner has already written. Audit finding F2 (Medium) from the ironclaw_secrets crate audit. * fix(processes): use versioned CAS for status transitions `update_status` previously read the record and wrote with `CasExpectation::Any`, relying on the per-instance `transition_lock` for atomicity. That lock only serializes within one process; a multi-process deployment sharing the same backend root could observe identical pre-transition state in both processes and clobber each other's status flips. Replace with a bounded CAS retry loop: read with version, validate the transition, write with `CasExpectation::Version`, retry on `VersionMismatch`. Backends that don't support versioning still fall through `put_with_byte_fallback_versioned` to the single- instance lock semantics. Audit finding F3 (Medium) from the ironclaw_processes crate audit. * fix(authorization): close lease-transition CAS race and log lease-read failures H4 (HIGH): The four lease state transitions (`claim`, `consume`, `revoke`) wrote with `CasExpectation::Any`, relying on a per-owner process-local `mutation_lock` for atomicity. Two host processes sharing the same backend root could observe identical pre-transition state and clobber each other — silently double-consuming a one-shot fingerprinted lease, or overwriting a `Consumed` marker with `Revoked`. Introduce `update_lease_cas`: read the lease with its `RecordVersion`, let the caller mutate, write with `CasExpectation::Version(_)`, retry up to `CAS_RETRY_ATTEMPTS` (3) on `VersionMismatch`. `revoke`, `claim`, `consume` are migrated through this helper; the inline state machine in `consume` is preserved. `ensure_claimable`/`ensure_consumable` re- run inside the loop so a race that flips status surfaces the proper typed error rather than a clobber. Backends without per-row versioning (`LocalFilesystem` and legacy byte-only mounts) reject `CasExpectation::Version(_)` with `Unsupported`. For those, `write_lease_raw` falls back to `CasExpectation::Any` and carries the safety invariant via the existing `mutation_lock` — same trade-off documented on `FilesystemCapabilityLeaseStore` and matched by `ironclaw_processes::put_with_byte_fallback`. Two new `CapabilityLeaseError` variants: - `VersionMismatch` (internal CAS-loop signal; never escapes the public API) - `CasExhausted` (transient; caller may retry at a higher level) H5 (HIGH): `LibSqlCapabilityLeaseStore::leases_for_scope` and `PostgresCapabilityLeaseStore::leases_for_scope` previously used `unwrap_or_default()` to collapse DB errors into an empty `Vec`. The trait signature (`-> Vec<CapabilityLease>`) provides no Result channel, so propagation requires a wider refactor. As a tactical mitigation, the error path now `tracing::warn!`s with the underlying error so a DB outage is visible to operators instead of masquerading as "no leases for this scope" (which still fails closed for dispatch, but did so invisibly). Each warn site is annotated `// silent-ok:` per `.claude/rules/error-handling.md`. Audit findings F1 (HIGH) and F2 (HIGH) from the ironclaw_authorization crate audit. * fix(outbound,reborn): clear CI clippy + no-panics gates Two CI-only fixes against the audit-fix integration: 1. `crates/ironclaw_outbound/src/filesystem_store.rs:464` — the `delivery_scope_index_key()` helper used `.expect()` in the inner `unwrap_or_else` arm, tripping the `No panics in production code` check (`scripts/check_no_panics.py`). Replace the per-call construction with a `OnceLock<IndexKey>` that initialises once and converts the "validate" arm to `unreachable!()` with an explicit `// safety:` annotation. Same external behaviour, zero runtime panic surface. 2. `crates/ironclaw_reborn/src/planned_driver.rs:302` — the test module imported `PLANNED_DRIVER_CHECKPOINT_SCHEMA_ID` from the crate root, but the alias resolves to the same const that `super::*` already brings in via the parent module's `use ironclaw_agent_loop:: {CHECKPOINT_SCHEMA_ID, ...}`. Newer clippy correctly flags the import as unused. Drop the redundant import and reference `CHECKPOINT_SCHEMA_ID` directly, matching the sibling assertion on `CHECKPOINT_SCHEMA_VERSION` two lines below. * fix(approvals,engine): close approved-but-no-lease recovery gap; atomize event dedup Two correctness items from the latest PR review (commit4eccad56d): approvals — Review found that inverting the lease/approve ordering in3e8b2303eleft no recovery path for the "approved but no lease" state: if `leases.issue(...)` fails after `approvals.approve(...)` persists, subsequent approve attempts return `NotPending` (status is `Approved`) and resume requires a matching lease that was never created. Security is preserved (fails closed) but availability regresses. Add `retry_lease_issue_for_dispatch` / `retry_lease_issue_for_spawn` that re-issue the lease against a request already in `Approved` status. Both helpers route through a shared `issue_lease_for_approved` that the first-time approve path also uses, so the lease-building shape stays canonical. New `ApprovalResolutionError::NotApproved` variant gates retry calls. engine — Review found that `append_events` did pre-read + write with `CasExpectation::Any`, which is not atomic across processes: two writers can both observe "missing" and last-writer-wins on the same event id. Switch to `CasExpectation::Absent` so the dedup check and insert collapse into a single backend operation; treat `FilesystemError::VersionMismatch` (the typed "already present" signal on Absent CAS) as the expected duplicate-ack. Backends without per-row versioning return `Unsupported` and we fall through to the legacy pre-read pattern so byte-only mounts keep working. * fix(outbound,processes,secrets): byte-only backend compat and AAD/path alignment Three findings from the second-pass PR review (commit4eccad56d): (1) LocalFilesystem byte-only backend compat — the new filesystem stores in `ironclaw_processes`, `ironclaw_outbound`, and `ironclaw_secrets` issue record-shape/`CasExpectation::Version` writes even though the docs and constructor signatures still claim any `RootFilesystem` works. `LocalFilesystem` rejects both with `Unsupported`, so processes started on a local engine filesystem fail during `complete`/`fail`/`kill` (`operation write_file is not supported`), leaving durable local process state stuck `Running`. Reviewer ran `filesystem_process_store_persists_under_resource_scope_engine_processes` and reproduced. - `ironclaw_processes::put_with_byte_fallback` previously retried the byte-stripped entry with the *same* `cas`. Now it also downgrades to `CasExpectation::Any` on the fallback, carrying the safety invariant via the existing `transition_lock`. Reviewer's test now passes. - `ironclaw_outbound::put_with_byte_fallback` is the new equivalent helper, threaded through `put_json` / `put_delivery_attempt_indexed`. - `ironclaw_secrets::put_with_version_fallback` is the equivalent for the lease-revoke / consume / session-use Version-CAS paths. (2) Filesystem secrets AAD/path scope mismatch — `filesystem_secret_aad` bound `mission_id`/`thread_id`/`invocation_id` (full scope), but `secret_path` and `same_scope_owner` key only on owner scope (`tenant/user/agent/project`). A secret written by one invocation appeared present to another invocation in the same owner scope (path layer allowed it) but decryption failed with a confusing "backend unavailable" error. Cross-invocation puts also silently overwrote the same path. Align AAD to `ScopeKey::from_account_scope` (owner-scope-only) so cross-invocation reads within the same owner succeed; cross-owner access still fails at the path layer. * refactor(engine): enforce tenant isolation via ScopedFilesystem in FilesystemStore Per-PR-review concern from serrrfirat on PR #3679 (commit4eccad56d): FilesystemStore must enforce tenant isolation via ScopedFilesystem instead of raw RootFilesystem. The engine was taking `Arc<F: RootFilesystem>` and speaking VirtualPath strings directly to the backend — so any composition layer that forgot to wrap the backend in a tenant scope would leak across tenants, with the type system saying nothing. Migrated the engine's filesystem store surface so tenant isolation is structural, not a convention engine code has to remember: - `FilesystemStore::new` now takes `Arc<ScopedFilesystem<F>>`. - Every path helper in `store/paths.rs` returns `ScopedPath` instead of `VirtualPath`. Path strings are unchanged — `/engine/threads/<id>.json` is now alias-relative under the `/engine` mount alias, and the composition-supplied MountView resolves the alias to a tenant-scoped VirtualPath at every op. - `list_dir`-derived child directory enumeration now strips the leaf segment and reconstructs the follow-up read as a `ScopedPath`, so the per-op ACL still applies (no `VirtualPath` shortcut). - The integration tests construct a ScopedFilesystem over InMemoryBackend with a tenant-scoped VirtualPath target. - New regression test `filesystem_store_isolates_two_tenants_with_same_user_project_ids`: two FilesystemStores share one InMemoryBackend but have different MountViews; writing the same (user_id, project_id, thread_id) on tenant A must not be visible from tenant B. The `Store` trait surface is unchanged. Other consumer crates (ironclaw_processes, ironclaw_secrets, ironclaw_outbound, ironclaw_authorization) are tracked separately and not touched here. Composition wiring (ironclaw_reborn_composition) is updated in a follow-up commit so the engine's `Arc<ScopedFilesystem<F>>` consumer gets the matching `/engine` mount with full read/write/list/delete on the tenant-scoped target. * docs(plans): scoped-filesystem tenant isolation design Captures the multi-tenancy design that the engine FilesystemStore refactor (commitac8e677f9) is the first instance of: - ScopedFilesystem + MountView are the existing Linux-permissions abstraction (read/write/list/delete + alias→VirtualPath rewrite) - Consumer stores accept Arc<ScopedFilesystem<F>> instead of Arc<F> - Composition layer builds one MountView per invocation that maps consumer-facing aliases (/engine, /secrets, ...) to tenant/user-scoped VirtualPaths - Consumer code is tenant-agnostic; tenant prefixing happens at the filesystem layer once, not in every consumer crate Tracks the migration status: engine landing in PR #3679; processes / secrets / outbound / authorization deferred to follow-up PRs that delete their manual tenant-prefixing in favor of the unified surface. References the original review concern from serrrfirat (PR #3679 on 2026-05-16) and the universal-FS-dispatch ADR (docs/reborn/2026-05-14-universal-fs-dispatch.md). * feat(host-api): add MountPermissions::read_write_list_delete() helper Owner-of-private-data convenience constructor that consumer-store migrations to ScopedFilesystem need. Engine FilesystemStore tests already construct this perm-set inline (commitac8e677f9); processes/secrets/outbound/authorization will need the same shape once they migrate to ScopedFilesystem. Centralising in MountPermissions keeps the per-user-owner contract consistent across all consumer crates instead of letting each crate spell out the bool struct literally. * refactor(outbound): enforce tenant isolation via ScopedFilesystem in FilesystemOutboundStateStore `FilesystemOutboundStateStore` now holds `Arc<ScopedFilesystem<F>>` instead of `Arc<F>`. Path helpers return `ScopedPath` rooted at the `/outbound` mount alias; the composition layer's `MountView` resolves the alias to a tenant/user-scoped `VirtualPath` and enforces ACL before any backend dispatch — so cross-tenant isolation is structural rather than something this crate must remember to thread through every path builder. CAS retry loops, indexed `scope` projection, paginated drain, and the byte-only-backend fallback (commit199137b57) are preserved. The `tenant_id` itself moves into the resolved mount prefix; the delivery `scope` index continues to discriminate within an already tenant-scoped subtree so `list_delivery_attempts(scope)` still serves backends without composite-index support. Adds a regression test `filesystem_outbound_store_isolates_two_tenants_with_same_user_project_ids` mirroring the engine-store shape from `ac8e677f9`: two stores share one `InMemoryBackend` but resolve `/outbound` to disjoint tenant subtrees; identical `(agent_id, project_id, thread_id)` policies and delivery attempts on tenant A must not be visible from tenant B. * refactor(authorization): enforce tenant isolation via ScopedFilesystem in FilesystemCapabilityLeaseStore Per the systemic finding tracked in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md` (HIGH on PR #3679, mirror of the engine FilesystemStore fix in commit `ac8e677f9`): `FilesystemCapabilityLeaseStore` was taking `&'a F: RootFilesystem` and hand-formatting every lease path with `/engine/tenants/<tenant_id>/users/<user_id>/...` prefixes. Any composition layer that wrapped the backend without remembering to re-prefix would leak across tenants, and the type system stayed silent. Migrated the store so tenant isolation is structural, not a convention the path builders have to remember: - `FilesystemCapabilityLeaseStore::new` now takes `&'a ScopedFilesystem<F>` (preserves the existing `'a`-borrow shape; no `Arc` wrapping at the call site). - Path helpers return `ScopedPath` rooted at the `/authorization` mount alias. `lease_invocation_root` / `lease_tenant_user_root` / `scoped_owner_root` are gone; the leading `/engine/tenants/{tenant_id}/users/{user_id}` prefix is now the MountView's responsibility, not this crate's. Within-tenant scope (agent/project/mission/thread/invocation) stays in the path. - `CapabilityLeaseIndex` now stores `Vec<ScopedPath>` (was `Vec<VirtualPath>`), so the indexed-listing fast path stays alias-relative end-to-end. - The `list_dir` → child-`get` flow strips the leaf name from the returned `VirtualPath` and rebuilds a `ScopedPath` under the owner prefix (same shape as `FilesystemStore::list_subdir_names` in `ironclaw_engine`), so the follow-up `get` still re-runs the per-op ACL check. Preserved verbatim: - The CAS-Version retry pattern in `update_lease_cas` (H4 fix from commit `4eccad56d`) — `revoke` / `claim` / `consume` still re-read + retry on `FilesystemError::VersionMismatch`. - The `Unsupported → CasExpectation::Any` fallback in `write_lease_raw` for byte-only backends (also H4) — adapted to operate on `ScopedFilesystem`, same logic. - `read_lease_index(...).await?.unwrap_or_default()` (H5 fix) and the rest of the idempotent revoke / claim-consume race rejection semantics. - The `CapabilityLeaseStore` trait surface is unchanged. Tests: - `capability_lease_contract.rs` now builds a `ScopedFilesystem` with `MountPermissions::read_write_list_delete()` on alias `/authorization` → tenant-scoped target. `CountingFilesystem` keeps its role in `filesystem_lease_store_lists_from_owner_index_without_scanning_invocation_roots` by wrapping `LocalFilesystem` *inside* the `ScopedFilesystem`. - New regression test `filesystem_capability_lease_store_isolates_two_tenants_with_same_user_project_ids`: two stores share one `InMemoryBackend` but have different `MountView`s; issuing a lease under tenant A's `(user_id, project_id, invocation_id)` triple must NOT make it visible from tenant B even when tenant B queries with tenant A's scope. All 26 contract tests + 5 DB-backed store tests pass; `cargo clippy -p ironclaw_authorization -p ironclaw_capabilities --all-features --tests -- -D warnings` is clean. * refactor(secrets): enforce tenant isolation via ScopedFilesystem in FilesystemSecretStore/Broker Mirrors the engine migration in commitac8e677f9: `FilesystemSecretStore` and `FilesystemCredentialBroker` previously held `Arc<F: RootFilesystem>` and manually encoded tenant/user identity in every path (`secret_owner_root` formatted `/secrets/tenants/{tenant}/users/{user}/agents/.../projects/...`). Any composition layer that forgot to wrap the backend in a tenant scope would leak across tenants with the type system saying nothing — exactly the HIGH-severity finding addressed in PR #3679 for `ironclaw_engine`. Migrated the secret/credential filesystem stores so tenant isolation is structural, not a convention this crate has to remember: - `FilesystemSecretStore::new` / `with_lease_ttl` and `FilesystemCredentialBroker::new` now take `Arc<ScopedFilesystem<F>>`. - Every path helper (`secret_path`, `lease_path`, `lease_root`, `credential_account_path`, `credential_account_root`, `credential_session_path`) returns `ScopedPath` instead of `VirtualPath`. Paths drop the `/secrets/tenants/<tenant>/users/<user>` prefix and become alias-relative under the `/secrets` mount alias; tenant + user are now resolved per-op by the caller's `MountView`. - `secret_owner_alias` retains the conditional agent/project suffix that the legacy `secret_owner_root` had, so secrets remain partitioned by (agent, project) inside a user's namespace. AAD (`filesystem_secret_aad`) already binds the same `(tenant, user, agent, project, handle)` tuple — sub-scope binding stays intact, and cross-owner reads still fail closed both at the path and at decrypt. - `put_with_version_fallback` now takes `&ScopedFilesystem<F>` / `&ScopedPath`. CAS-retry loops on `consume` / `revoke` / `consume_session_use` and the `Unsupported`→`Any` fallback for byte-only backends are unchanged in behaviour. The constant-time `subtle` compare in `legacy_store` and the revoke/consume CAS race fixes from `df8ae8d1c`/`f04a99813`/`9c079dd76` are untouched. - `leases_for_scope` and `accounts_for_scope` now reconstruct child `ScopedPath`s from `list_dir`'s `VirtualPath` results (via `join_scoped_secret` / `join_scoped_broker`) so the per-op ACL still runs on the follow-up `get`. - Tests build a `ScopedFilesystem` with `MountPermissions::read_write_list_delete()` on alias `/secrets` → tenant-scoped target. The encryption-at-rest assertions resolve the alias-relative path through the same `MountView` to read raw backend bytes. The CAS-retry `VersionRacingBackend` tests watch the resolved `VirtualPath` so the racing semantics are preserved. - Added regression test `filesystem_secret_store_isolates_two_tenants_with_same_user_project_ids`: two `FilesystemSecretStore`s share one `InMemoryBackend` but have different MountViews. Writing the same `(user_id, project_id, handle)` on tenant A must not be visible from tenant B — `metadata` returns `None`, `lease_once` returns `UnknownSecret`, and `leases_for_scope` is empty. - Added regression test `filesystem_secret_store_aad_validates_cross_invocation_within_same_owner` locking in the AAD-alignment fix from commit `199137b57`: AAD binds the owner scope only, so two reads under different invocation/mission/thread but identical `(tenant, user, agent, project)` successfully round-trip through `consume` after the ScopedFilesystem migration. - Replaced the now-meaningless `filesystem_secret_store_isolates_scopes` test (which wrote two different-tenant secrets to one store) with `filesystem_secret_store_isolates_projects_within_same_mount`: within a single store, cross-project consume still fails closed because project remains in the path. Trait surfaces (`SecretStore`, `CredentialAccountStore`, `CredentialSessionStore`) are unchanged. Composition wiring (`ironclaw_reborn_composition`) and other consumer crates are not touched here — to be migrated separately as listed in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. File size note: `filesystem_store.rs` grows from 1740 to ~2020 lines. The added bulk is regression tests + crypto-bound migration helpers that must live alongside the migrated code. The file already exceeds the 1500-line architecture threshold; tracked under the broader ScopedFilesystem migration plan above. * refactor(processes): enforce tenant isolation via ScopedFilesystem in FilesystemProcessStore Mirrors the engine migration in commitac8e677f9and the parallel secrets/authorization/outbound refactors: `FilesystemProcessStore` and `FilesystemProcessResultStore` previously held `Arc<F: RootFilesystem>` and manually encoded tenant/user identity in every path via `resource_owner_root` (formatted `/engine/tenants/<tenant>/users/<user>/agents/.../projects/...`). Any composition layer that forgot to wrap the backend in a tenant scope would leak across tenants with the type system saying nothing — the HIGH-severity class of finding tracked in PR #3679 for `ironclaw_engine`. Migrated the process lifecycle / result stores so tenant isolation is structural rather than something this crate has to re-derive from `ResourceScope.tenant_id` / `user_id`: - `FilesystemProcessStore::new` and `from_arc`, plus `FilesystemProcessResultStore::new` and `from_arc`, now take `Arc<ScopedFilesystem<F>>`. The internal `FilesystemHandle` borrow/Arc enum is gone — both surfaces share the same shape. - Path helpers (`process_record_path`, `process_records_root`, `process_result_path`, `process_output_path`) return `ScopedPath` instead of `VirtualPath`. The on-disk layout under the `/processes` mount alias is: /processes[/agents/<agent>][/projects/<project>][/missions/<mission>][/threads/<thread>]/records/<process_id>.json /processes[/agents/<agent>][/...]/results/<process_id>.json /processes[/agents/<agent>][/...]/outputs/<process_id>/output.json The leading `/engine/tenants/<tenant>/users/<user>` prefix is gone; the caller's `MountView` resolves the `/processes` alias to the tenant/user-scoped target at every op. Sub-scope axes (agent/project/mission/thread) stay in the alias-relative path because they are within-tenant scoping, not covered by the per-tenant MountAlias. - `put_with_byte_fallback` now takes `&ScopedFilesystem<F>` / `&ScopedPath`. The fallback semantics are unchanged: record-shaped entries and non-`Any` CAS are stripped/downgraded if the backend reports `Unsupported`, so byte-only mounts (LocalFilesystem) keep working through the single-instance `transition_lock`. - `records_for_scope_via_list` reconstructs each child `ScopedPath` from the `list_dir`-returned `VirtualPath` leaf so the per-op ACL applies to the follow-up `get` (mirrors the engine store's `join_scoped` shape). - `FilesystemProcessResultStore` resolves the `ScopedPath` output blob to a `VirtualPath` via `ScopedFilesystem::mounts().resolve()` before recording it on the `ProcessResultRecord`, so the on-wire `output_ref` shape stays a tenant-scoped `VirtualPath` and the existing forged-ref rejection in `output()` still works (now comparing against the resolved expected path, not a hand-formatted string). - `ProcessServices::filesystem` constructor takes `Arc<ScopedFilesystem<F>>` instead of `Arc<F>`. Composition wiring in `ironclaw_reborn_composition` is tracked separately under the scoped-filesystem epic and is not touched here. Tests: - `tests/process_store_contract.rs` and `tests/process_services_contract.rs` now construct a `ScopedFilesystem` over `LocalFilesystem` / `InMemoryBackend` with a `MountPermissions::read_write_list_delete()` grant on the `/processes` alias pointing at a tenant1/user1 target. The existing cross-scope assertions (`filesystem_process_result_store_persists_under_resource_scope`, `filesystem_process_result_store_rejects_unexpected_output_refs`, etc.) keep working because the post-read `same_scope_owner` check still filters out forged records whose in-body scope differs from the request scope. - New regression test `filesystem_process_store_isolates_two_tenants_with_same_user_project_ids` wires two `FilesystemProcessStore`s over one `InMemoryBackend` with different `MountView` targets but identical `user_id` / `project_id` request scopes. Writing on tenant A must not be visible from tenant B — fails closed if the ScopedFilesystem wrapping ever regresses to raw `Arc<F: RootFilesystem>`. - Reworked `filesystem_process_store_records_for_scope_uses_index_on_record_backend` to drive within-tenant project discrimination (the new scope axis that lives in the path), since cross-tenant discrimination via the path is now a separate-stores test rather than a single-store test. `cargo clippy -p ironclaw_processes --all-features --tests -- -D warnings` and `cargo test -p ironclaw_processes --all-features` both clean (65 integration tests + 1 unit test). * fix(composition,host-runtime): wire ScopedFilesystem for migrated consumer stores The consumer-crate ScopedFilesystem migrations (commits6ecca195d,5e7688d3b,4ae56769b,81664dd29) changed the public constructors: - FilesystemProcessStore<F> / FilesystemProcessResultStore<F> (lifetime dropped) - ProcessServices::filesystem now takes Arc<ScopedFilesystem<F>> - FilesystemCapabilityLeaseStore<'a, F> now takes &'a ScopedFilesystem<F> Downstream wiring needed catching up: - `ironclaw_reborn_composition` libsql + postgres production builders + the in-process `reborn_app_factory` paths now wrap the raw RootFilesystem in a `ScopedFilesystem` via the new `default_singleton_mount_view()` helper. The view grants read+write+list+delete on the canonical consumer-store aliases (/processes, /secrets, /authorization, /outbound, /engine) mapped to top-level VirtualPath roots — the single-tenant default that preserves current production behaviour while making per-tenant routing a MountView decision instead of a code change. - Type aliases drop the obsolete `'static` lifetime parameter. - `RebornCompositionError::Mount` and `RebornBuildError::Mount` variants surface HostApiError from the MountView construction. - The `reborn_durable_restart_integration` test wraps its LocalFilesystem in a ScopedFilesystem before passing to ProcessServices and FilesystemCapabilityLeaseStore. Per `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. * chore: untrack local test artifacts accidentally committedc60ff0af5included three files that should not have been tracked: - `.anvil/` — local SQLite databases (events.db, memory.db, json cache) created by `cargo test` against the libsql/in-memory backends. Now in .gitignore. - `.claude/rules/architecture.md` and `docs/plans/2026-05-02-engine-architecture-simplification.md` — pre-existing untracked files in the working tree at the start of this session, not part of the migration work. Remove them from the index; .anvil/ also gets a gitignore entry so future test runs don't re-stage the same artifacts. * feat(composition,host-api): per-invocation mount view + shared/system aliases Items 1-3 + 5 from `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`: 1. `invocation_mount_view(scope: &ResourceScope) -> MountView` (pub) — rewrites consumer-store aliases to `/tenants/<tenant>/users/<user>/<alias>` virtual paths. Plus `wrap_scoped_for_invocation(root, scope)` convenience for request handlers building tenant-isolated consumer stores. Five unit tests pin the rewriting contract (cross-tenant disjointness, shared/system carve-outs). 2. `/tenant-shared` mount alias — `read_write` permissions (no delete: tenants can mutate but not erase shared state). In `invocation_mount_view` resolves to `/tenants/<tenant>/shared`; in the singleton default resolves to `/tenant-shared`. 3. `/system` mount alias — `read_only` permissions. Globally readable system data (capability defs, system prompts). The ACL layer rejects writes before any backend dispatch. 5. Doc status table updated to reflect the actual shipped state (engine + processes + secrets + outbound + authorization all migrated to `ScopedFilesystem` in this PR). Adds a "Composition entry points" section explaining when callers use `default_singleton_mount_view` vs `invocation_mount_view`. Plus: `VIRTUAL_ROOTS` in `ironclaw_host_api::path` extended with `/processes`, `/authorization`, `/outbound`, `/tenant-shared`, and `/tenants` so the rewritten target VirtualPaths validate. `/system` generalized from per-subroot entries to a single root. Item #4 (tenant_id indexed projection across consumer crates) is in flight in a separate agent and will land next. * feat(processes): index tenant_id as defense-in-depth scoping projection Add a regression test asserting that every `FilesystemProcessStore::start` write decorates its `Entry` with a `tenant_id` indexed projection. Path-prefix scoping via `ScopedFilesystem` is the primary tenant-isolation boundary; the indexed projection (already in place on the write path since the migration commit) is belt-and-suspenders so an admin-tier query can filter explicitly by tenant and a path-rewriting bug surfaces as a query-time mismatch rather than silent cross-tenant leakage. Refs docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md. * feat(secrets): index tenant_id as defense-in-depth scoping projection Decorate every `FilesystemSecretStore` / `FilesystemCredentialBroker` `Entry` write with a `tenant_id` indexed projection so an admin-tier query can filter explicitly by tenant and a path-rewriting bug surfaces as a query-time mismatch rather than silent cross-tenant leakage. The projection is additive — `ScopedFilesystem` path-prefix scoping remains the primary isolation boundary. Adds: - `index_key_tenant_id()` / `index_name_secrets_tenant()` helpers - `tag_entry_with_tenant()` decorator used by every write site (`write_secret`, `write_lease`, lease consume/revoke transitions, `put_account`, `issue_session`, `consume_session_use`) - `ensure_tenant_id_index_secret`/`_broker` declared on the relevant per-owner subtree; tolerates `Unsupported` for byte-only backends so LocalFilesystem stays writeable - Regression test that writes a secret and asserts both a positive query (matching tenant id) returns one row and a negative query (different tenant id) returns zero rows Refs docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md. * feat(outbound): index tenant_id as defense-in-depth scoping projection Decorate every `FilesystemOutboundStateStore` write (`put_json` and `put_delivery_attempt_indexed`) with a `tenant_id` indexed projection so an admin-tier query can filter explicitly by tenant and a path-rewriting bug surfaces as a query-time mismatch rather than silent cross-tenant leakage. The projection is additive — `ScopedFilesystem` path-prefix scoping remains the primary isolation boundary. `put_json` now takes a `&TenantId` so policies and subscription records can both project their tenant axis from the correct source (`policy.scope.tenant_id` vs `record.scope.stream.tenant_id`). `ensure_tenant_id_index` is declared on the `/outbound/policies`, `/outbound/subscriptions`, and `/outbound/deliveries` roots; tolerates `Unsupported` for byte-only backends. Regression test asserts a positive query (matching tenant id) returns one row and a negative query (different tenant id) returns zero rows. Refs docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md. * feat(authorization): index tenant_id as defense-in-depth scoping projection Decorate every `FilesystemCapabilityLeaseStore::write_lease_raw` and `write_lease_index` `Entry` with a `tenant_id` indexed projection so an admin-tier query can filter explicitly by tenant and a path-rewriting bug surfaces as a query-time mismatch rather than silent cross-tenant leakage. The projection is additive — `ScopedFilesystem` path-prefix scoping remains the primary isolation boundary. Adds: - `index_key_tenant_id()` / `index_name_authorization_tenant()` helpers and `ensure_tenant_id_index` declared on the per-owner lease prefix - Tenant projection on lease and lease-index entries - Byte-only-backend fallback on both writes (strip the projection and downgrade CAS to `Any` when LocalFilesystem returns `Unsupported`), matching the sibling crates' fallback shape - Regression test that issues a lease and asserts a positive `Filter::Eq` query (matching tenant) returns rows and a negative query (different tenant) returns zero rows Refs docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md. * docs(plans): scope tenant_id indexed projection to non-engine consumers Clarify in the tenant-isolation plan's "What this gives us" section that the `tenant_id` defense-in-depth projection covers processes/secrets/ outbound/authorization but deliberately excludes the engine `Store`. The engine trait is single-tenant-by-construction per Open Question 2 — it never sees `tenant_id` internally and has no scope source to drive the projection at the write site, so it relies on path-prefix scoping alone. * fix(host-api,composition,docs): align VIRTUAL_ROOTS with storage-placement contract The architecture boundary test `reborn_virtual_roots_match_storage_placement_contract` (`crates/ironclaw_architecture/tests/reborn_dependency_boundaries.rs`) asserts that `ironclaw_host_api::VIRTUAL_ROOTS` matches the canonical roots in `docs/reborn/contracts/storage-placement.md` and `docs/reborn/contracts/filesystem.md`. The previous commit (`b30c31eae`) added five consumer-store roots to `VIRTUAL_ROOTS` without updating the contract docs and collapsed three `/system/*` subroots into a single `/system` entry, which broke the test in CI. - Restore `/system/settings`, `/system/extensions`, `/system/skills` in `VIRTUAL_ROOTS` (they are still load-bearing — used by `ironclaw_dispatcher` for extension discovery and mount roots). - Keep the five new consumer-store roots (`/processes`, `/authorization`, `/outbound`, `/tenant-shared`, `/tenants`) and add matching rows to both contract docs explaining what each one carries. - Split the unified `/system` mount alias in `default_singleton_mount_view` / `invocation_mount_view` into one alias per canonical subroot, since each subroot is the smallest VirtualPath unit reserved by the contract. Tests updated to match. After this commit, `cargo test -p ironclaw_architecture --test reborn_dependency_boundaries` passes and the composition mount-view tests still pass. * refactor(outbound): delete legacy LibSql/Postgres stores These per-backend OutboundStateStore impls predate the universal-FS dispatch design. Now that FilesystemOutboundStateStore routes through RootFilesystem, the backend choice happens at the RootFilesystem layer (LibSqlRootFilesystem vs PostgresRootFilesystem vs InMemoryBackend) — no need for outbound-specific persistence impls. - Delete src/libsql_store.rs (LibSqlOutboundStateStore) - Delete src/postgres_store.rs (PostgresOutboundStateStore) - Delete src/db.rs (helpers used only by the above) - Delete the four contract tests that exercised them (libsql_persists*, libsql_rejects*, postgres_persists*, postgres_rejects*); durability across reopen is now a property of the RootFilesystem backend, not of outbound-specific persistence - Inline-gate the OutboundPushKind/OutboundDeliveryStatus as_str helpers as #[allow(dead_code)] — they were only consumed by the deleted SQL bodies No production callers; no composition rewiring needed (composition already uses FilesystemOutboundStateStore). * chore: untrack pre-session local docs file * docs(plans): document remaining legacy-store cleanup + trait collapse follow-ups After deleting LibSql/PostgresOutboundStateStore (`d4bf7b3c2`), the remaining cleanup splits cleanly into three independent follow-up PRs: - secrets: needs the master-key decryptability check reproduced on FilesystemSecretStore before composition can switch off LibSqlSecretsStore / PostgresSecretsStore. - authorization: FilesystemCapabilityLeaseStore needs to move from borrowed `&'a ScopedFilesystem<F>` to `Arc<ScopedFilesystem<F>>` so it fits `Arc<dyn CapabilityLeaseStore>` for composition wiring. - run_state: not yet migrated to ScopedFilesystem; the same shape as the other migrated consumers is the prerequisite before LibSqlRunStateStore / PostgresRunStateStore can be deleted. Each is independent (200-400 lines + composition rewiring + tests) and shouldn't pile onto PR #3679. The "trait collapse" further step (rename Filesystem*Store → *Store, delete the trait) widens HostRuntimeServices type signatures and is a deeper structural change deserving its own PR after the SQL impls are gone. * refactor(secrets): add filesystem master-key sentinel check Adds `FilesystemSecretStore::verify_can_decrypt_existing_secrets`, porting the fail-loud-on-master-key-mismatch contract from `LibSqlSecretsStore` / `PostgresSecretsStore` so composition can drop the SQL backends. The sentinel lives at `/secrets/_master_key_check.json` under the caller's MountView; constant-time compare against the fixed plaintext mirrors the libSQL / Postgres implementations the SQL backends carry today. Driven by the "Legacy per-backend store cleanup — ironclaw_secrets" entry in docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md. * refactor(authorization): delete legacy LibSql/Postgres lease stores These per-backend CapabilityLeaseStore impls predate the universal-FS dispatch design. Now that FilesystemCapabilityLeaseStore routes through RootFilesystem (via ScopedFilesystem), the backend choice happens at the RootFilesystem layer (LibSqlRootFilesystem vs PostgresRootFilesystem vs InMemoryBackend) — no need for an authorization-specific persistence impl per backend. Cites the cleanup entry in docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md ("Legacy per-backend store cleanup" → ironclaw_authorization row). - Migrate FilesystemCapabilityLeaseStore from `<'a, F>` + `&'a ScopedFilesystem<F>` to `<F>` + `Arc<ScopedFilesystem<F>>` so composition can hold it as `Arc<dyn CapabilityLeaseStore>` without erasing a lifetime parameter. Same shape as FilesystemSecretStore, FilesystemOutboundStateStore, FilesystemProcessStore. - Preserve the AUTH-CRITICAL H4 CAS-Version retry + Unsupported→Any fallback in update_lease_cas / write_lease_raw — only the borrow→Arc shape changed; serialize/deserialize, mutation_lock, tenant_id projection, and revoke/claim/consume semantics are untouched. - Delete crates/ironclaw_authorization/src/db.rs (LibSqlCapabilityLeaseStore + PostgresCapabilityLeaseStore impls + helpers used only by them). - Delete tests/db_capability_lease_store_contract.rs (the legacy contract test exercised the SQL stores directly; durability across reopen is now a property of the RootFilesystem backend, not of an authorization-specific persistence impl, and the filesystem store reload paths are exercised by capability_lease_contract.rs). - Drop the `libsql` and `postgres` cargo features on ironclaw_authorization (and their `dep:libsql` / `dep:deadpool-postgres` / `dep:tokio-postgres` / `tracing` dependencies — only the deleted SQL code used them). - Composition (lib.rs + factory.rs): replace Arc::new(LibSql/PostgresCapabilityLeaseStore::new(db|pool)) + `.run_migrations().await?` with Arc::new(FilesystemCapabilityLeaseStore::new(Arc::clone(&scoped_filesystem))). Migrations are now the RootFilesystem's responsibility, already run on `filesystem.run_migrations()` earlier in the same function. - Update ironclaw_reborn_composition's `libsql`/`postgres` features to drop their `ironclaw_authorization/libsql` and `/postgres` flags. - Update the host_runtime durable-restart integration test from the leaked `&'static ScopedFilesystem<_>` shape to a plain `Arc<ScopedFilesystem<LocalFilesystem>>`; the Box::leak is gone. Verified: - cargo clippy -p ironclaw_authorization -p ironclaw_capabilities -p ironclaw_reborn_composition --all-features --tests -- -D warnings clean - cargo test -p ironclaw_authorization --all-features: 27 passed - cargo check --workspace --all-features clean * refactor(secrets): delete legacy LibSql/Postgres stores, switch composition The universal-FS dispatch design says backend choice happens at the RootFilesystem layer, not at the consumer-store layer. FilesystemSecretStore + FilesystemCredentialBroker already route through ScopedFilesystem over any RootFilesystem (LibSqlRootFilesystem / PostgresRootFilesystem / InMemoryBackend / LocalFilesystem), so the LibSqlSecretsStore / PostgresSecretsStore / LibSqlCredentialStore / PostgresCredentialStore siblings are vestigial duplication. Composition (`crates/ironclaw_reborn_composition/src/{lib,factory}.rs`) now constructs FilesystemSecretStore over the existing scoped_filesystem (wrap_scoped). The master-key decryptability check that used to live on LibSqlSecretsStore was ported to FilesystemSecretStore::verify_can_decrypt_existing_secrets in commit 36d230e28; composition calls it on every production startup so a master-key mismatch still fails loud. Deleted: - `crates/ironclaw_secrets/src/db.rs` (LibSql + Postgres impls + helpers) - `crates/ironclaw_secrets/tests/db_secret_store_contract.rs` - `crates/ironclaw_secrets/tests/db_credential_store_contract.rs` - `crates/ironclaw_secrets/tests/security_findings_poc.rs` (only exercised the SQL stores) - `pub use db::{LibSqlSecretsStore, …}` re-exports in `lib.rs` - `libsql` / `postgres` features on `ironclaw_secrets` (no consumer left) - `deadpool-postgres` / `libsql` / `tokio-postgres` deps from `ironclaw_secrets/Cargo.toml` - Feature-union references in `ironclaw_reborn` / `ironclaw_reborn_composition` Public `SecretStore` and `CredentialBroker` traits are unchanged; the new `FilesystemSecretStore` already implements them. The `ScopedSecretsStoreAdapter` shim that previously wrapped the legacy SQL stores is also unused now and can be removed in a follow-up (left in place this commit to keep the diff focused on the legacy deletion). Per the design-doc entry "Legacy per-backend store cleanup — ironclaw_secrets" in docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md. * refactor(run-state): enforce tenant isolation via ScopedFilesystem in Filesystem{RunState,ApprovalRequest}Store Mirrors the engine migration in commitac8e677f9and the parallel processes / secrets / authorization / outbound refactors: `FilesystemRunStateStore` and `FilesystemApprovalRequestStore` previously held a `&'a F: RootFilesystem` borrow and manually encoded tenant/user identity in every path via `tenant_user_root` (formatted `/engine/tenants/<tenant>/users/<user>/agents/.../projects/.../runs|approvals`). Any composition layer that forgot to wrap the backend in a tenant scope would leak across tenants with the type system saying nothing — the HIGH-severity class of finding tracked in PR #3679 for `ironclaw_engine`. Migrated the run-state / approval stores so tenant isolation is structural rather than something this crate has to re-derive from `ResourceScope.tenant_id` / `user_id`: - `FilesystemRunStateStore::new` and `FilesystemApprovalRequestStore::new` now take `Arc<ScopedFilesystem<F>>`. The `'a` lifetime parameter on the struct is gone — composition can hold these as `Arc<dyn RunStateStore>` / `Arc<dyn ApprovalRequestStore>` without pinning a borrow. - Path helpers (`run_record_path`, `run_records_root`, `approval_record_path`, `approval_records_root`) return `ScopedPath` instead of `VirtualPath`. The on-disk layout under the `/run-state` and `/approvals` mount aliases is: /run-state[/agents/<agent>][/projects/<project>][/missions/<mission>][/threads/<thread>]/runs/<invocation_id>.json /approvals[/agents/<agent>][/projects/<project>][/missions/<mission>][/threads/<thread>]/<request_id>.json The leading `/engine/tenants/<tenant>/users/<user>` prefix is gone; the caller's `MountView` resolves the alias to a tenant/user-scoped target at every op. Sub-scope axes (agent/project/mission/thread) stay in the alias-relative path because they are within-tenant scoping not covered by the per-tenant `MountAlias`. - `records_for_scope` on both stores reconstructs each child `ScopedPath` from the `list_dir`-returned `VirtualPath` leaf via a new `join_scoped` helper so the per-op ACL still runs on the follow-up `get` (mirrors the engine / processes / secrets / outbound store's `join_scoped` shape). - `put_with_cas` now takes `&ScopedFilesystem<F>` / `&ScopedPath`. The `Unsupported`→`Any` fallback semantics are unchanged, so byte-only backends (`LocalFilesystem`) keep working through the per-path `FILESYSTEM_RECORD_LOCKS` map. - `filesystem_record_lock` now keys on `ScopedPath` instead of `VirtualPath` so the lock is alias-relative (one process-local lock per logical record, not per resolved target). - `tenant_user_root` is removed; its agent/project/mission/thread segments move into the new alias-relative `scope_owner_alias_string` helper. Two sibling mount aliases (`/run-state` + `/approvals`) on one `Arc<ScopedFilesystem<F>>`: composition wires both aliases on the same shared `MountView`, so this crate can drive run-state and approvals through one filesystem handle while keeping the on-disk subtrees distinct. Tests: - `tests/run_state_contract.rs` now constructs a `ScopedFilesystem` over `LocalFilesystem` with `MountPermissions::read_write_list_delete()` grants on both `/run-state` and `/approvals` aliases pointing at tenant1/user1 targets. The existing cross-tenant assertions still pass because the post-read `same_scope_owner` check filters out records whose in-body scope differs from the request scope. - `tests/approval_resolution_contract.rs` follows the same shape with a local `scoped_run_state_fs` helper. - New regression test `filesystem_run_state_store_isolates_two_tenants_with_same_user_project_ids` wires two `FilesystemRunStateStore`s + `FilesystemApprovalRequestStore`s over one `LocalFilesystem` with different `MountView` targets but identical `user_id` / `project_id` / `invocation_id` request scopes. Writing on tenant A must not be visible from tenant B's stores — `get`, `records_for_scope`, `complete`, and `approve` all fail closed. Fails closed if the ScopedFilesystem wrapping ever regresses to raw `&F: RootFilesystem`. Composition + contract changes (so the per-invocation `MountView` and boundary tests stay aligned): - `crates/ironclaw_host_api/src/path.rs::VIRTUAL_ROOTS` adds `/run-state` and `/approvals` (with the other migrated consumer-store roots). - `crates/ironclaw_reborn_composition/src/lib.rs::PER_USER_ALIASES` adds `/run-state` and `/approvals` so both `default_singleton_mount_view` and `invocation_mount_view` expose them with full per-user-owner permissions. - `docs/reborn/contracts/storage-placement.md` and `docs/reborn/contracts/filesystem.md` add matching `/run-state` and `/approvals` rows so the `reborn_virtual_roots_match_storage_placement_contract` boundary test stays green. - `crates/ironclaw_host_runtime/tests/reborn_durable_restart_integration.rs` is rewired to construct an `Arc<ScopedFilesystem<LocalFilesystem>>` (with the new aliases) and pass it to both filesystem run-state stores. The capability-lease store keeps the legacy `&'static ScopedFilesystem<F>` shape until its own migration lands; `leaked_scoped_engine_filesystem` is preserved as the borrow source for now. Trait surfaces (`RunStateStore`, `ApprovalRequestStore`, `RunStateApprovalStore`) are unchanged. The legacy per-backend LibSql / Postgres run-state + approval stores stay in this PR; their deletion is the second pass tracked in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. Plan: docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md * refactor(run-state): delete legacy LibSql/Postgres stores, switch composition Completes the third leg of the legacy-store cleanup. Now that FilesystemRunStateStore + FilesystemApprovalRequestStore route through ScopedFilesystem (commita96d1f63b), the per-backend siblings (LibSqlRunStateStore, PostgresRunStateStore, LibSqlApprovalRequestStore, PostgresApprovalRequestStore, LibSqlRunStateApprovalStore, PostgresRunStateApprovalStore) are vestigial duplication of what RootFilesystem already abstracts. Deleted: - `crates/ironclaw_run_state/src/db.rs` (LibSql + Postgres + shared helpers; ~one large module). - `crates/ironclaw_run_state/tests/db_run_state_store_contract.rs` (was the only suite exercising the SQL impls directly; the ScopedFilesystem-backed contract suite over InMemoryBackend + RootFilesystem covers the same surface). - `pub use db::{LibSql*, Postgres*}` re-exports in `lib.rs`. - `with_libsql_run_state_approval_store` / `with_postgres_run_state_approval_store` builder methods on `HostRuntimeServices` (composition now wires FilesystemRunStateStore + FilesystemApprovalRequestStore directly off the shared `Arc<ScopedFilesystem<F>>` it already constructs). Composition (`crates/ironclaw_reborn_composition/src/{lib,factory}.rs`) and the durable-restart integration test (`crates/ironclaw_host_runtime/tests/reborn_durable_restart_integration.rs`) now use the Arc-based construction shape for all three filesystem-backed stores in the trio (run-state, approval-requests, capability-leases) — same `Arc<ScopedFilesystem<LocalFilesystem>>` shared between them. Per the design-doc entry "Legacy per-backend store cleanup — ironclaw_run_state" in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. * chore(event-projections): delete test orphaned by merge resolution The test was deleted on origin/reborn-integration in commit34886c355(`refactor(crates): group workspace crates by domain`) but my recent merge (7efef49c8) kept the file: independently-modified-on-one-side + deleted-on-the-other resolves to "keep" by default. The kept file references `RebornLibSqlMemoryDocumentRepository` which no longer exists (renamed to `LibSqlMemoryDocumentRepository`), breaking compile. Delete the file to match upstream's intent — the contract is covered elsewhere by `ironclaw_memory::tests::db_memory_repository_contract` and the audit-projection tests inside `ironclaw_event_projections` that don't depend on the renamed type. * refactor(memory): consolidate to FilesystemMemoryDocumentRepository The reborn-native libSQL/Postgres memory document repositories (`RebornLibSqlMemoryDocumentRepository`, `RebornPostgresMemoryDocumentRepository`) are vestigial duplication of what `RootFilesystem` already abstracts — per the same universal-FS dispatch design that drove the secrets/auth/run_state cleanup. `FilesystemMemoryDocumentRepository` over `LibSqlRootFilesystem` / `PostgresRootFilesystem` covers both surfaces with one impl. Implements `MemoryDocumentIndexRepository` (chunk replace + lifecycle hooks) on `FilesystemMemoryDocumentRepository` so the existing `ChunkingMemoryDocumentIndexer` can drive memory indexing through the unified repo — previously only the reborn-native libSQL impl participated. Deleted: - `crates/ironclaw_memory/src/repo/native_libsql.rs` - `crates/ironclaw_memory/src/repo/native_postgres.rs` - All five `reborn_native_*` contract tests in `crates/ironclaw_memory/tests/` — covered by the `FilesystemMemoryDocumentRepository` contract suite over `InMemoryBackend` plus the `RootFilesystem` backend contract tests in `ironclaw_filesystem` itself. `ChunkingMemoryDocumentIndexer` (`indexer.rs`) is unchanged; it delegates to anything that implements `MemoryDocumentIndexRepository` and now has a wider universe of impls to choose from. The dangling `RebornLibSqlMemoryDocumentRepository` reference in `crates/ironclaw_event_projections/tests/memory_significant_events_projection_contract.rs` is updated to use `FilesystemMemoryDocumentRepository` over `InMemoryBackend` — same contract, no DB dependency. CLAUDE.md updated to drop the "legacy `LibSqlMemoryDocumentRepository` / `PostgresMemoryDocumentRepository` still own their own per-backend behavioral coverage" sentence, since those tests are now deleted. Per the design-doc entry "Legacy per-backend store cleanup" in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. * refactor(memory): delete legacy LibSql/Postgres stores The universal-FS dispatch design says backend choice happens at the RootFilesystem layer, not the consumer-store layer. The reborn-native duplicates were already deleted in 27485c575; this commit removes the remaining *legacy* per-backend memory document repositories. They duplicated what FilesystemMemoryDocumentRepository<F> already covers over RootFilesystem (LibSqlRootFilesystem / PostgresRootFilesystem / InMemoryBackend / LocalFilesystem). No production callers — composition has been on FilesystemMemoryDocumentRepository for a while, and no other crate imported LibSqlMemoryDocumentRepository / PostgresMemoryDocumentRepository. Deleted: - `crates/ironclaw_memory/src/repo/libsql.rs` - `crates/ironclaw_memory/src/repo/postgres.rs` - `pub use repo::{LibSql,Postgres}MemoryDocumentRepository` re-exports - `mod libsql` / `mod postgres` declarations in `repo/mod.rs` - Feature-gated helpers `scoped_memory_agent_id`, `db_path_for_memory_document`, `memory_document_from_db_path` in `repo/mod.rs` — only consumed by the deleted impls - `#[cfg(feature = "libsql")]` `encode_embedding_blob` / `decode_embedding_blob` / `cosine_similarity` in `src/embedding.rs` and `escape_fts5_query` in `src/search.rs` — same, only consumed by the deleted libsql repo (the filesystem repo has its own local `encode_embedding_blob`) - `crates/ironclaw_memory/tests/db_memory_repository_contract.rs` - `crates/ironclaw_memory/tests/e2e_hybrid_search.rs` - `crates/ironclaw_memory/tests/e2e_persistence_versioning.rs` - `crates/ironclaw_memory/tests/e2e_scope_isolation_safety.rs` - `crates/ironclaw_memory/tests/memory_filesystem_vertical_integration.rs` - `libsql` / `postgres` features on `ironclaw_memory` (no consumer in the workspace enabled them) and the corresponding optional deadpool-postgres / libsql / pgvector / tokio-postgres deps Memory substrate semantics (versioning, chunk replace, metadata cascade, hybrid search fusion, protected-path safety) retain coverage through `FilesystemMemoryDocumentRepository` over `InMemoryBackend` in `memory_backend_contract.rs` + `memory_filesystem_contract.rs` + the inline `repo/filesystem.rs` `#[cfg(test)]` suite. Backend-specific durability across reopen is now a `RootFilesystem` property and lives in `ironclaw_filesystem`'s own backend contract tests. CLAUDE.md drops the dangling "legacy `LibSqlMemoryDocumentRepository` / `PostgresMemoryDocumentRepository` still own their own per-backend behavioral coverage until their migration is scheduled" tail — the first half of that sentence was already removed in 27485c575; this commit removes the second half, since the tests no longer exist. The `tests/e2e_trace_memory_isolation.rs` ignore-message reference to the deleted `e2e_scope_isolation_safety.rs` test is repointed at the InMemoryBackend contract tests that own the substrate-level SOUL.md rejection coverage now. The CI `Run ironclaw_memory tests` step drops `--features libsql` (no such feature exists any more). Per the design-doc entry "Legacy per-backend store cleanup — ironclaw_memory" in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. * refactor(reborn-event-store): delete legacy LibSql/Postgres stores, route through filesystem The universal-FS dispatch design says backend choice happens at the RootFilesystem layer, not at the durable-log impl layer. FilesystemDurableEventLog / FilesystemDurableAuditLog already route through ScopedFilesystem over any RootFilesystem (LibSqlRootFilesystem / PostgresRootFilesystem / InMemoryBackend), so the LibSql*/Postgres* DurableEventLog / DurableAuditLog siblings that spoke SQL directly are vestigial duplication. The `RebornEventStoreConfig::Libsql { path_or_url, auth_token }` and `::Postgres { url }` variants stay (production composition still matches on the same shape) — but the dispatch now opens a backend- specific RootFilesystem from the same input and wraps it in the unified filesystem-backed durable log. Production policy gates (production InMemory rejection, libSQL cleartext / ambiguous-bare URL rejection, Postgres sslmode=disable rejection, TLS connector construction) move into the event-store crate's `libsql_backed` / `postgres_backed` modules. Deleted: - `crates/ironclaw_reborn_event_store/src/libsql_store.rs` (LibSqlDurableEventLog, LibSqlDurableAuditLog, LibSqlStore, build_libsql_event_stores) - `crates/ironclaw_reborn_event_store/src/postgres_store.rs` (PostgresDurableEventLog, PostgresDurableAuditLog, PostgresStore, build_postgres_event_stores) - `crates/ironclaw_reborn_event_store/src/sql_common.rs` (helpers used only by the deleted SQL impls) - `crates/ironclaw_reborn_event_store/migrations/{libsql,postgres}/*` (replaced by RootFilesystem-layer migrations) - SQL-table corruption fixture tests in `durable_event_store_contract` that punched holes in `reborn_event_entries` — the table no longer exists for events; the contiguity contract is enforced by the RootFilesystem's `append`/`tail` invariants and covered by the `FilesystemDurableEventLog` contract suite (backend-agnostic) plus the surviving public-surface rebuild tests - `dep:url` and `dep:uuid` (only used by the deleted Postgres SQL body); kept `tokio-postgres-rustls` / `rustls` / etc. since the Postgres path still parses the URL and builds the TLS connector before handing the pool to `PostgresRootFilesystem` No composition changes: `with_reborn_event_store_config` still takes `RebornEventStoreConfig` and matches on the same variants. Per the design-doc entry "Legacy per-backend store cleanup — ironclaw_reborn_event_store" in docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md. * refactor(threads): enforce tenant isolation via ScopedFilesystem in FilesystemSessionThreadService Mirrors the run-state / processes / secrets / authorization migrations (commitsa96d1f63b,81664dd29,4ae56769b,5e7688d3b) plus the analogous engine migration inac8e677f9. The new `FilesystemSessionThreadService<F: RootFilesystem>` accepts an `Arc<ScopedFilesystem<F>>` and routes every read/write through the `/threads` mount alias. The composition layer's per-invocation MountView resolves `/threads` to a tenant/user-scoped VirtualPath subtree, so tenant isolation is structural rather than something this crate has to re-derive from `ThreadScope.tenant_id`. Path layout under the alias keeps sub-scope (`agent_id`, `project_id`, `owner_user_id`, `mission_id`) plus the thread id in the alias-relative path so a single tenant/user can own multiple cells: /threads[/agents/<agent>][/projects/<project>][/owners/<owner_user>][/missions/<mission>]/threads/<thread_id>/thread.json /threads.../threads/<thread_id>/messages/<message_id>.json /threads.../threads/<thread_id>/summaries/<summary_id>.json /threads/idempotency/<sha256>.json The thread record holds the per-thread monotonic `next_sequence` counter; `accept_inbound_message` and `append_assistant_draft` do an optimistic CAS on the thread record to reserve a sequence, then write the new per-message record with `CasExpectation::Absent`. Every multi-step transition retries on `FilesystemError::VersionMismatch` and falls back to `CasExpectation::Any` on `Unsupported` so byte-only `LocalFilesystem` backends keep working (matches the existing run-state / authorization store fallback shape). Idempotency records flatten under one `/threads/idempotency/` directory keyed by SHA-256 of the (scope, source_binding_id, external_event_id) tuple — two scopes hash to different keys, so the flat layout is safe. `replay_accepted_inbound_message`, which has no scope input, scans that directory and matches binding/event id against the persisted record body. Wiring: - `/threads` added to `PER_USER_ALIASES` in `ironclaw_reborn_composition` so both `default_singleton_mount_view` and `invocation_mount_view` expose the alias with full r/w/l/d permissions. - `/threads` added to `VIRTUAL_ROOTS` in `ironclaw_host_api::path` so the alias resolves to a recognized top-level subtree. - `docs/reborn/contracts/{filesystem,storage-placement}.md` updated to list `/threads` alongside the other consumer-store mount aliases. Cross-tenant isolation regression test (`filesystem_session_thread_service_isolates_two_tenants_with_same_user_project_ids`) asserts that two `FilesystemSessionThreadService`s sharing one `InMemoryBackend` cannot see each other's threads, messages, or idempotency records when their `MountView`s resolve `/threads` to different tenant subtrees — identical `(agent_id, project_id, owner_user_id, thread_id)` tuples on both sides. Per the design-doc entry "ironclaw_threads" in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. * refactor(threads): delete legacy LibSql/Postgres stores, switch composition Completes the fourth leg of the legacy per-backend store cleanup (after `ironclaw_outbound`, `ironclaw_secrets`, `ironclaw_authorization`, and `ironclaw_run_state`). Now that `FilesystemSessionThreadService` routes through `ScopedFilesystem` (previous commit), the per-backend siblings (`LibSqlSessionThreadService`, `PostgresSessionThreadService`) are vestigial duplication of what `RootFilesystem` already abstracts. Deleted: - `crates/ironclaw_threads/src/db.rs` (LibSql + Postgres impls plus the shared `DurableState` snapshot/replace helpers — one ~1.8k line module). - `crates/ironclaw_threads/tests/db_session_thread_contract.rs` (was the only suite exercising the SQL impls directly; the `FilesystemSessionThreadService` contract suite over `InMemoryBackend` + the in-memory `session_thread_contract.rs` semantic suite cover the same surface, and durability across reopen is now a `RootFilesystem` property tested at the `ironclaw_filesystem` layer). - `pub use db::{LibSql*,Postgres*}` re-exports in `lib.rs`. - `libsql` + `postgres` Cargo features on `ironclaw_threads` (no remaining feature-gated code paths). `deadpool-postgres`, `tokio-postgres`, `libsql`, `tracing`, and `tempfile` (dev) dependencies removed alongside. Composition / call-site wiring: - `crates/ironclaw_reborn/tests/loop_driver_host.rs`: the `turn_runner_worker_completes_after_libsql_turn_and_thread_services_reopen` restart regression now builds a `LibSqlRootFilesystem` + a `ScopedFilesystem` wrapping `/threads → /threads` and constructs the thread service via `FilesystemSessionThreadService::new`. The `libsql-restart-tests` feature now pulls `ironclaw_filesystem/libsql` instead of `ironclaw_threads/libsql` (which no longer exists). `ironclaw_turns/libsql` is still wired because `LibSqlTurnStateStore` has not migrated yet. Per the design-doc entry "Legacy per-backend store cleanup — ironclaw_threads" in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. * refactor(conversations): enforce tenant isolation via ScopedFilesystem in FilesystemConversationStateStore Mirrors the engine / processes / secrets / outbound / authorization / run-state migrations: the legacy `RebornLibSqlConversationStateStore` / `RebornPostgresConversationStateStore` held the substrate handle directly (`Arc<libsql::Database>` / `deadpool_postgres::Pool`) and the caller had to construct per-tenant substrates to keep two tenants' conversation state apart. The new `FilesystemConversationStateStore<F>` takes `Arc<ScopedFilesystem<F>>`; the `MountView` resolves the `/conversations` alias to a tenant/user-scoped `VirtualPath` before any backend dispatch, so tenant isolation is structural rather than a convention the caller must remember. Pass 1 of the two-pass migration described in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. The legacy `RebornLibSql*` / `RebornPostgres*` stores stay in this PR; their deletion is the second pass (mirrors the `run-state` paired pattern in commits `a96d1f63b` + `a238050a2`). What changes: - New `FilesystemConversationStateStore<F: RootFilesystem>` implementing the existing `ConversationStateRepository` trait. Persists at `/conversations/state.json`. Uses `CasExpectation::Version` + bounded retry with `Unsupported` -> `Any` fallback (same `put_with_byte_fallback` shape as the other migrated stores). Writes carry a `tenant_ids` indexed projection — defense-in-depth scoping that mirrors the other migrated consumer crates; the conversation state blob is multi-tenant in shape, so the projection lists every tenant present in the snapshot (under per-invocation MountView, the rewritten path slices to one tenant per file). - New `RebornFilesystemConversationServices` wrapper that wires an `InMemoryConversationServices` over the filesystem store — same shape as `RebornLibSqlConversationServices` / `RebornPostgresConversationServices`. - `state_store` module is now unconditional (was feature-gated to `libsql`/`postgres`). The `state_repository` plumbing on `InMemoryConversationServices` and `InMemoryState::persistence_revision` are unconditional too so the filesystem store can plug in without a new feature flag. - JSON wire envelope `StoredConversationState` flattens HashMaps keyed by struct values (`ActorKey`, `BindingKey`, `ThreadKey`, `ExternalEventRouteKey`, `MessageIdempotencyKey`, `AcceptedMessageReplayKey`, `AcceptedMessageRef`) to `Vec<(K, V)>` because JSON requires HashMap keys to be strings. Legacy libSQL/ Postgres adapters did the same shape via individual rows; this envelope keeps the equivalent contract within a single document. Composition + contracts: - `PER_USER_ALIASES` in `ironclaw_reborn_composition` gains `/conversations`; both `default_singleton_mount_view` and `invocation_mount_view` rewrite it. - `VIRTUAL_ROOTS` in `ironclaw_host_api::path` gains `/conversations`. - `docs/reborn/contracts/storage-placement.md` and `docs/reborn/contracts/filesystem.md` add matching `/conversations` rows so the `reborn_virtual_roots_match_storage_placement_contract` boundary test stays green. Tests: - `tests/filesystem_store_contract.rs` adds two regressions: - `filesystem_conversation_services_round_trip_persisted_state_on_reopen`: writes pair + binding under one services instance, drops, reopens a fresh services instance over the same backend, replays the same resolve and asserts the binding rehydrated from durable storage. - `filesystem_conversation_state_store_isolates_two_tenants_with_same_user_project_ids`: two `RebornFilesystemConversationServices` instances share one `InMemoryBackend` with different `MountView` targets but identical `(user_id, project_id)` request scopes; writing on tenant A must not be visible from tenant B, even though the alias-relative path is identical. Fails closed if the `ScopedFilesystem` wrapping ever regresses to raw `Arc<F: RootFilesystem>`. - All 44 pre-existing `inbound_contract` tests keep passing unchanged. `cargo clippy --workspace --all-features --tests -- -D warnings` and `cargo test -p ironclaw_conversations --all-features` both clean (44 inbound + 2 filesystem-store contract tests). Plan: docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md * refactor(conversations): delete legacy LibSql/Postgres stores Completes the second leg of the conversations migration. Now that FilesystemConversationStateStore routes through ScopedFilesystem (prev commit a46edd88c), the per-backend siblings (RebornLibSqlConversationStateStore, RebornPostgresConversationStateStore, RebornLibSqlConversationServices, RebornPostgresConversationServices) are vestigial duplication of what RootFilesystem already abstracts. Deleted: - `crates/ironclaw_conversations/src/libsql.rs` (787 lines). - `crates/ironclaw_conversations/src/postgres.rs` (785 lines). - `pub use libsql::{...}` / `pub use postgres::{...}` re-exports in `lib.rs` and the matching `mod` declarations. - The pub(crate) `ExternalConversationIdentity::conversation_fingerprint` helper (gated to libsql/postgres; only used by the deleted SQL store body). The public `ExternalConversationRef::conversation_fingerprint` on the ref type stays — adapter crates still depend on it for composite key digesting. - `libsql`, `postgres`, `dep:libsql`, `dep:deadpool-postgres`, `dep:tokio-postgres` features + manifest deps from Cargo.toml. - The transitively-only-used `sha2` crate dependency (the digest helper that backed `conversation_fingerprint` for the SQL composite key indexes is also gone). - Three legacy-store tests in `tests/inbound_contract.rs`: `libsql_conversation_services_survive_restart_for_retry_replay`, `libsql_stale_service_write_fails_without_overwriting_newer_rows`, `postgres_conversation_services_round_trip_restart_replay_when_available`, and the `table_count` helper they shared. Durability across reopen is now a `RootFilesystem` property covered by `filesystem_conversation_services_round_trip_persisted_state_on_reopen` in `tests/filesystem_store_contract.rs`. Composition is untouched because no production composition wired conversations through these legacy stores yet — the migration plan listed conversations under "future work" prior to this PR, and the top-level Reborn composition has no `with_libsql_conversation_state_store` / equivalent builder. New composition wires `RebornFilesystemConversationServices` directly off the shared `Arc<ScopedFilesystem<F>>`. Per the "Legacy per-backend store cleanup — ironclaw_conversations" entry in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. `cargo clippy --workspace --all-features --tests -- -D warnings` and `cargo test -p ironclaw_conversations --all-features` both clean (41 inbound + 2 filesystem-store contract tests). * refactor(turns): enforce tenant isolation via ScopedFilesystem in FilesystemTurnStateStore Mirrors the engine migration in commitac8e677f9and the parallel processes / secrets / authorization / outbound / run-state refactors: adds `FilesystemTurnStateStore<F>` that holds `Arc<ScopedFilesystem<F>>` and routes every read/write through a fixed alias-relative path (`/turns/state.json`). Any composition layer that hands the store a `ScopedFilesystem` constructed with a tenant/user-scoped `MountView` gets cross-tenant isolation structurally — the type system enforces what previously had to be re-derived from `TurnScope.tenant_id` at every call site. Internally the store reuses the existing `InMemoryTurnStateStore::from_persistence_snapshot_with_admission_limit_provider` load + `persistence_snapshot` write pair, so the in-memory state machine (idempotency, admission reservations, active locks, checkpoint sequencing, lifecycle event projection) remains the canonical implementation. The filesystem store wraps every mutating trait method in a CAS-aware read-modify-write loop with bounded retry; reads project through the in-memory store without writing back. Trait surface implemented: - `TurnStateStore` (submit / resume / request_cancel / get_run_state) - `TurnRunTransitionPort` (claim/heartbeat/recover/record_model_route/ block/complete/cancel/fail/record_recovery_required/ apply_validated_loop_exit) - `TurnEventProjectionSource` (read_turn_events_after) - `LoopCheckpointStore` (put/get loop checkpoints) The on-disk layout under the `/turns` mount alias is fixed: /turns/state.json Tenant + user identity moves into the caller's `MountView` per the per-tenant `MountAlias` rewriting, so neither prefix is encoded in the path itself. Within-tenant axes (agent/project/thread) stay in the persisted `TurnScope` on each record because they are not covered by the per-tenant `MountAlias`. Implementation details mirroring the run-state store: - CAS-Version + retry on `VersionMismatch`, `Unsupported -> Any` fallback so byte-only `LocalFilesystem` keeps working through the per-process `FILESYSTEM_RECORD_LOCKS` map. - `submit_turn` runs the run-profile resolver once *outside* the CAS retry loop and threads a pre-resolved resolver into the apply closure — keeps the per-path async lock from spanning the resolver future. - `TurnError::Unavailable { reason }` for filesystem operation failures (matches the existing `db_error` mapping). Composition + contract changes: - `crates/ironclaw_host_api/src/path.rs::VIRTUAL_ROOTS` adds `/turns`. - `crates/ironclaw_reborn_composition/src/lib.rs::PER_USER_ALIASES` adds `/turns` so both `default_singleton_mount_view` and `invocation_mount_view` expose it with per-user-owner permissions. - `docs/reborn/contracts/{filesystem,storage-placement}.md` add the matching `/turns` rows so the `reborn_virtual_roots_match_storage_placement_contract` boundary test stays green. Tests: - New regression test `tests/filesystem_turn_state_contract.rs`: - `filesystem_turn_state_store_persists_submit_and_reopens` — asserts a submitted turn rehydrates after the store is re-constructed from the same scoped filesystem (durability contract). - `filesystem_turn_state_store_hides_records_from_other_tenants_via_mount_view` wires two `FilesystemTurnStateStore`s over one `LocalFilesystem` with different `MountView` targets but identical `(tenant_id, agent_id, project_id, thread_id, idempotency_key)`. Writing on tenant A must not be visible from tenant B's store — `get_run_state` reports `ScopeNotFound`, and tenant B mints its own distinct `run_id` on re-submit. Fails closed if the ScopedFilesystem wrapping ever regresses to a raw `&F: RootFilesystem`. Trait surface (`TurnStateStore`, `TurnRunTransitionPort`, `TurnEventProjectionSource`, `LoopCheckpointStore`) is unchanged. The legacy per-backend `LibSqlTurnStateStore` / `PostgresTurnStateStore` stay in this PR; their deletion is the second pass tracked in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`. Plan: docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md * refactor(turns): delete legacy LibSql/Postgres stores, switch composition Completes the fourth leg of the legacy-store cleanup. Now that FilesystemTurnStateStore routes through ScopedFilesystem (commit ff88e75ca), the per-backend siblings (LibSqlTurnStateStore, PostgresTurnStateStore) are vestigial duplication of what RootFilesystem already abstracts. Their internal load-snapshot / replace-snapshot machinery and their SQL-table schema/migrations are no longer the source of truth — the snapshot blob format is, and the backend choice happens at the RootFilesystem layer. Deleted: - `crates/ironclaw_turns/src/db.rs` (LibSql + Postgres + shared schema/migration/snapshot-load/replace helpers). - `crates/ironclaw_turns/tests/db_turn_state_store_contract.rs` (was the only suite exercising the SQL impls directly; the ScopedFilesystem-backed contract suite added in the previous commit covers the same surface). - `pub use db::{LibSqlTurnStateStore, PostgresTurnStateStore}` and the `mod db` declaration in `lib.rs`. - `with_libsql_turn_state_store` / `with_postgres_turn_state_store` builder methods on `HostRuntimeServices`. Composition now wires a single `with_filesystem_turn_state_store` off the shared `Arc<ScopedFilesystem<F>>` it already constructs. - The `libsql` and `postgres` features on `ironclaw_turns` (along with `libsql`, `tokio-postgres`, `deadpool-postgres` deps). Wiring updates: - `crates/ironclaw_host_runtime/Cargo.toml`: drop `ironclaw_turns/{libsql,postgres}` feature wires. - `crates/ironclaw_host_runtime/src/services.rs`: replace the two per-backend builders with a single `with_filesystem_turn_state_store<F>(Arc<ScopedFilesystem<F>>)`. - `crates/ironclaw_reborn_composition/{lib,factory}.rs`: switch both libSQL and PostgreSQL production wirings to `.with_filesystem_turn_state_store(Arc::clone(&scoped_filesystem))` off the same shared scoped filesystem already used for run-state. - `crates/ironclaw_reborn_composition/Cargo.toml`: drop the `ironclaw_turns/{libsql,postgres}` feature wires. - `crates/ironclaw_reborn/Cargo.toml`: replace `ironclaw_turns/libsql` in `libsql-restart-tests` with `ironclaw_filesystem/libsql` (the libSQL backend lives at the filesystem layer now). Test migrations: - `crates/ironclaw_host_runtime/tests/host_runtime_services_contract.rs`: the three `production_turn_*` tests are rewired around a new `libsql_scoped_turns_fs` helper that constructs an `Arc<ScopedFilesystem<LibSqlRootFilesystem>>` with the canonical `/turns` alias. Production readiness assertions, coordinator submit+reopen, and missing-resolver fail-closed coverage are all preserved against the filesystem-backed store. - `crates/ironclaw_reborn/tests/loop_driver_host.rs`: the `turn_runner_worker_completes_after_libsql_turn_and_thread_services_reopen` restart test is rewired around a new `libsql_filesystem_turn_store` helper. Both the pre-restart and post-restart sides now build a `FilesystemTurnStateStore<LibSqlRootFilesystem>` instead of `LibSqlTurnStateStore`. The thread-side `LibSqlSessionThreadService` is untouched (its own migration is independent). Restart durability is exercised against the libSQL-backed filesystem snapshot. - `crates/ironclaw_turns/tests/loop_checkpoint_store_contract.rs`: drops the libsql/postgres-specific `turn_loop_checkpoints`-table assertions (which were validating SQL-schema separation between `turn_checkpoints` and `turn_loop_checkpoints`) and replaces them with a filesystem-backed roundtrip + snapshot-reopen assertion that exercises the same `loop_checkpoints` vs `checkpoints` projection invariant against `FilesystemTurnStateStore`. InMemory coverage is unchanged. Per the design-doc entry "Legacy per-backend store cleanup — ironclaw_turns" in `docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md` (marked Done in this commit). * refactor(resources): enforce tenant isolation via ScopedFilesystem; delete legacy LibSql/Postgres stores Mirrors the run-state, processes, secrets, authorization, and outbound migrations to the universal-FS-dispatch pattern, completing the legacy per-backend cleanup in `ironclaw_resources`. Pass 1 — add `FilesystemResourceGovernorStore<F: RootFilesystem>`: - New `crates/ironclaw_resources/src/filesystem_store.rs` defines `FilesystemResourceGovernorStore<F>`, an `Arc<ScopedFilesystem<F>>`- based `ResourceGovernorStore` impl that serializes the whole governor snapshot (limits + reservations + usage tallies) as a single record at `/resources/snapshot.json` under the new consumer-store mount alias. Tenant/user identity moves into the caller's `MountView`, so the path itself is alias-relative — same shape as the other consumer stores migrated in this rework. - `ResourceGovernorStore::update` is a sync trait (in-memory + legacy SQL impls were sync); the new filesystem impl crosses to async via a lazily-spawned current-thread tokio worker thread (same shape as the deleted libSQL/Postgres `AsyncStorageWorker`). - Concurrency: a process-local per-path async `Mutex` map serializes in-process read-modify-write cycles; the inner CAS-Version precondition surfaces cross-process races as `ResourceError::Storage` errors (the trait's `FnOnce` closure shape forbids retry, mirroring the libSQL/Postgres `BEGIN IMMEDIATE` / `LOCK TABLE` semantics). - Byte-only backend fallback: `LocalFilesystem` rejects `CasExpectation::Version`/`Absent`, so the store falls back to `CasExpectation::Any` under the same per-path async lock with an emulated `Absent` precheck — matches the other consumer-store migrations. - Adds `/resources` to `VIRTUAL_ROOTS` in `crates/ironclaw_host_api/src/path.rs`, `PER_USER_ALIASES` in `crates/ironclaw_reborn_composition/src/lib.rs`, and the `/resources` row in `docs/reborn/contracts/{filesystem,storage-placement}.md` so the per-invocation `MountView` and boundary tests stay aligned. - Cross-tenant isolation regression test in `filesystem_store.rs::tests::isolates_two_tenants_with_same_user_project_ids` wires two `FilesystemResourceGovernorStore`s over one `InMemoryBackend` with different `MountView` targets but identical `user_id`/`project_id`. Writing on tenant A must not be visible from tenant B's governor — fails closed if the wrapping ever regresses to raw `Arc<F: RootFilesystem>`. Pass 2 — delete legacy stores and switch composition: - Deleted `LibSqlResourceGovernorStore`, `PostgresResourceGovernorStore`, the `AsyncStorageWorker` plumbing, the SQL schema constant, and the worker-cell head-of-line-block regression test from `lib.rs`. - Replaced `with_libsql_resource_governor` / `with_postgres_resource_governor` on `HostRuntimeServices` with a single `with_filesystem_resource_governor<FsBackend>(Arc<ScopedFilesystem>)` builder method. Backend choice is now a `RootFilesystem` property. - `with_resource_governor` is no longer feature-gated — filesystem resource governor wiring is universal. - `crates/ironclaw_reborn_composition/src/{lib,factory}.rs` now construct `FilesystemResourceGovernorStore::new(Arc::clone(&scoped_filesystem))` over the shared `Arc<ScopedFilesystem<F>>` composition already builds via `wrap_scoped` — same wiring shape as the other migrated consumer stores. - `LibSqlProductionHostRuntimeServices` and `PostgresProductionHostRuntimeServices` type aliases now parameterize on `FilesystemResourceGovernorStore<{LibSql,Postgres}RootFilesystem>`. - `crates/ironclaw_resources/Cargo.toml`: `tokio` becomes a non-optional dep with `sync` feature added (always needed for the per-path async lock); `ironclaw_filesystem` added; `libsql` / `postgres` features drop their `tokio` dep (already non-optional). - Tests in `crates/ironclaw_resources/tests/resource_governor_contract.rs` and `crates/ironclaw_host_runtime/tests/host_runtime_services_contract.rs` replace the SQL-specific reload + builder tests with filesystem-backed equivalents over `InMemoryBackend + ScopedFilesystem`; the on-disk snapshot round-trip via a fresh store handle covers the same durability contract the SQL versions did. Tests: `cargo test -p ironclaw_resources --all-features` passes (4 unit + 38 contract); `cargo test -p ironclaw_host_runtime --all-features --test host_runtime_services_contract` passes (81); `cargo clippy --workspace --all-features --tests -- -D warnings` clean; `cargo check --workspace --all-features` clean. Plan: docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md * test(host-runtime): mount consumer virtual roots in durable restart fixture `reborn_durable_restart_integration::approval_resume_*` and `process_result_and_output_*` were panicking with "no backend mount found for virtual path /processes/..." because `mounted_engine_filesystem` only mounted `/engine` on the underlying `LocalFilesystem`, while the `ScopedFilesystem` exposed `/processes`, `/authorization`, `/run-state`, and `/approvals` aliases to the filesystem-backed stores. Add a backend mount for each of those virtual roots, pointing at a sibling subdirectory under `engine_root` so the same on-disk tree is reopened across the restart fixture's service graphs. * test(architecture): permit ironclaw_filesystem dep on threads/turns/resources The boundary suite still forbade `ironclaw_filesystem` from `ironclaw_threads`, `ironclaw_turns`, and `ironclaw_resources`, but each of those crates now routes durable storage through `ScopedFilesystem` under the universal-fs-dispatch rework (matches the same exception already in place for `ironclaw_secrets`). Mirror the secrets-rule comment shape so the intentional edge is documented alongside the rule rather than just removed. * feat(composition): add TenantStoreCache machinery for per-tenant store dispatch Production composition holds one shared `Arc<RootFilesystem>` for the whole process. Consumer-store records must land under `/tenants/<tenant_id>/users/<user_id>/<alias>/…` so two tenants with identically-shaped scopes (same agent, same project, same handle) cannot collide on a shared backend. `invocation_mount_view(scope)` already produces the per-tenant `MountView` rewrite. This commit adds the runtime companion: `TenantStoreCache<F, S>` lazily builds one `Arc<S>` per `(TenantId, UserId)` over a per-tenant `ScopedFilesystem` and caches it for the process lifetime — matching the per-tenant long-lived option in "Open Question 1" of the migration plan. This is commit 1 of 2 (machinery only). Commit 2 will wrap each consumer-store trait in a tenant-dispatching shim that uses this cache, rewire the factories, and delete `wrap_scoped` / `default_singleton_mount_view`. * Revert "feat(composition): add TenantStoreCache machinery for per-tenant store dispatch" This reverts commit544db8770b. * feat(host-api): add ResourceScope::system + from_trusted on id newtypes Adds two pieces used by the upcoming tenant-aware ScopedFilesystem: - `ResourceScope::system()` returns a synthetic scope used for FS ops that have no real per-tenant identity (migrations, admin tooling). Tenant and user fields hold `SYSTEM_RESERVED_ID`, an ASCII Unit-Separator-bracketed sentinel that `TenantId::new` / `UserId::new` reject during validation (`has_forbidden_control`), so no user-supplied identifier can collide. - `<IdNewtype>::from_trusted(String)` lets the system-scope constructor produce sentinel values bypass-validating the canonical id rules. Doc string limits the helper to known-reserved values. No call sites yet — plumbed in the next commit when ScopedFilesystem becomes resolver-based and gains a `&ResourceScope` first arg on every op. * refactor(filesystem): make ScopedFilesystem tenant-aware via MountViewResolver `ScopedFilesystem<F>` now holds a `MountViewResolver` closure instead of a fixed `MountView`. Every op takes `scope: &ResourceScope` as its first argument; the resolver maps that scope to the per-call `MountView` used for permission check + path resolution. Production composition will wire `invocation_mount_view` as the resolver in the follow-up commit, so the same shared `Arc<F>` root produces a tenant-isolated view per call — closing the cross-tenant collision gap that finding #1 of the 2026-05-17 review described. Single-tenant tests / dev fixtures use a new `ScopedFilesystem::with_fixed_view(root, view)` constructor that wraps a constant view in a `_scope`-ignoring resolver. All existing fixtures moved to this shape. Consumer-store changes: - `ironclaw_secrets`, `ironclaw_authorization`, `ironclaw_run_state`, `ironclaw_processes`, `ironclaw_outbound`, `ironclaw_threads`, `ironclaw_conversations`, `ironclaw_turns`, `ironclaw_resources`, `ironclaw_reborn_event_store` — every `self.filesystem.<op>` call site threads its `&ResourceScope` (or a converted `to_resource_scope()` helper for `ThreadScope` / `TurnScope`) into the new signature. - Process-global records (resource governor snapshot, durable event log, conversation state, turn snapshot, idempotency directory) route through `ResourceScope::system()` because their paths already encode per-tenant identity (via stream key or hash) or are intentionally global. Master-key sentinel deletion: - `FilesystemSecretStore::verify_can_decrypt_existing_secrets`, `KEY_CHECK_PATH`, `StoredKeyCheck`, `secret_store_key_check_aad`, and `AAD_DOMAIN_SECRET_STORE_KEY_CHECK` are removed. - The master key continues to come from config/env. A wrong key now surfaces on the first per-tenant decrypt op rather than at startup — by design (per PR #3679 design discussion); the alternative was a per-tenant sentinel that lost the single-readiness-signal anyway. `ResourceScope::system()` + `<IdNewtype>::from_trusted(String)` (from the previous commit) provide the synthetic scope used by these process-global paths; the reserved sentinel uses control characters that `<IdNewtype>::new` rejects, so no caller-supplied identifier can collide. This is the foundation for the production composition rewire + two-tenant regression test that land in the next commits. * feat(composition): wire invocation_mount_view as production resolver Production composition now constructs a single `ScopedFilesystem::new(root, invocation_mount_view)` per process. The resolver rewrites every consumer-store alias to `/tenants/<tenant>/users/<user>/<alias>` on each call, so two tenants sharing the underlying `RootFilesystem` cannot collide on identically-shaped paths. Deleted: - `default_singleton_mount_view` — the identity-mapping view that left finding #1 of the 2026-05-17 review open. - `wrap_scoped_for_invocation` — folded into the new `wrap_scoped` which is now resolver-based. - `default_singleton_mount_view_has_all_consumer_aliases` unit test. - The remaining `verify_can_decrypt_existing_secrets()` startup call sites in `secrets_store_arc` and `factory.rs::build_*_production`, matching the master-key sentinel deletion in the previous commit. Added regression test `two_tenants_with_same_agent_project_handle_do_not_collide_on_put` in `two_tenant_isolation_tests`: builds production-shape composition over `InMemoryBackend`, drives `SecretStore::put`/`lease_once`/`consume` from two distinct `(tenant, user)` scopes with identical agent/project/handle, asserts each tenant reads back their own secret. A regression that restores the singleton resolver trips this test directly. * fix(memory,filesystem): address findings #2-#6 from serrrfirat review Five concrete correctness/security fixes from the 2026-05-17 review: - **#2 — memory document version archive written before CAS** (high correctness). `write_document_with_options` and `compare_and_append_document_with_options` archived the prior content via `save_document_version` *before* the document CAS write. A losing writer therefore left a phantom `.versions/<n>` record for an overwrite/append that never happened. Reordered: CAS write first; archive only after success. - **#3 — stale reindexer could delete winning reindexer's fresh chunks** (high correctness). `replace_document_chunks_if_current` swept the chunk subtree before re-checking the document hash. A racing fresh reindex landing chunks between the initial read and the sweep had its chunks deleted. Moved the second hash check before the sweep so a stale worker bails out without touching the chunk subtree. - **#4 — memory FTS/Vector index never declared** (high correctness). `search_documents` issued `Filter::Fts` / `Filter::VectorNearest` against libSQL / Postgres backends that need a registered index before they can translate the query. Added `ensure_search_indexes(prefix, embedding_dim)` declaring both an FTS index on `content` and a Vector index sized to the query embedding; tolerates `Unsupported` for byte-only backends. Added `MemorySearchRequest::query_embedding_dim` accessor. - **#5 — internal sidecar paths collide with legal user document paths** (high correctness). A user document named `foo.meta` shared its backend path with the metadata sidecar of `foo`. Tightened `validated_memory_relative_path` to reject path segments ending in `.meta`, `.chunks`, or `.versions` (the three reserved sidecar suffixes), with explicit accept/reject tests. - **#6 — `HsmBackend::list_dir` delegated despite no declared List capability** (medium security). The HSM surface advertises only Read/Write/Stat/Delete (`declared_capabilities`) but `list_dir` ran through to the inner backend. Now returns `FilesystemError::Unsupported` so runtime behavior matches the declared capability set, with a regression test. Also removes a dead `wrong_crypto` test helper left over from the deleted master-key sentinel suite (clippy `-D warnings`). * fix(ci): post-merge resolution fixes for reborn-integration - ironclaw_reborn/src/secrets.rs: use ScopedFilesystem::with_fixed_view (resolver-closure new() signature requires a closure now); drop the verify_can_decrypt_existing_secrets call removed in PR #3679 — master key correctness is verified on first per-tenant decrypt op. - ironclaw_turns/src/scope.rs: rewrite TurnScope::to_resource_scope to derive from ResourceScope::system() rather than literally naming InvocationId, satisfying the architecture-boundary test that forbids lower runtime/process identifiers in the turns public surface. - ironclaw_host_runtime/tests/host_runtime_services_contract.rs: drop the scope arg from RootFilesystem::{read,write}_file — the integration branch tightened those signatures to take only the VirtualPath. - cargo fmt normalization across the touched files.
17 lines
720 B
SQL
17 lines
720 B
SQL
-- Reborn RootFilesystem event-plane (`append`/`tail`) backing table.
|
|
-- Stores monotonic per-path event records. `id` is the assigned `SeqNo`;
|
|
-- `path` matches the canonical virtual path. `created_at` is informational
|
|
-- only — ordering is by `id` so a clock skew cannot reshuffle the stream.
|
|
|
|
CREATE TABLE IF NOT EXISTS root_filesystem_events (
|
|
id BIGSERIAL PRIMARY KEY,
|
|
path TEXT NOT NULL CHECK (path LIKE '/%'),
|
|
payload BYTEA NOT NULL,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
|
);
|
|
|
|
-- Index supports the canonical `tail(path, from)` query:
|
|
-- `WHERE path = $1 AND id > $2 ORDER BY id ASC`.
|
|
CREATE INDEX IF NOT EXISTS idx_root_filesystem_events_path_id
|
|
ON root_filesystem_events(path, id);
|