lywsvip/ironclaw
1
0
Fork 0
mirror of https://github.com/nearai/ironclaw.git synced 2026-09-03 08:06:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor(run-state): enforce tenant isolation via ScopedFilesystem in Filesystem{RunState,ApprovalRequest}Store

Browse Source 
Mirrors the engine migration in commit ac8e677f9 and the parallel
processes / secrets / authorization / outbound refactors:
`FilesystemRunStateStore` and `FilesystemApprovalRequestStore`
previously held a `&'a F: RootFilesystem` borrow and manually encoded
tenant/user identity in every path via `tenant_user_root` (formatted
`/engine/tenants/<tenant>/users/<user>/agents/.../projects/.../runs|approvals`).
Any composition layer that forgot to wrap the backend in a tenant
scope would leak across tenants with the type system saying nothing —
the HIGH-severity class of finding tracked in PR #3679 for
`ironclaw_engine`.

Migrated the run-state / approval stores so tenant isolation is
structural rather than something this crate has to re-derive from
`ResourceScope.tenant_id` / `user_id`:

- `FilesystemRunStateStore::new` and `FilesystemApprovalRequestStore::new`
  now take `Arc<ScopedFilesystem<F>>`. The `'a` lifetime parameter on
  the struct is gone — composition can hold these as
  `Arc<dyn RunStateStore>` / `Arc<dyn ApprovalRequestStore>` without
  pinning a borrow.
- Path helpers (`run_record_path`, `run_records_root`,
  `approval_record_path`, `approval_records_root`) return `ScopedPath`
  instead of `VirtualPath`. The on-disk layout under the `/run-state`
  and `/approvals` mount aliases is:

      /run-state[/agents/<agent>][/projects/<project>][/missions/<mission>][/threads/<thread>]/runs/<invocation_id>.json
      /approvals[/agents/<agent>][/projects/<project>][/missions/<mission>][/threads/<thread>]/<request_id>.json

  The leading `/engine/tenants/<tenant>/users/<user>` prefix is gone;
  the caller's `MountView` resolves the alias to a tenant/user-scoped
  target at every op. Sub-scope axes (agent/project/mission/thread)
  stay in the alias-relative path because they are within-tenant
  scoping not covered by the per-tenant `MountAlias`.
- `records_for_scope` on both stores reconstructs each child
  `ScopedPath` from the `list_dir`-returned `VirtualPath` leaf via a
  new `join_scoped` helper so the per-op ACL still runs on the
  follow-up `get` (mirrors the engine / processes / secrets / outbound
  store's `join_scoped` shape).
- `put_with_cas` now takes `&ScopedFilesystem<F>` / `&ScopedPath`. The
  `Unsupported`→`Any` fallback semantics are unchanged, so byte-only
  backends (`LocalFilesystem`) keep working through the per-path
  `FILESYSTEM_RECORD_LOCKS` map.
- `filesystem_record_lock` now keys on `ScopedPath` instead of
  `VirtualPath` so the lock is alias-relative (one process-local lock
  per logical record, not per resolved target).
- `tenant_user_root` is removed; its agent/project/mission/thread
  segments move into the new alias-relative `scope_owner_alias_string`
  helper.

Two sibling mount aliases (`/run-state` + `/approvals`) on one
`Arc<ScopedFilesystem<F>>`: composition wires both aliases on the same
shared `MountView`, so this crate can drive run-state and approvals
through one filesystem handle while keeping the on-disk subtrees
distinct.

Tests:

- `tests/run_state_contract.rs` now constructs a `ScopedFilesystem`
  over `LocalFilesystem` with `MountPermissions::read_write_list_delete()`
  grants on both `/run-state` and `/approvals` aliases pointing at
  tenant1/user1 targets. The existing cross-tenant assertions still
  pass because the post-read `same_scope_owner` check filters out
  records whose in-body scope differs from the request scope.
- `tests/approval_resolution_contract.rs` follows the same shape with
  a local `scoped_run_state_fs` helper.
- New regression test
  `filesystem_run_state_store_isolates_two_tenants_with_same_user_project_ids`
  wires two `FilesystemRunStateStore`s + `FilesystemApprovalRequestStore`s
  over one `LocalFilesystem` with different `MountView` targets but
  identical `user_id` / `project_id` / `invocation_id` request scopes.
  Writing on tenant A must not be visible from tenant B's stores —
  `get`, `records_for_scope`, `complete`, and `approve` all fail closed.
  Fails closed if the ScopedFilesystem wrapping ever regresses to raw
  `&F: RootFilesystem`.

Composition + contract changes (so the per-invocation `MountView` and
boundary tests stay aligned):

- `crates/ironclaw_host_api/src/path.rs::VIRTUAL_ROOTS` adds
  `/run-state` and `/approvals` (with the other migrated consumer-store
  roots).
- `crates/ironclaw_reborn_composition/src/lib.rs::PER_USER_ALIASES`
  adds `/run-state` and `/approvals` so both
  `default_singleton_mount_view` and `invocation_mount_view` expose
  them with full per-user-owner permissions.
- `docs/reborn/contracts/storage-placement.md` and
  `docs/reborn/contracts/filesystem.md` add matching `/run-state` and
  `/approvals` rows so the
  `reborn_virtual_roots_match_storage_placement_contract` boundary
  test stays green.
- `crates/ironclaw_host_runtime/tests/reborn_durable_restart_integration.rs`
  is rewired to construct an `Arc<ScopedFilesystem<LocalFilesystem>>`
  (with the new aliases) and pass it to both filesystem run-state
  stores. The capability-lease store keeps the legacy
  `&'static ScopedFilesystem<F>` shape until its own migration lands;
  `leaked_scoped_engine_filesystem` is preserved as the borrow source
  for now.

Trait surfaces (`RunStateStore`, `ApprovalRequestStore`,
`RunStateApprovalStore`) are unchanged. The legacy per-backend
LibSql / Postgres run-state + approval stores stay in this PR; their
deletion is the second pass tracked in
`docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md`.

Plan: docs/plans/2026-05-16-scoped-filesystem-tenant-isolation.md
This commit is contained in:
ilblackdragon@gmail.com
2026-05-16 23:06:32 -07:00
parent f0abb79d67
commit a96d1f63b0
8 changed files with 402 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
crates/ironclaw_host_api/src/path.rs
View File

@@ -78,6 +78,8 @@ const VIRTUAL_ROOTS: &[&str] = &[
"/processes",
"/authorization",
"/outbound",
"/run-state",
"/approvals",
"/tenant-shared",
"/tenants",
];

36
crates/ironclaw_host_runtime/tests/reborn_durable_restart_integration.rs
View File

@@ -332,20 +332,20 @@ type DurableHostRuntimeServices = HostRuntimeServices<
struct DurableServices {
services: DurableHostRuntimeServices,
run_state: Arc<FilesystemRunStateStore<'static, LocalFilesystem>>,
approval_requests: Arc<FilesystemApprovalRequestStore<'static, LocalFilesystem>>,
run_state: Arc<FilesystemRunStateStore<LocalFilesystem>>,
approval_requests: Arc<FilesystemApprovalRequestStore<LocalFilesystem>>,
capability_leases: Arc<FilesystemCapabilityLeaseStore<LocalFilesystem>>,
events: RebornEventStores,
}
async fn durable_services(engine_root: &Path, event_root: &Path) -> DurableServices {
let event_stores = jsonl_event_stores(event_root).await;
let run_state_fs = leaked_engine_filesystem(engine_root);
let approval_fs = leaked_engine_filesystem(engine_root);
let lease_scoped_fs = scoped_engine_filesystem(engine_root);
let run_state = Arc::new(FilesystemRunStateStore::new(run_state_fs));
let approval_requests = Arc::new(FilesystemApprovalRequestStore::new(approval_fs));
let capability_leases = Arc::new(FilesystemCapabilityLeaseStore::new(lease_scoped_fs));
// All three filesystem-backed stores now take `Arc<ScopedFilesystem<F>>`
// (run_state migrated in commit 475588153; capability lease in 34e3c68cb).
let scoped_fs = scoped_engine_filesystem(engine_root);
let run_state = Arc::new(FilesystemRunStateStore::new(Arc::clone(&scoped_fs)));
let approval_requests = Arc::new(FilesystemApprovalRequestStore::new(Arc::clone(&scoped_fs)));
let capability_leases = Arc::new(FilesystemCapabilityLeaseStore::new(Arc::clone(&scoped_fs)));
let services = base_services(
engine_root,
event_stores.clone(),
@@ -426,10 +426,26 @@ fn durable_mount_view() -> MountView {
VirtualPath::new("/authorization").unwrap(),
MountPermissions::read_write_list_delete(),
),
MountGrant::new(
MountAlias::new("/run-state").unwrap(),
VirtualPath::new("/run-state").unwrap(),
MountPermissions::read_write_list_delete(),
),
MountGrant::new(
MountAlias::new("/approvals").unwrap(),
VirtualPath::new("/approvals").unwrap(),
MountPermissions::read_write_list_delete(),
),
])
.unwrap()
}
/// Build a fresh [`ScopedFilesystem`] over a [`LocalFilesystem`] rooted at
/// `engine_root`. The restart contract spawns multiple service graphs against
/// the same on-disk root, so each call here constructs a distinct
/// `ScopedFilesystem` over a freshly-mounted `LocalFilesystem`; identity of
/// the wrapping struct is irrelevant — durability lives on disk, and the
/// per-path lock map is process-global by design.
fn scoped_engine_filesystem(engine_root: &Path) -> Arc<ScopedFilesystem<LocalFilesystem>> {
Arc::new(ScopedFilesystem::new(
Arc::new(mounted_engine_filesystem(engine_root)),
@@ -449,10 +465,6 @@ fn mounted_engine_filesystem(engine_root: &Path) -> LocalFilesystem {
filesystem
}
fn leaked_engine_filesystem(engine_root: &Path) -> &'static LocalFilesystem {
Box::leak(Box::new(mounted_engine_filesystem(engine_root)))
}
async fn block_for_approval(
runtime: &impl HostRuntime,
context: ExecutionContext,

2
crates/ironclaw_reborn_composition/src/lib.rs
View File

@@ -105,6 +105,8 @@ const PER_USER_ALIASES: &[&str] = &[
"/secrets",
"/authorization",
"/outbound",
"/run-state",
"/approvals",
"/engine",
];

218
crates/ironclaw_run_state/src/lib.rs
View File

@@ -3,9 +3,13 @@
//! `ironclaw_run_state` stores the current lifecycle state for host-managed
//! invocations. It is separate from runtime events: events are append-only
//! history, while run state answers "what is this invocation waiting on now?".
//! Feature-gated PostgreSQL and libSQL stores provide transactional durable
//! backends for production composition; in-memory and filesystem stores remain
//! useful for tests, local demos, and single-process profiles.
//!
//! Durable persistence is provided by [`FilesystemRunStateStore`] and
//! [`FilesystemApprovalRequestStore`] over a
//! [`ScopedFilesystem`](ironclaw_filesystem::ScopedFilesystem). The
//! `RootFilesystem` choice (libSQL-backed, PostgreSQL-backed, in-memory, or
//! local-disk) is made at the filesystem layer — the consumer-store level no
//! longer carries per-backend impls.
#[cfg(any(feature = "libsql", feature = "postgres"))]
mod db;
@@ -23,11 +27,11 @@ use std::{
use async_trait::async_trait;
use ironclaw_filesystem::{
CasExpectation, ContentType, Entry, FilesystemError, FilesystemOperation, RecordVersion,
RootFilesystem,
RootFilesystem, ScopedFilesystem,
};
use ironclaw_host_api::{
AgentId, ApprovalRequest, ApprovalRequestId, CapabilityId, HostApiError, InvocationId,
MissionId, ProjectId, ResourceScope, TenantId, ThreadId, UserId, VirtualPath,
MissionId, ProjectId, ResourceScope, ScopedPath, TenantId, ThreadId, UserId,
};
use serde::{Deserialize, Serialize};
use thiserror::Error;
@@ -545,19 +549,29 @@ impl ApprovalRequestStore for InMemoryApprovalRequestStore {
/// expected to be a backend error, not a routine condition.
const FILESYSTEM_CAS_RETRIES: usize = 8;
/// Filesystem-backed run-state store under resource-owner-scoped `/engine` paths.
pub struct FilesystemRunStateStore<'a, F>
/// Filesystem-backed run-state store under the `/run-state` mount alias.
///
/// Construct with a [`ScopedFilesystem`] over any [`RootFilesystem`]. The
/// [`ScopedFilesystem`] resolves the `/run-state` alias to a
/// tenant/user-scoped [`VirtualPath`](ironclaw_host_api::VirtualPath) per
/// its [`MountView`](ironclaw_host_api::MountView) and enforces per-op ACL
/// before any backend dispatch — so tenant isolation is structural rather
/// than something this crate has to re-derive from `ResourceScope.tenant_id`
/// / `user_id`. Within-tenant axes (agent/project/mission/thread) remain in
/// the alias-relative path because they are not covered by the per-tenant
/// `MountAlias`.
pub struct FilesystemRunStateStore<F>
where
F: RootFilesystem,
{
filesystem: &'a F,
filesystem: Arc<ScopedFilesystem<F>>,
}
impl<'a, F> FilesystemRunStateStore<'a, F>
impl<F> FilesystemRunStateStore<F>
where
F: RootFilesystem,
{
pub fn new(filesystem: &'a F) -> Self {
pub fn new(filesystem: Arc<ScopedFilesystem<F>>) -> Self {
Self { filesystem }
}
@@ -608,7 +622,7 @@ where
mutate(&mut record);
let entry = Self::record_entry(&record)?;
match put_with_cas(
self.filesystem,
self.filesystem.as_ref(),
&path,
entry,
CasExpectation::Version(version),
@@ -628,7 +642,7 @@ where
}
#[async_trait]
impl<F> RunStateStore for FilesystemRunStateStore<'_, F>
impl<F> RunStateStore for FilesystemRunStateStore<F>
where
F: RootFilesystem,
{
@@ -645,7 +659,14 @@ where
error_kind: None,
};
let entry = Self::record_entry(&record)?;
match put_with_cas(self.filesystem, &path, entry, CasExpectation::Absent).await {
match put_with_cas(
self.filesystem.as_ref(),
&path,
entry,
CasExpectation::Absent,
)
.await
{
Ok(()) => Ok(record),
Err(PutError::VersionMismatch) => Err(RunStateError::InvocationAlreadyExists {
invocation_id: record.invocation_id,
@@ -745,7 +766,11 @@ where
let mut records = Vec::new();
for entry in entries {
if entry.name.ends_with(".json") {
let Some(versioned) = self.filesystem.get(&entry.path).await? else {
// `list_dir` returns post-resolution `VirtualPath`s; reconstruct
// the alias-relative `ScopedPath` so the follow-up `get` still
// runs through the per-op ACL.
let child = join_scoped(&root, &entry.name)?;
let Some(versioned) = self.filesystem.get(&child).await? else {
continue;
};
let record = deserialize::<RunRecord>(&versioned.entry.body)?;
@@ -759,19 +784,24 @@ where
}
}
/// Filesystem-backed approval request store under resource-owner-scoped `/engine` paths.
pub struct FilesystemApprovalRequestStore<'a, F>
/// Filesystem-backed approval request store under the `/approvals` mount alias.
///
/// See [`FilesystemRunStateStore`] for the structural-tenant-isolation
/// rationale; this store applies the same shape to approval-request records
/// under a sibling mount alias so a single composition can wire run state
/// and approvals to distinct alias targets while sharing one backend.
pub struct FilesystemApprovalRequestStore<F>
where
F: RootFilesystem,
{
filesystem: &'a F,
filesystem: Arc<ScopedFilesystem<F>>,
}
impl<'a, F> FilesystemApprovalRequestStore<'a, F>
impl<F> FilesystemApprovalRequestStore<F>
where
F: RootFilesystem,
{
pub fn new(filesystem: &'a F) -> Self {
pub fn new(filesystem: Arc<ScopedFilesystem<F>>) -> Self {
Self { filesystem }
}
@@ -821,7 +851,7 @@ where
record.status = status;
let entry = Self::record_entry(&record)?;
match put_with_cas(
self.filesystem,
self.filesystem.as_ref(),
&path,
entry,
CasExpectation::Version(version),
@@ -841,7 +871,7 @@ where
}
#[async_trait]
impl<F> ApprovalRequestStore for FilesystemApprovalRequestStore<'_, F>
impl<F> ApprovalRequestStore for FilesystemApprovalRequestStore<F>
where
F: RootFilesystem,
{
@@ -859,7 +889,14 @@ where
status: ApprovalStatus::Pending,
};
let entry = Self::record_entry(&record)?;
match put_with_cas(self.filesystem, &path, entry, CasExpectation::Absent).await {
match put_with_cas(
self.filesystem.as_ref(),
&path,
entry,
CasExpectation::Absent,
)
.await
{
Ok(()) => Ok(record),
Err(PutError::VersionMismatch) => Err(RunStateError::ApprovalRequestAlreadyExists {
request_id: record.request.id,
@@ -938,7 +975,12 @@ where
let mut records = Vec::new();
for entry in entries {
if entry.name.ends_with(".json") {
let Some(versioned) = self.filesystem.get(&entry.path).await? else {
// See `FilesystemRunStateStore::records_for_scope` — `list_dir`
// returns post-resolution `VirtualPath`s; rebuild the
// alias-relative `ScopedPath` so the follow-up `get` runs
// through the per-op ACL.
let child = join_scoped(&root, &entry.name)?;
let Some(versioned) = self.filesystem.get(&child).await? else {
continue;
};
let record = deserialize::<ApprovalRecord>(&versioned.entry.body)?;
@@ -952,34 +994,98 @@ where
}
}
// Path layout under the `/run-state` and `/approvals` mount aliases:
//
// /run-state[/agents/<agent>][/projects/<project>][/missions/<mission>][/threads/<thread>]/runs/<invocation_id>.json
// /approvals[/agents/<agent>][/projects/<project>][/missions/<mission>][/threads/<thread>]/<request_id>.json
//
// Tenant + user identity moves into the caller's `MountView` per the
// per-tenant `MountAlias` rewriting, so neither prefix is encoded in the
// path itself. Within-tenant sub-scope axes (agent/project/mission/thread)
// stay in the alias-relative path because they are within-tenant scoping
// not covered by the per-tenant `MountAlias`.
const RUN_STATE_PREFIX: &str = "/run-state";
const APPROVALS_PREFIX: &str = "/approvals";
fn run_record_path(
scope: &ResourceScope,
invocation_id: InvocationId,
) -> Result<VirtualPath, RunStateError> {
VirtualPath::new(format!(
) -> Result<ScopedPath, RunStateError> {
scoped_path(&format!(
"{}/{invocation_id}.json",
run_records_root(scope)?.as_str()
run_records_root_string(scope)
))
.map_err(invalid_path)
}
fn run_records_root(scope: &ResourceScope) -> Result<VirtualPath, RunStateError> {
VirtualPath::new(format!("{}/runs", tenant_user_root(scope))).map_err(invalid_path)
fn run_records_root(scope: &ResourceScope) -> Result<ScopedPath, RunStateError> {
scoped_path(&run_records_root_string(scope))
}
fn run_records_root_string(scope: &ResourceScope) -> String {
format!("{}/runs", scope_owner_alias_string(RUN_STATE_PREFIX, scope))
}
fn approval_record_path(
scope: &ResourceScope,
request_id: ApprovalRequestId,
) -> Result<VirtualPath, RunStateError> {
VirtualPath::new(format!(
) -> Result<ScopedPath, RunStateError> {
scoped_path(&format!(
"{}/{request_id}.json",
approval_records_root(scope)?.as_str()
approval_records_root_string(scope)
))
.map_err(invalid_path)
}
fn approval_records_root(scope: &ResourceScope) -> Result<VirtualPath, RunStateError> {
VirtualPath::new(format!("{}/approvals", tenant_user_root(scope))).map_err(invalid_path)
fn approval_records_root(scope: &ResourceScope) -> Result<ScopedPath, RunStateError> {
scoped_path(&approval_records_root_string(scope))
}
fn approval_records_root_string(scope: &ResourceScope) -> String {
scope_owner_alias_string(APPROVALS_PREFIX, scope)
}
/// Build the alias-relative owner prefix for a scope under the given mount
/// alias. Tenant and user are intentionally absent — they live in the
/// `MountView` the caller supplied. Sub-scope axes (agent/project/mission/
/// thread) stay in the path so within-tenant cross-scope isolation still
/// works for stores sharing one alias target.
fn scope_owner_alias_string(prefix: &'static str, scope: &ResourceScope) -> String {
let mut base = String::from(prefix);
if let Some(agent_id) = &scope.agent_id {
base.push_str("/agents/");
base.push_str(agent_id.as_str());
}
if let Some(project_id) = &scope.project_id {
base.push_str("/projects/");
base.push_str(project_id.as_str());
}
if let Some(mission_id) = &scope.mission_id {
base.push_str("/missions/");
base.push_str(mission_id.as_str());
}
if let Some(thread_id) = &scope.thread_id {
base.push_str("/threads/");
base.push_str(thread_id.as_str());
}
base
}
fn scoped_path(raw: &str) -> Result<ScopedPath, RunStateError> {
ScopedPath::new(raw).map_err(invalid_path)
}
/// Join a leaf segment onto a [`ScopedPath`] prefix. Mirrors the engine /
/// processes / secrets / outbound stores' `join_scoped` helper: `list_dir`
/// returns post-resolution [`VirtualPath`](ironclaw_host_api::VirtualPath)s,
/// but the follow-up `get` must run through the `ScopedFilesystem` so the
/// per-op ACL is enforced — so callers strip the leaf name and rejoin it
/// onto the original `ScopedPath` prefix.
fn join_scoped(prefix: &ScopedPath, leaf: &str) -> Result<ScopedPath, RunStateError> {
scoped_path(&format!(
"{}/{}",
prefix.as_str().trim_end_matches('/'),
leaf
))
}
type FilesystemRecordLock = Arc<tokio::sync::Mutex<()>>;
@@ -1002,7 +1108,7 @@ type FilesystemRecordLock = Arc<tokio::sync::Mutex<()>>;
static FILESYSTEM_RECORD_LOCKS: OnceLock<Mutex<HashMap<String, Weak<tokio::sync::Mutex<()>>>>> =
OnceLock::new();
fn filesystem_record_lock(path: &VirtualPath) -> FilesystemRecordLock {
fn filesystem_record_lock(path: &ScopedPath) -> FilesystemRecordLock {
let locks = FILESYSTEM_RECORD_LOCKS.get_or_init(|| Mutex::new(HashMap::new()));
let mut guard = locks
.lock()
@@ -1023,27 +1129,6 @@ fn filesystem_record_lock(path: &VirtualPath) -> FilesystemRecordLock {
fresh
}
fn tenant_user_root(scope: &ResourceScope) -> String {
let mut base = format!(
"/engine/tenants/{}/users/{}",
scope.tenant_id.as_str(),
scope.user_id.as_str()
);
if let Some(agent_id) = &scope.agent_id {
base = format!("{base}/agents/{}", agent_id.as_str());
}
if let Some(project_id) = &scope.project_id {
base = format!("{base}/projects/{}", project_id.as_str());
}
if let Some(mission_id) = &scope.mission_id {
base = format!("{base}/missions/{}", mission_id.as_str());
}
if let Some(thread_id) = &scope.thread_id {
base = format!("{base}/threads/{}", thread_id.as_str());
}
base
}
fn invalid_path(error: HostApiError) -> RunStateError {
RunStateError::InvalidPath(error.to_string())
}
@@ -1100,8 +1185,8 @@ enum PutError {
/// in-process lock map; cross-process callers fall back to the documented
/// process-local limitation.
async fn put_with_cas<F>(
filesystem: &F,
path: &VirtualPath,
filesystem: &ScopedFilesystem<F>,
path: &ScopedPath,
entry: Entry,
cas: CasExpectation,
) -> Result<(), PutError>
@@ -1153,10 +1238,7 @@ mod lock_map_tests {
#[test]
fn filesystem_record_lock_returns_same_arc_while_holders_alive() {
let path = VirtualPath::new(
"/engine/tenants/lockmap-share/users/u/projects/p/runs/share.json".to_string(),
)
.unwrap();
let path = ScopedPath::new("/run-state/projects/p/runs/share.json".to_string()).unwrap();
let a = filesystem_record_lock(&path);
let b = filesystem_record_lock(&path);
assert!(
@@ -1172,14 +1254,8 @@ mod lock_map_tests {
// first path's entry must no longer be reachable. Demonstrates
// the map does not grow unboundedly with tenant/path churn
// (audit finding F1).
let path = VirtualPath::new(
"/engine/tenants/lockmap-prune/users/u/projects/p/runs/prune.json".to_string(),
)
.unwrap();
let other = VirtualPath::new(
"/engine/tenants/lockmap-prune/users/u/projects/p/runs/other.json".to_string(),
)
.unwrap();
let path = ScopedPath::new("/run-state/projects/p/runs/prune.json".to_string()).unwrap();
let other = ScopedPath::new("/run-state/projects/p/runs/other.json".to_string()).unwrap();
let arc = filesystem_record_lock(&path);
assert!(

37
crates/ironclaw_run_state/tests/approval_resolution_contract.rs
View File

@@ -1,3 +1,6 @@
use std::sync::Arc;
use ironclaw_filesystem::{RootFilesystem, ScopedFilesystem};
use ironclaw_host_api::*;
use ironclaw_run_state::*;
@@ -101,8 +104,8 @@ async fn approval_store_rejects_duplicate_pending_save() {
#[tokio::test]
async fn filesystem_approval_store_rejects_second_resolution_attempt() {
let fs = engine_filesystem();
let store = FilesystemApprovalRequestStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemApprovalRequestStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
let approval = approval_request(invocation_id);
@@ -124,8 +127,8 @@ async fn filesystem_approval_store_rejects_second_resolution_attempt() {
#[tokio::test]
async fn filesystem_approval_store_rejects_duplicate_pending_save() {
let fs = engine_filesystem();
let store = FilesystemApprovalRequestStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemApprovalRequestStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
let approval = approval_request(invocation_id);
@@ -163,6 +166,32 @@ fn engine_filesystem() -> ironclaw_filesystem::LocalFilesystem {
fs
}
/// Build a [`ScopedFilesystem`] exposing `/run-state` and `/approvals`
/// aliases under a single tenant/user subtree of the underlying mount.
/// Mirrors the production composition shape where one `MountView` covers
/// both consumer aliases for a given tenant/user.
fn scoped_run_state_fs<F>(backend: Arc<F>) -> Arc<ScopedFilesystem<F>>
where
F: RootFilesystem,
{
let mounts = MountView::new(vec![
MountGrant::new(
MountAlias::new("/run-state").expect("alias"),
VirtualPath::new("/engine/tenants/test-tenant/users/test-user/run-state")
.expect("target"),
MountPermissions::read_write_list_delete(),
),
MountGrant::new(
MountAlias::new("/approvals").expect("alias"),
VirtualPath::new("/engine/tenants/test-tenant/users/test-user/approvals")
.expect("target"),
MountPermissions::read_write_list_delete(),
),
])
.expect("mount view");
Arc::new(ScopedFilesystem::new(backend, mounts))
}
fn sample_scope(invocation_id: InvocationId, tenant: &str, user: &str) -> ResourceScope {
ResourceScope {
tenant_id: TenantId::new(tenant).unwrap(),

220
crates/ironclaw_run_state/tests/run_state_contract.rs
View File

@@ -1,4 +1,5 @@
use std::{
sync::Arc,
sync::atomic::{AtomicBool, AtomicUsize, Ordering},
time::Duration,
};
@@ -6,6 +7,7 @@ use std::{
use async_trait::async_trait;
use ironclaw_filesystem::{
DirEntry, FileStat, FilesystemError, FilesystemOperation, LocalFilesystem, RootFilesystem,
ScopedFilesystem,
};
use ironclaw_host_api::*;
use ironclaw_run_state::*;
@@ -143,8 +145,8 @@ async fn in_memory_run_state_rejects_duplicate_invocation_in_same_tenant_user()
#[tokio::test]
async fn filesystem_run_state_rejects_duplicate_invocation_in_same_tenant_user() {
let fs = engine_filesystem();
let store = FilesystemRunStateStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemRunStateStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
@@ -182,9 +184,10 @@ async fn filesystem_run_state_rejects_duplicate_invocation_in_same_tenant_user()
#[tokio::test]
async fn filesystem_run_state_duplicate_start_is_serialized_across_store_instances() {
let fs = ConcurrentMissingReadFilesystem::new(engine_filesystem());
let first_store = FilesystemRunStateStore::new(&fs);
let second_store = FilesystemRunStateStore::new(&fs);
let fs = Arc::new(ConcurrentMissingReadFilesystem::new(engine_filesystem()));
let scoped = scoped_run_state_fs(fs);
let first_store = FilesystemRunStateStore::new(Arc::clone(&scoped));
let second_store = FilesystemRunStateStore::new(scoped);
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
@@ -301,9 +304,10 @@ async fn in_memory_run_state_hides_records_from_other_tenants_and_users() {
}
#[tokio::test]
async fn filesystem_run_state_store_persists_records_under_tenant_user_engine_runs() {
let fs = engine_filesystem();
let store = FilesystemRunStateStore::new(&fs);
async fn filesystem_run_state_store_persists_records_under_run_state_alias() {
let fs = Arc::new(engine_filesystem());
let scoped = scoped_run_state_fs(fs);
let store = FilesystemRunStateStore::new(Arc::clone(&scoped));
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
let approval = approval_request(invocation_id);
@@ -321,7 +325,7 @@ async fn filesystem_run_state_store_persists_records_under_tenant_user_engine_ru
.await
.unwrap();
let reloaded = FilesystemRunStateStore::new(&fs)
let reloaded = FilesystemRunStateStore::new(Arc::clone(&scoped))
.get(&scope, invocation_id)
.await
.unwrap()
@@ -330,7 +334,7 @@ async fn filesystem_run_state_store_persists_records_under_tenant_user_engine_ru
assert_eq!(reloaded.status, RunStatus::BlockedApproval);
assert_eq!(reloaded.approval_request_id, Some(approval.id));
assert_eq!(
FilesystemRunStateStore::new(&fs)
FilesystemRunStateStore::new(scoped)
.records_for_scope(&scope)
.await
.unwrap()
@@ -341,8 +345,8 @@ async fn filesystem_run_state_store_persists_records_under_tenant_user_engine_ru
#[tokio::test]
async fn filesystem_run_state_store_hides_records_from_other_tenants_and_users() {
let fs = engine_filesystem();
let store = FilesystemRunStateStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemRunStateStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let tenant_a = sample_scope(invocation_id, "tenant1", "user1");
let tenant_b = sample_scope(invocation_id, "tenant2", "user1");
@@ -371,10 +375,10 @@ async fn filesystem_run_state_store_hides_records_from_other_tenants_and_users()
}
#[tokio::test]
async fn filesystem_approval_request_store_persists_pending_requests_under_tenant_user_engine_approvals()
{
let fs = engine_filesystem();
let store = FilesystemApprovalRequestStore::new(&fs);
async fn filesystem_approval_request_store_persists_pending_requests_under_approvals_alias() {
let fs = Arc::new(engine_filesystem());
let scoped = scoped_run_state_fs(fs);
let store = FilesystemApprovalRequestStore::new(Arc::clone(&scoped));
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
let approval = approval_request(invocation_id);
@@ -387,7 +391,7 @@ async fn filesystem_approval_request_store_persists_pending_requests_under_tenan
assert_eq!(record.scope, scope);
assert_eq!(record.status, ApprovalStatus::Pending);
assert_eq!(record.request, approval);
let reloaded = FilesystemApprovalRequestStore::new(&fs)
let reloaded = FilesystemApprovalRequestStore::new(scoped)
.get(&record.scope, record.request.id)
.await
.unwrap()
@@ -397,9 +401,10 @@ async fn filesystem_approval_request_store_persists_pending_requests_under_tenan
#[tokio::test]
async fn filesystem_approval_request_duplicate_save_is_serialized_across_store_instances() {
let fs = ConcurrentMissingReadFilesystem::new(engine_filesystem());
let first_store = FilesystemApprovalRequestStore::new(&fs);
let second_store = FilesystemApprovalRequestStore::new(&fs);
let fs = Arc::new(ConcurrentMissingReadFilesystem::new(engine_filesystem()));
let scoped = scoped_run_state_fs(fs);
let first_store = FilesystemApprovalRequestStore::new(Arc::clone(&scoped));
let second_store = FilesystemApprovalRequestStore::new(scoped);
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
let approval = approval_request(invocation_id);
@@ -436,8 +441,8 @@ async fn filesystem_approval_request_duplicate_save_is_serialized_across_store_i
#[tokio::test]
async fn filesystem_approval_request_listing_ignores_records_deleted_after_list() {
let fs = DisappearingApprovalReadFilesystem::new(engine_filesystem());
let store = FilesystemApprovalRequestStore::new(&fs);
let fs = Arc::new(DisappearingApprovalReadFilesystem::new(engine_filesystem()));
let store = FilesystemApprovalRequestStore::new(scoped_run_state_fs(Arc::clone(&fs)));
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
let approval = approval_request(invocation_id);
@@ -468,8 +473,8 @@ async fn in_memory_approval_request_store_discards_pending_request() {
#[tokio::test]
async fn filesystem_approval_request_store_discards_pending_request() {
let fs = engine_filesystem();
let store = FilesystemApprovalRequestStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemApprovalRequestStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let scope = sample_scope(invocation_id, "tenant1", "user1");
let approval = approval_request(invocation_id);
@@ -522,8 +527,8 @@ async fn in_memory_approval_store_allows_same_request_id_in_different_tenants()
#[tokio::test]
async fn approval_request_store_hides_records_from_other_tenants_and_users() {
let fs = engine_filesystem();
let store = FilesystemApprovalRequestStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemApprovalRequestStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let tenant_a = sample_scope(invocation_id, "tenant1", "user1");
let tenant_b = sample_scope(invocation_id, "tenant2", "user1");
@@ -579,8 +584,8 @@ async fn run_state_isolates_records_by_agent_scope() {
#[tokio::test]
async fn filesystem_run_state_uses_agent_scoped_paths() {
let fs = engine_filesystem();
let store = FilesystemRunStateStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemRunStateStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let agent_a = sample_scope_with_agent(invocation_id, "tenant1", "user1", Some("agent-a"));
let agent_b = sample_scope_with_agent(invocation_id, "tenant1", "user1", Some("agent-b"));
@@ -656,8 +661,8 @@ async fn run_state_isolates_records_by_project_scope() {
#[tokio::test]
async fn filesystem_run_state_isolates_records_by_project_scope() {
let fs = engine_filesystem();
let store = FilesystemRunStateStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemRunStateStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let project_a = sample_scope(invocation_id, "tenant1", "user1");
let mut project_b = project_a.clone();
@@ -758,8 +763,8 @@ async fn approval_request_store_isolates_records_by_project_scope() {
#[tokio::test]
async fn filesystem_approval_request_store_isolates_records_by_project_scope() {
let fs = engine_filesystem();
let store = FilesystemApprovalRequestStore::new(&fs);
let fs = Arc::new(engine_filesystem());
let store = FilesystemApprovalRequestStore::new(scoped_run_state_fs(fs));
let invocation_id = InvocationId::new();
let project_a = sample_scope(invocation_id, "tenant1", "user1");
let mut project_b = project_a.clone();
@@ -785,6 +790,113 @@ async fn filesystem_approval_request_store_isolates_records_by_project_scope() {
assert_eq!(store.records_for_scope(&project_a).await.unwrap().len(), 1);
}
/// Regression for the ScopedFilesystem migration: two stores share one
/// underlying [`RootFilesystem`] but each is constructed with a
/// [`MountView`] whose `/run-state` and `/approvals` aliases resolve to a
/// different tenant-scoped [`VirtualPath`] subtree. Writing the same
/// `(user_id, project_id, invocation_id)` tuple on tenant A's store must
/// NOT make the record visible from tenant B's store. Before this
/// migration, the filesystem run-state store held a raw `&F: RootFilesystem`
/// and encoded tenant identity in the path itself — any composition layer
/// that forgot to prefix the path with tenant would leak across tenants,
/// with the type system saying nothing. The structural fix routes every op
/// through `ScopedFilesystem`, so two MountViews over the same backend
/// cannot see each other's data.
#[tokio::test]
async fn filesystem_run_state_store_isolates_two_tenants_with_same_user_project_ids() {
let backend = Arc::new(engine_filesystem());
let scoped_a = scoped_run_state_fs_at(Arc::clone(&backend), "tenant-a", "alice");
let scoped_b = scoped_run_state_fs_at(Arc::clone(&backend), "tenant-b", "alice");
let runs_a = FilesystemRunStateStore::new(Arc::clone(&scoped_a));
let runs_b = FilesystemRunStateStore::new(Arc::clone(&scoped_b));
let approvals_a = FilesystemApprovalRequestStore::new(scoped_a);
let approvals_b = FilesystemApprovalRequestStore::new(scoped_b);
// Identical `(user_id, project_id, invocation_id)` for both — the only
// thing keeping the two stores apart is the mount-time tenant prefix.
let invocation_id = InvocationId::new();
let scope_a = ResourceScope {
tenant_id: TenantId::new("tenant-a").unwrap(),
user_id: UserId::new("alice").unwrap(),
agent_id: None,
project_id: Some(ProjectId::new("project-1").unwrap()),
mission_id: None,
thread_id: None,
invocation_id,
};
let scope_b = ResourceScope {
tenant_id: TenantId::new("tenant-b").unwrap(),
..scope_a.clone()
};
let approval = approval_request(invocation_id);
let request_id = approval.id;
runs_a
.start(RunStart {
invocation_id,
capability_id: CapabilityId::new("echo.say").unwrap(),
scope: scope_a.clone(),
})
.await
.unwrap();
approvals_a
.save_pending(scope_a.clone(), approval)
.await
.unwrap();
// Tenant A sees its own run and approval.
assert!(
runs_a.get(&scope_a, invocation_id).await.unwrap().is_some(),
"tenant A must see the run it just wrote"
);
assert!(
approvals_a
.get(&scope_a, request_id)
.await
.unwrap()
.is_some(),
"tenant A must see the approval it just wrote"
);
// Tenant B's stores do NOT see tenant A's records, despite identical
// (user_id, project_id, invocation_id, request_id). Both `get` and
// `records_for_scope` must fail closed; transitions targeted at
// tenant B's view of the same id must report unknown.
assert!(
runs_b.get(&scope_b, invocation_id).await.unwrap().is_none(),
"tenant B must NOT see tenant A's run (cross-tenant path leak)"
);
assert!(
approvals_b
.get(&scope_b, request_id)
.await
.unwrap()
.is_none(),
"tenant B must NOT see tenant A's approval (cross-tenant path leak)"
);
assert!(
runs_b.records_for_scope(&scope_b).await.unwrap().is_empty(),
"tenant B records_for_scope must be empty under shared (user, project)"
);
assert!(
approvals_b
.records_for_scope(&scope_b)
.await
.unwrap()
.is_empty(),
"tenant B approvals records_for_scope must be empty under shared (user, project)"
);
assert!(matches!(
runs_b.complete(&scope_b, invocation_id).await.unwrap_err(),
RunStateError::UnknownInvocation { .. }
));
assert!(matches!(
approvals_b.approve(&scope_b, request_id).await.unwrap_err(),
RunStateError::UnknownApprovalRequest { .. }
));
}
struct ConcurrentMissingReadFilesystem {
inner: LocalFilesystem,
missing_reads: AtomicUsize,
@@ -953,6 +1065,11 @@ impl RootFilesystem for DisappearingApprovalReadFilesystem {
}
}
/// Build a [`LocalFilesystem`] with `/engine` mounted to a tempdir. The
/// `/run-state` and `/approvals` mount aliases on the outer
/// [`ScopedFilesystem`] resolve under `/engine/...` so the legacy local-disk
/// fault-injection wrappers (which match by post-resolution path) keep
/// working unchanged.
fn engine_filesystem() -> LocalFilesystem {
let storage = tempfile::tempdir().unwrap().keep();
let mut fs = LocalFilesystem::new();
@@ -964,6 +1081,45 @@ fn engine_filesystem() -> LocalFilesystem {
fs
}
/// Wrap a [`RootFilesystem`] in a [`ScopedFilesystem`] that exposes
/// `/run-state` and `/approvals` aliases, both rooted under a single
/// tenant/user subtree of the underlying mount. Tests share one
/// `MountView` between the run-state and approval stores so a single
/// composition can drive both surfaces over the same backend (the
/// production composition shape).
fn scoped_run_state_fs<F>(backend: Arc<F>) -> Arc<ScopedFilesystem<F>>
where
F: RootFilesystem,
{
scoped_run_state_fs_at(backend, "test-tenant", "test-user")
}
/// Variant of [`scoped_run_state_fs`] that resolves the `/run-state` and
/// `/approvals` aliases under a caller-chosen tenant/user prefix. Used by
/// the cross-tenant isolation regression test to materialize two
/// `ScopedFilesystem`s with disjoint `MountView` targets over the same
/// `RootFilesystem`.
fn scoped_run_state_fs_at<F>(backend: Arc<F>, tenant: &str, user: &str) -> Arc<ScopedFilesystem<F>>
where
F: RootFilesystem,
{
let tenant_user_prefix = format!("/engine/tenants/{tenant}/users/{user}");
let mounts = MountView::new(vec![
MountGrant::new(
MountAlias::new("/run-state").expect("alias"),
VirtualPath::new(format!("{tenant_user_prefix}/run-state")).expect("target"),
MountPermissions::read_write_list_delete(),
),
MountGrant::new(
MountAlias::new("/approvals").expect("alias"),
VirtualPath::new(format!("{tenant_user_prefix}/approvals")).expect("target"),
MountPermissions::read_write_list_delete(),
),
])
.expect("mount view");
Arc::new(ScopedFilesystem::new(backend, mounts))
}
fn sample_scope(invocation_id: InvocationId, tenant: &str, user: &str) -> ResourceScope {
ResourceScope {
tenant_id: TenantId::new(tenant).unwrap(),

4
docs/reborn/contracts/filesystem.md
View File

@@ -102,6 +102,8 @@ Frozen V1 canonical virtual roots (aligned with `storage-placement.md`):
/processes
/authorization
/outbound
/run-state
/approvals
/tenant-shared
/tenants
```
@@ -124,6 +126,8 @@ Recommended meaning:
| `/processes` | background-process records and result/output blobs (consumer-store mount alias under `ironclaw_processes`) |
| `/authorization` | capability lease records (consumer-store mount alias under `ironclaw_authorization`) |
| `/outbound` | outbound delivery policy/subscription/attempt records (consumer-store mount alias under `ironclaw_outbound`) |
| `/run-state` | invocation-lifecycle run-state records (consumer-store mount alias under `ironclaw_run_state`) |
| `/approvals` | approval-request lifecycle records (sibling consumer-store mount alias under `ironclaw_run_state`) |
| `/tenant-shared` | data shared between users/agents in the same tenant; resolves to `/tenants/<tenant_id>/shared/...` per [scoped-filesystem-tenant-isolation](../../plans/2026-05-16-scoped-filesystem-tenant-isolation.md) |
| `/tenants` | reserved root for tenant-scoped target subtrees written by the per-invocation `MountView` (`/tenants/<tenant_id>/users/<user_id>/<alias>/...`); not consumed directly by stores |

2
docs/reborn/contracts/storage-placement.md
View File

@@ -142,6 +142,8 @@ not bypass domain invariants by mutating primitive storage rows directly.
| `/processes` | typed process-lifecycle repository routed through `ironclaw_filesystem` (records, results, outputs) | process APIs | no | Consumer mount alias for `ironclaw_processes`; alias-relative under the per-invocation `MountView`. |
| `/authorization` | typed capability-lease repository routed through `ironclaw_filesystem` | lease APIs | no | Consumer mount alias for `ironclaw_authorization`; alias-relative under the per-invocation `MountView`. |
| `/outbound` | typed outbound-delivery repository routed through `ironclaw_filesystem` (policies, subscriptions, attempts) | outbound APIs | indexed scope projection | Consumer mount alias for `ironclaw_outbound`; alias-relative under the per-invocation `MountView`. |
| `/run-state` | typed invocation-lifecycle repository routed through `ironclaw_filesystem` (run records) | run-state APIs | no | Consumer mount alias for `ironclaw_run_state`; alias-relative under the per-invocation `MountView`. |
| `/approvals` | typed approval-request repository routed through `ironclaw_filesystem` (approval records) | run-state APIs | no | Sibling consumer mount alias for `ironclaw_run_state`; alias-relative under the per-invocation `MountView`. |
| `/tenant-shared` | per-tenant shared mount; resolves to `/tenants/<tenant_id>/shared/...` under the per-invocation `MountView` | scoped filesystem | no | Data shared between users/agents in the same tenant. |
| `/tenants` | reserved root for tenant-scoped target subtrees written by the per-invocation `MountView` | scoped filesystem | no | Not a consumer-visible alias; only consumed at the mount-table layer by the rewritten `VirtualPath` targets (`/tenants/<tenant_id>/users/<user_id>/<alias>/...`). |