lywsvip/ironclaw
1
0
Fork 0
mirror of https://github.com/nearai/ironclaw.git synced 2026-09-03 08:06:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(tests): per-test isolated Postgres databases for the two WS12 parity-blocking contract suites

Browse Source 
The WS12 gauntlet (ws12-gauntlet-report.md §P6/§P8) measured the Postgres
legs of ironclaw_event_store's durable_event_store_contract and
ironclaw_assistant's durable_ledger_contract as test-isolation-defective:
absolute database-global asserts (event cursors; settled-entry prune
bookkeeping) run against the single external database named by their
IRONCLAW_*_POSTGRES_URL env vars. Every failing test passes alone on a
virgin database - store semantics correct, suites not self-isolating
(PROPOSAL §12.13 D-T).

Fix: each affected test provisions a private database on the configured
server - the fabric contract's IsolatedDatabase pattern
(db_root_filesystem_contract.rs) ported locally into each suite: CREATE
DATABASE per test, store/pool + migrations against it, courtesy
DROP ... WITH (FORCE), and a once-per-binary stale-name sweep. Every
assertion preserved byte-identical; libsql/jsonl twins untouched. In the
ledger suite only the two retention tests move - the other six Postgres
tests keep their proven fingerprint-suffix isolation.

Regression pins are the fixed tests themselves:
- postgres_replay_advances_next_cursor_past_trailing_filtered_records
- postgres_runtime_and_audit_logs_survive_rebuild_with_filtered_cursor_semantics
- postgres_settled_entry_limit_prunes_oldest_when_configured
- postgres_settled_prune_interval_defers_until_interval_when_configured
Green proven on a shared dirty database twice in a row (parallel default
threading) and serially on a virgin database; red-first reproduction
captured before the fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
BenKurrek
2026-08-05 22:22:53 -04:00
parent 4a652010ba
commit 864d93ee9b
3 changed files with 306 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
crates/events/ironclaw_event_store/Cargo.toml
View File

@@ -39,4 +39,8 @@ webpki-roots = "1.0"
chrono = { version = "0.4", features = ["serde"] }
ironclaw_common = { path = "../../contracts/ironclaw_common", version = "0.4.2" }
tempfile = "3"
tokio = { version = "1", features = ["macros", "rt"] }
tokio = { version = "1", features = ["macros", "rt", "sync"] }
# The Postgres contract leg provisions a private database per test (the
# fabric contract's isolation pattern), which needs an admin connection
# beside the store-owned pool.
tokio-postgres = "0.7"

159
crates/events/ironclaw_event_store/tests/durable_event_store_contract.rs
View File

@@ -501,12 +501,148 @@ async fn libsql_runtime_and_audit_logs_survive_rebuild_with_filtered_cursor_sema
Some("project-a".to_string())
);
}
// ─── Postgres per-test isolation ──────────────────────────────────────────
//
// The two `postgres_*` tests below assert *absolute* cursor values
// (`EventCursor::new(1)`…), and Postgres cursor assignment draws on
// database-wide state: any other row in the database shifts the numbers.
// Unique scope suffixes isolate record filtering but not cursor assignment,
// so the suite ran red as one invocation against the single database named
// by `IRONCLAW_REBORN_EVENT_STORE_POSTGRES_URL` while every test passed
// alone on a virgin database (WS12 gauntlet report, §P6). Each test
// therefore provisions a private database on the configured server — the
// fabric contract's `IsolatedDatabase` pattern
// (`crates/substrates/ironclaw_filesystem/tests/db_root_filesystem_contract.rs`)
// — which keeps the absolute assertions meaningful, exactly as the
// jsonl/libsql twins keep theirs through per-test temp files.
struct IsolatedPostgresDatabase {
/// Connection string for the private database, in the same libpq form
/// the configured URL used.
url: String,
admin: tokio_postgres::Client,
name: String,
}
impl IsolatedPostgresDatabase {
/// Drop the database on the way out of a passing test.
///
/// A courtesy, not a guarantee: a failing assertion unwinds straight
/// past it, so a red run can leave its database behind — that is what
/// the sweep in `isolated_postgres_database` collects on the next run.
/// `FORCE` closes store-pool connections that have not gone away by the
/// time the handles drop.
async fn cleanup(self) {
let Self { admin, name, .. } = self;
let _ = admin
.execute(&format!("DROP DATABASE IF EXISTS {name} WITH (FORCE)"), &[])
.await;
}
}
/// One stale-database sweep per test binary, ahead of every creation.
/// Sweeping per provisioning call (as the fabric contract does) can race a
/// sibling test of the same binary between its `CREATE DATABASE` and first
/// connection; a single up-front sweep removes that window while still
/// collecting what failed runs left behind.
static STALE_SWEEP: tokio::sync::OnceCell<()> = tokio::sync::OnceCell::const_new();
/// Distinguishes the databases of concurrently provisioning tests.
static NEXT_DATABASE: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
async fn isolated_postgres_database() -> Option<IsolatedPostgresDatabase> {
let base_url = match std::env::var("IRONCLAW_REBORN_EVENT_STORE_POSTGRES_URL") {
Ok(url) => url,
Err(_) => {
eprintln!(
"skipping postgres event-store contract: IRONCLAW_REBORN_EVENT_STORE_POSTGRES_URL not set"
);
return None;
}
};
// Past a configured URL, every failure below is a broken environment
// rather than an unconfigured one, so it panics — this leg has no CI
// executor, and a silent skip would let the suite pass while testing
// nothing.
let admin_config = base_url
.parse::<tokio_postgres::Config>()
.expect("IRONCLAW_REBORN_EVENT_STORE_POSTGRES_URL parses as a postgres connection string");
let (admin, connection) = admin_config
.connect(tokio_postgres::NoTls)
.await
.expect("connect to the configured postgres server");
tokio::spawn(async move {
let _ = connection.await;
});
STALE_SWEEP
.get_or_init(|| async {
// Collect databases previously failed runs unwound past. No
// `FORCE`: a database another live run still holds open refuses
// to drop, which is the outcome we want when two runs share a
// server.
if let Ok(stale) = admin
.query(
"SELECT datname FROM pg_database WHERE datname LIKE 'evstore_isolated_%'",
&[],
)
.await
{
for row in stale {
let name = row.get::<_, String>(0);
let _ = admin
.execute(&format!("DROP DATABASE IF EXISTS {name}"), &[])
.await;
}
}
})
.await;
// Identifiers cannot be bind parameters in DDL. The interpolations are
// process-generated (pid + counter) or come from `pg_database`, never
// caller input.
let name = format!(
"evstore_isolated_{}_{}",
std::process::id(),
NEXT_DATABASE.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
);
admin
.execute(&format!("CREATE DATABASE {name}"), &[])
.await
.expect("create the isolated database (the role needs CREATEDB)");
Some(IsolatedPostgresDatabase {
url: connection_string_with_dbname(&base_url, &name),
admin,
name,
})
}
/// Rewrites the database name inside a libpq connection string, preserving
/// every other component. `tokio_postgres::Config` parses both libpq forms
/// but cannot serialise back, and `build_reborn_event_stores` takes the raw
/// string, so the rewrite happens at the string level:
/// - URL form (`postgres://…`): replace the path segment, keep any query.
/// - Key-value form: append `dbname=…` — `tokio_postgres` applies keys in
/// order, so the appended one wins.
fn connection_string_with_dbname(base: &str, dbname: &str) -> String {
match base.find("://") {
Some(scheme_idx) => {
let after_scheme = scheme_idx + "://".len();
let (without_query, query) = match base[after_scheme..].find('?') {
Some(offset) => base.split_at(after_scheme + offset),
None => (base, ""),
};
let authority_end = without_query[after_scheme..]
.find('/')
.map(|offset| after_scheme + offset)
.unwrap_or(without_query.len());
format!("{}/{dbname}{query}", &without_query[..authority_end])
}
None => format!("{base} dbname={dbname}"),
}
}
#[tokio::test]
async fn postgres_replay_advances_next_cursor_past_trailing_filtered_records() {
let Ok(url) = std::env::var("IRONCLAW_REBORN_EVENT_STORE_POSTGRES_URL") else {
eprintln!(
"skipping postgres event-store cursor contract: IRONCLAW_REBORN_EVENT_STORE_POSTGRES_URL not set"
);
let Some(db) = isolated_postgres_database().await else {
return;
};
let suffix = std::time::SystemTime::now()
@@ -520,7 +656,7 @@ async fn postgres_replay_advances_next_cursor_past_trailing_filtered_records() {
let stores = build_reborn_event_stores(
RebornProfile::Production,
RebornEventStoreConfig::Postgres {
url: SecretString::new(url.into_boxed_str()),
url: SecretString::new(db.url.clone().into_boxed_str()),
tls_options: Default::default(),
},
)
@@ -559,13 +695,12 @@ async fn postgres_replay_advances_next_cursor_past_trailing_filtered_records() {
EventCursor::new(2),
"filtered trailing records must advance Postgres replay cursor"
);
drop(stores);
db.cleanup().await;
}
#[tokio::test]
async fn postgres_runtime_and_audit_logs_survive_rebuild_with_filtered_cursor_semantics() {
let Ok(url) = std::env::var("IRONCLAW_REBORN_EVENT_STORE_POSTGRES_URL") else {
eprintln!(
"skipping postgres event-store contract: IRONCLAW_REBORN_EVENT_STORE_POSTGRES_URL not set"
);
let Some(db) = isolated_postgres_database().await else {
return;
};
let suffix = std::time::SystemTime::now()
@@ -579,7 +714,7 @@ async fn postgres_runtime_and_audit_logs_survive_rebuild_with_filtered_cursor_se
let stores = build_reborn_event_stores(
RebornProfile::Production,
RebornEventStoreConfig::Postgres {
url: SecretString::new(url.clone().into_boxed_str()),
url: SecretString::new(db.url.clone().into_boxed_str()),
tls_options: Default::default(),
},
)
@@ -628,7 +763,7 @@ async fn postgres_runtime_and_audit_logs_survive_rebuild_with_filtered_cursor_se
let stores = build_reborn_event_stores(
RebornProfile::Production,
RebornEventStoreConfig::Postgres {
url: SecretString::new(url.into_boxed_str()),
url: SecretString::new(db.url.clone().into_boxed_str()),
tls_options: Default::default(),
},
)
@@ -661,6 +796,8 @@ async fn postgres_runtime_and_audit_logs_survive_rebuild_with_filtered_cursor_se
.status,
Some("project-a".to_string())
);
drop(stores);
db.cleanup().await;
}
#[tokio::test]

162
crates/product/ironclaw_assistant/tests/durable_ledger_contract.rs
View File

@@ -217,30 +217,38 @@ async fn postgres_duplicate_reservation_contention_serializes_when_configured()
}
#[tokio::test]
async fn postgres_settled_entry_limit_prunes_oldest_when_configured() {
let Some(filesystem) = postgres_filesystem().await else {
let Some(db) = isolated_postgres_filesystem().await else {
return;
};
let ledger =
RebornPostgresIdempotencyLedger::with_root_lease(filesystem, Duration::seconds(10))
.with_settled_entry_limit(NonZeroUsize::new(1).expect("non-zero limit"));
let ledger = RebornPostgresIdempotencyLedger::with_root_lease(
Arc::clone(&db.filesystem),
Duration::seconds(10),
)
.with_settled_entry_limit(NonZeroUsize::new(1).expect("non-zero limit"));
assert_settled_entry_limit_prunes_oldest(&ledger, &unique_suffix("postgres-retention")).await;
drop(ledger);
db.cleanup().await;
}
#[tokio::test]
async fn postgres_settled_prune_interval_defers_until_interval_when_configured() {
let Some(filesystem) = postgres_filesystem().await else {
let Some(db) = isolated_postgres_filesystem().await else {
return;
};
let ledger =
RebornPostgresIdempotencyLedger::with_root_lease(filesystem, Duration::seconds(10))
.with_settled_entry_limit(NonZeroUsize::new(1).expect("non-zero limit"))
.with_settled_prune_interval(NonZeroUsize::new(3).expect("non-zero interval"));
let ledger = RebornPostgresIdempotencyLedger::with_root_lease(
Arc::clone(&db.filesystem),
Duration::seconds(10),
)
.with_settled_entry_limit(NonZeroUsize::new(1).expect("non-zero limit"))
.with_settled_prune_interval(NonZeroUsize::new(3).expect("non-zero interval"));
assert_settled_prune_interval_defers_until_interval(
&ledger,
&unique_suffix("postgres-prune-interval"),
)
.await;
drop(ledger);
db.cleanup().await;
}
#[tokio::test]
async fn postgres_superseded_reservation_cannot_settle_when_configured() {
@@ -298,6 +306,142 @@ async fn postgres_actor_identity_is_part_of_fingerprint_path_when_configured() {
)
.await;
}
/// A private database for the settled-entry retention tests — the fabric
/// contract's `IsolatedDatabase` pattern
/// (`crates/substrates/ironclaw_filesystem/tests/db_root_filesystem_contract.rs`).
///
/// Unique fingerprint suffixes isolate every other test's rows, but the
/// settled-entry prune bookkeeping is global to the ledger root: it counts
/// and orders *all* settled entries under it, so sibling tests' entries
/// change which entry a limit of 1 prunes and when an interval of 3 fires —
/// and a limit-1 pruner running beside the other tests deletes *their*
/// settled rows in turn ("conflict row disappeared"). A name suffix cannot
/// isolate that; a private database can (the libsql twins get exactly that
/// from per-test temp files). The WS12 gauntlet report, §P8, measured the
/// defect; the two retention tests above are its regression pin.
struct IsolatedPostgresFilesystem {
filesystem: Arc<PostgresRootFilesystem>,
admin: tokio_postgres::Client,
name: String,
}
impl IsolatedPostgresFilesystem {
/// Drop the database on the way out of a passing test.
///
/// A courtesy, not a guarantee: a failing assertion unwinds straight
/// past it, so a red run can leave its database behind — that is what
/// the sweep in `isolated_postgres_filesystem` collects on the next
/// run. `FORCE` closes pool connections that have not gone away by the
/// time the handles drop.
async fn cleanup(self) {
let Self {
filesystem,
admin,
name,
} = self;
drop(filesystem);
let _ = admin
.execute(&format!("DROP DATABASE IF EXISTS {name} WITH (FORCE)"), &[])
.await;
}
}
/// One stale-database sweep per test binary, ahead of every creation.
/// Sweeping per provisioning call (as the fabric contract does) can race a
/// sibling test of the same binary between its `CREATE DATABASE` and first
/// connection; a single up-front sweep removes that window while still
/// collecting what failed runs left behind.
static STALE_SWEEP: tokio::sync::OnceCell<()> = tokio::sync::OnceCell::const_new();
/// Distinguishes the databases of concurrently provisioning tests.
static NEXT_DATABASE: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0);
async fn isolated_postgres_filesystem() -> Option<IsolatedPostgresFilesystem> {
let url = match std::env::var("IRONCLAW_PRODUCT_WORKFLOW_POSTGRES_URL") {
Ok(url) => url,
Err(_) => {
eprintln!(
"skipping postgres product workflow ledger contract: IRONCLAW_PRODUCT_WORKFLOW_POSTGRES_URL not set"
);
return None;
}
};
let config = match url.parse::<tokio_postgres::Config>() {
Ok(config) => config,
Err(error) => {
eprintln!("skipping postgres product workflow ledger contract: invalid url ({error})");
return None;
}
};
// Reachability keeps `postgres_filesystem`'s skip semantics. Past a
// reachable server, provisioning failures panic instead: this leg has no
// CI executor, and a silent skip would let the retention tests pass
// while testing nothing.
let (admin, connection) = match config.connect(tokio_postgres::NoTls).await {
Ok(connected) => connected,
Err(error) => {
eprintln!(
"skipping postgres product workflow ledger contract: database unavailable ({error})"
);
return None;
}
};
tokio::spawn(async move {
let _ = connection.await;
});
STALE_SWEEP
.get_or_init(|| async {
// Collect databases previously failed runs unwound past. No
// `FORCE`: a database another live run still holds open refuses
// to drop, which is the outcome we want when two runs share a
// server.
if let Ok(stale) = admin
.query(
"SELECT datname FROM pg_database WHERE datname LIKE 'pwledger_isolated_%'",
&[],
)
.await
{
for row in stale {
let name = row.get::<_, String>(0);
let _ = admin
.execute(&format!("DROP DATABASE IF EXISTS {name}"), &[])
.await;
}
}
})
.await;
// Identifiers cannot be bind parameters in DDL. The interpolations are
// process-generated (pid + counter) or come from `pg_database`, never
// caller input.
let name = format!(
"pwledger_isolated_{}_{}",
std::process::id(),
NEXT_DATABASE.fetch_add(1, std::sync::atomic::Ordering::Relaxed)
);
admin
.execute(&format!("CREATE DATABASE {name}"), &[])
.await
.expect("create the isolated database (the role needs CREATEDB)");
let mut isolated = config.clone();
isolated.dbname(&name);
let manager = deadpool_postgres::Manager::new(isolated, tokio_postgres::NoTls);
let pool = deadpool_postgres::Pool::builder(manager)
.max_size(4)
.build()
.expect("postgres pool builds against the isolated database");
let filesystem = Arc::new(PostgresRootFilesystem::new(pool));
filesystem
.run_migrations()
.await
.expect("migrate the isolated database");
Some(IsolatedPostgresFilesystem {
filesystem,
admin,
name,
})
}
async fn postgres_filesystem() -> Option<Arc<PostgresRootFilesystem>> {
let url = match std::env::var("IRONCLAW_PRODUCT_WORKFLOW_POSTGRES_URL") {
Ok(url) => url,