feat(stress): scripted tool-call workload with durable write read-back (#7360) (#7382)

* feat(stress): scripted tool-call workload with durable write read-back (#7360)

Phase 1 of issue #7360: teach the stress harness to drive real builtin
and memory tool calls through the production capability path and verify
their durable side effects.

The api-user-capacity mock LLM sidecar learns a deterministic scripted
state machine: the driver embeds an `ironclaw-stress-tool` marker in the
user message, the sidecar emits the scripted tool call for a tool
advertised in the request, the server executes it through the real
capability host, and the driver verifies the read-back verdict in the
final assistant message. Verdicts: confirmed / contended (same-user CAS
race, counted) / leak (cross-user isolation, hard failure) / missing
(write lost, hard failure) / undisclosed (tool never advertised).

Scripts: write_file_roundtrip (write_file + read_file of a unique
workspace path), memory_roundtrip / memory_grow / memory_mixed
(ironclaw.memory.write/replace-append + read of the shared
stress/shared.md target — every run doubles as a same-relative-path
isolation check). --api-scripted-doc-sizes cycles 4 KiB..1 MiB
documents with per-size buckets and submit-to-tool-visible /
submit-to-finalize stage latencies; --api-hot-writers spawns concurrent
same-user writers for hot-document CAS contention. Gated tools are
exercised through the per-user Tools auto-approve setting enabled during
setup via the production settings API.

Wired as a nightly leg in the hosted-single-tenant Postgres job (the
existing server stays up; the leg rebinds the mock sidecar on the same
port). Unit coverage: marker parsing, per-op step sequencing, tool-name
resolution (encoded/dotted/bare), verdict computation incl. leak
precedence, disclosure fallback, timeline helpers, per-size summary
buckets, and flag validation.

* fix(stress): hot writers on distinct user threads, size floor, CI server lifecycle

Review fixes for the #7360 Phase 1 scripted workload:

- Hot writers now run on distinct threads of the first user instead of
  sharing one thread, so concurrent operations exercise real per-user
  memory-document CAS contention rather than per-thread turn
  serialization. setup_users creates and records one extra thread per
  hot writer for user 0; run_hot_writer picks its own thread.
- Scripted document sizes are floored at 4 KiB (the token-dominated
  region below is meaningless and the issue's workloads start there);
  enforced in marker parsing and --api-scripted-doc-sizes validation.
- The CI scripted leg runs inside the server's run block so the trap
  does not kill the server before it starts; artifacts upload together.
- Wire-shape tests: mock_tool_call_response deserializes as the rig
  OpenAI CompletionResponse (stringified arguments, finish_reason
  tool_calls) and streaming tool-call chunks carry indexed delta
  tool_calls.

* fix(stress): hot-writer client action ids collide with the primary writer

A hot writer and the first user's regular writer shared the same user
label and operation index, so their client_action_id values were
identical and the server rejected the second submit with a 409
duplicate conflict. Include the scripted op prefix (h{k}-) in the
operation ref so concurrent writers always submit distinct action ids.

Found by a full local E2E run of the scripted leg against a real
hosted-single-tenant server: after the fix, memory_roundtrip with one
hot writer runs 9/9 clean (6 confirmed + 3 contended, 0 leaks) and
memory_grow runs 8/8 confirmed.

* fix(stress): address coderabbit review — verdict integrity, op-scoped tool counts, typed script key (#7382)

- compute_verdict: verdict comes from read steps only (write echoes can no
  longer mask missing/contended)
- timeline tool evidence: count tool results by sequence above the op's
  baseline instead of subtracting page-limited absolute counts
- timeline verdict match: delimit prefix by trailing space so op 1 cannot
  terminate on op 10's message; parse_result_verdict aligns on substring
- background users namespace markers as b{index} so cross-cohort leaks
  cannot read back as their own token
- hot writers drain in a dedicated JoinSet (no run_virtual_user refills)
- fraction chunks derive from cumulative boundaries so split writes persist
  exactly the configured size (regression test at 4097)
- ScriptKey derives clap::ValueEnum: CLI, marker wire format, and parsing
  share one string mapping; --api-hot-writers rejects write_file_roundtrip
- parse_marker bounds identity grammar; poisoned mutex recovery; sorted
  stage latencies; single conversation parse per completion request;
  CLI-level scripted validation test and doc-size bound coverage

* test(stress): cover --api-wait-for-assistant gate in CLI-level scripted test (#7382)

.github/workflows/ironclaw-stress.yml

@@ -377,6 +377,36 @@ jobs:
             > target/ironclaw-stress/postgres-api-capacity-summary.json \
             2> target/ironclaw-stress/postgres-api-capacity-report.txt
 
+          # Scripted tool-write leg: the first stress invocation exited,
+          # releasing the mock sidecar port; the server keeps running and
+          # this leg rebinds the sidecar at the same address the server's
+          # LLM base_url points at. Each operation is a scripted memory
+          # write/read-back through the real capability host (issue #7360
+          # Phase 1); the scripted summary reports read-back verdicts and
+          # stage latencies per document size. Runs inside this block so
+          # the server stays alive until both workloads complete.
+          target/release/ironclaw_stress \
+            --backend libsql \
+            --scenario api-user-capacity \
+            --api-base-url http://127.0.0.1:18080 \
+            --api-admin-bearer-token "$IRONCLAW_REBORN_WEBUI_TOKEN" \
+            --users 6 \
+            --concurrency 3 \
+            --operations 2 \
+            --api-scripted-tool memory_roundtrip \
+            --api-scripted-doc-sizes 4096,32768,131072,1048576 \
+            --api-hot-writers 2 \
+            --mock-llm-bind 127.0.0.1:19090 \
+            --mock-llm-latency-ms 250 \
+            --progress-interval-seconds 0 \
+            --human-read \
+            --bottleneck-report \
+            --output-jsonl target/ironclaw-stress/postgres-api-scripted.jsonl \
+            --max-failure-rate 0.05 \
+            --max-p95-ms 30000 \
+            > target/ironclaw-stress/postgres-api-scripted-summary.json \
+            2> target/ironclaw-stress/postgres-api-scripted-report.txt
+
       - name: Upload Postgres API capacity artifacts
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -387,4 +417,7 @@ jobs:
             target/ironclaw-stress/postgres-api-capacity.jsonl
             target/ironclaw-stress/postgres-api-capacity-summary.json
             target/ironclaw-stress/postgres-api-capacity-report.txt
+            target/ironclaw-stress/postgres-api-scripted.jsonl
+            target/ironclaw-stress/postgres-api-scripted-summary.json
+            target/ironclaw-stress/postgres-api-scripted-report.txt
           if-no-files-found: ignore

tools/ironclaw_stress/README.md

@@ -137,7 +137,6 @@ Use `--scenario` for a single workload.
 | `api-user-capacity` | End-to-end WebUI API send/read pressure against a running Reborn server. |
 | `cpu-burn` | Process-local CPU pressure control. |
 | `memory-churn` | Process-local allocation/RSS pressure control. |
-
 `api-user-capacity` can use either pre-minted users from `--api-users-jsonl`,
 a shared bearer via `--api-bearer-token`, or real per-user provisioning through
 the WebUI admin surface:
@@ -156,6 +155,62 @@ cargo run -p ironclaw_stress -- \
   --mock-llm-bind 127.0.0.1:3911
 ```
 
+### Scripted tool writes (issue #7360)
+
+Scripted mode drives each API operation through a real builtin/memory tool
+sequence: the driver embeds a marker in the user message, the mock LLM sidecar
+emits the scripted tool calls, the server executes them through the production
+capability host, and the driver verifies the read-back verdict in the final
+assistant message. Verdicts:
+
+- `confirmed`: the read-back returned exactly this operation's content marker.
+- `contended`: a concurrent same-user write won the document between write and
+  read (expected under `--api-hot-writers`, counted not failed).
+- `leak`: another user's content marker appeared in the read-back (cross-user
+  isolation violation, hard failure).
+- `missing`: the read-back lost the content (durable write failure, hard
+  failure).
+- `undisclosed`: the required tool was never advertised to the model
+  (disclosure/agent-surface regression, hard failure).
+
+Scripts:
+
+| `--api-scripted-tool` | Tool sequence |
+| --- | --- |
+| `write_file_roundtrip` | `builtin.write_file` then `builtin.read_file` of a unique workspace path. |
+| `memory_roundtrip` | `ironclaw.memory.write` (replace) then `ironclaw.memory.read` of `stress/shared.md`. |
+| `memory_grow` | Write a quarter, append three quarters, then read — growing-append slope. |
+| `memory_mixed` | Write half, read, append half, read — mixed read/write. |
+
+All memory scripts target the same relative path for every user, so each run
+also exercises same-relative-path isolation. `--api-scripted-doc-sizes` cycles
+document sizes per operation (default `4096,32768,131072,1048576`, minimum
+4096); results are bucketed per size with submit-to-tool-visible and
+submit-to-finalize stage latencies. `--api-hot-writers N` spawns N extra
+concurrent writers on distinct threads of the first user so they contend on
+the same per-user memory document without per-thread turn serialization.
+
+Scripted mode requires `--mock-llm-bind` (the sidecar serves the tool calls)
+and `--api-wait-for-assistant`. Gated tools (`builtin.write_file`, memory
+writes) are exercised through the per-user Tools auto-approve setting, which
+the driver enables during user setup through the same settings API a real
+user would use.
+
+```bash
+cargo run -p ironclaw_stress -- \
+  --backend libsql \
+  --scenario api-user-capacity \
+  --api-base-url http://127.0.0.1:3900 \
+  --api-admin-bearer-token "$IRONCLAW_REBORN_WEBUI_BEARER_TOKEN" \
+  --users 6 \
+  --concurrency 3 \
+  --operations 2 \
+  --api-scripted-tool memory_roundtrip \
+  --api-scripted-doc-sizes 4096,32768,131072,1048576 \
+  --api-hot-writers 2 \
+  --mock-llm-bind 127.0.0.1:3911
+```
+
 ## Presets
 
 Use `--preset` for a named single workload. Explicit CLI flags override preset

tools/ironclaw_stress/src/main.rs

@@ -12,6 +12,7 @@ mod ramp;
 mod redaction;
 mod report;
 mod resource_ops;
+mod scripted;
 mod secret_ops;
 mod suite;
 mod summary;
@@ -226,6 +227,26 @@ pub(crate) struct Args {
     #[arg(long, default_value_t = 16)]
     pub(crate) api_setup_concurrency: usize,
 
+    /// Scripted tool-call workload for api-user-capacity: each operation is
+    /// driven through a configured builtin/memory tool sequence and verified
+    /// by its read-back verdict (issue #7360 Phase 1).
+    #[arg(long, value_enum)]
+    pub(crate) api_scripted_tool: Option<scripted::ScriptKey>,
+
+    /// Document sizes in bytes to cycle through for scripted operations.
+    #[arg(
+        long,
+        value_delimiter = ',',
+        default_value = "4096,32768,131072,1048576"
+    )]
+    pub(crate) api_scripted_doc_sizes: Vec<usize>,
+
+    /// Concurrent scripted writers sharing the first user's thread so they
+    /// contend on the same durable document. 0 disables hot-writer
+    /// contention.
+    #[arg(long, default_value_t = 0)]
+    pub(crate) api_hot_writers: usize,
+
     /// Threads created per API user during setup. Values above 1 exercise
     /// listing/read paths against large sidebars; sends still target each
     /// user's first thread.
@@ -1109,6 +1130,41 @@ fn validate_args(args: &Args) -> Result<(), String> {
         );
     }
     api_capacity::validate_read_mix(&args.api_read_mix)?;
+    if let Some(key) = &args.api_scripted_tool {
+        if !args.scenario.is_api_capacity() {
+            return Err("--api-scripted-tool requires --scenario api-user-capacity".to_string());
+        }
+        if *key == scripted::ScriptKey::WriteFileRoundtrip && args.api_hot_writers > 0 {
+            return Err(
+                "--api-hot-writers requires a memory script (memory_roundtrip, memory_grow, \
+                 or memory_mixed); write_file_roundtrip targets a per-operation path and \
+                 cannot exercise shared-document contention"
+                    .to_string(),
+            );
+        }
+        if args.mock_llm_bind.is_none() {
+            return Err("--api-scripted-tool requires --mock-llm-bind".to_string());
+        }
+        if !args.api_wait_for_assistant {
+            return Err("--api-scripted-tool requires --api-wait-for-assistant".to_string());
+        }
+        if args.api_scripted_doc_sizes.is_empty() {
+            return Err("--api-scripted-doc-sizes must not be empty".to_string());
+        }
+        for size in &args.api_scripted_doc_sizes {
+            if *size < scripted::MIN_SCRIPTED_DOC_SIZE_BYTES
+                || *size > scripted::MAX_SCRIPTED_DOC_SIZE_BYTES
+            {
+                return Err(format!(
+                    "--api-scripted-doc-sizes values must be between {} and {} bytes",
+                    scripted::MIN_SCRIPTED_DOC_SIZE_BYTES,
+                    scripted::MAX_SCRIPTED_DOC_SIZE_BYTES
+                ));
+            }
+        }
+    } else if args.api_hot_writers > 0 {
+        return Err("--api-hot-writers requires --api-scripted-tool".to_string());
+    }
     if !(0.0..=1.0).contains(&args.mock_llm_failure_rate) {
         return Err("--mock-llm-failure-rate must be between 0.0 and 1.0".to_string());
     }

tools/ironclaw_stress/src/tests.rs

@@ -148,6 +148,158 @@ fn chat_turn_rejects_multi_process_runs() {
     assert!(error.contains("--scenario chat-turn requires --processes 1"));
 }
 
+fn scripted_api_args() -> Args {
+    let mut args = test_args();
+    args.scenario = Scenario::ApiUserCapacity;
+    args.api_base_url = Some("http://127.0.0.1:4216".to_string());
+    args.api_scripted_tool = Some(scripted::ScriptKey::MemoryRoundtrip);
+    args.mock_llm_bind = Some("127.0.0.1:19090".parse().expect("bind address parses"));
+    args
+}
+
+#[test]
+fn scripted_api_requires_api_scenario() {
+    let mut args = scripted_api_args();
+    args.scenario = Scenario::ChatTurn;
+
+    let error = validate_args(&args).expect_err("scripted mode needs the api scenario");
+
+    assert!(error.contains("--api-scripted-tool requires --scenario api-user-capacity"));
+}
+
+#[test]
+fn scripted_api_requires_mock_llm_bind() {
+    let mut args = scripted_api_args();
+    args.mock_llm_bind = None;
+
+    let error = validate_args(&args).expect_err("scripted mode needs the mock sidecar");
+
+    assert!(error.contains("--api-scripted-tool requires --mock-llm-bind"));
+}
+
+#[test]
+fn scripted_api_requires_wait_for_assistant() {
+    let mut args = scripted_api_args();
+    args.api_wait_for_assistant = false;
+
+    let error = validate_args(&args).expect_err("scripted mode needs verdict polling");
+
+    assert!(error.contains("--api-scripted-tool requires --api-wait-for-assistant"));
+}
+
+#[test]
+fn scripted_api_rejects_out_of_range_doc_sizes() {
+    let mut args = scripted_api_args();
+
+    // Below the floor: 1024 bytes cannot survive the /4 memory-grow split
+    // and stay above the read-back token, so the recorded size bucket would
+    // not describe the bytes actually written.
+    args.api_scripted_doc_sizes = vec![1024];
+    let error = validate_args(&args).expect_err("sub-4KiB documents are meaningless");
+    assert!(error.contains("--api-scripted-doc-sizes values must be between 4096 and"));
+
+    // Above the ceiling: stops an operator from requesting a multi-gigabyte
+    // durable write against the hosted Postgres target.
+    args.api_scripted_doc_sizes = vec![scripted::MAX_SCRIPTED_DOC_SIZE_BYTES + 1];
+    let error = validate_args(&args).expect_err("oversized documents are rejected");
+    assert!(error.contains("--api-scripted-doc-sizes values must be between"));
+
+    args.api_scripted_doc_sizes = Vec::new();
+    let error = validate_args(&args).expect_err("an empty size list is rejected");
+    assert!(error.contains("--api-scripted-doc-sizes must not be empty"));
+}
+
+#[test]
+fn scripted_api_hot_writers_reject_file_roundtrip_script() {
+    let mut args = scripted_api_args();
+    args.api_scripted_tool = Some(scripted::ScriptKey::WriteFileRoundtrip);
+    args.api_hot_writers = 2;
+
+    let error = validate_args(&args)
+        .expect_err("hot writers need a shared document; write_file_roundtrip uses per-op paths");
+
+    assert!(error.contains("--api-hot-writers requires a memory script"));
+}
+
+#[test]
+fn scripted_cli_flag_combination_parses_and_gates_execution() {
+    // Drive the scripted flag combination through the CLI parser (clap
+    // typing and default resolution) and then through the validation gate
+    // that runs before any workload starts.
+    let args = Args::try_parse_from([
+        "ironclaw_stress",
+        "--backend",
+        "libsql",
+        "--scenario",
+        "api-user-capacity",
+        "--api-base-url",
+        "http://127.0.0.1:4216",
+        "--api-scripted-tool",
+        "memory_roundtrip",
+        "--mock-llm-bind",
+        "127.0.0.1:19090",
+    ])
+    .expect("scripted flag combination parses");
+    assert_eq!(
+        args.api_scripted_tool,
+        Some(scripted::ScriptKey::MemoryRoundtrip)
+    );
+    assert_eq!(
+        args.api_scripted_doc_sizes,
+        vec![4096, 32768, 131072, 1048576]
+    );
+    validate_args(&args).expect("valid scripted configuration passes the startup gate");
+
+    // An unknown script key is rejected by clap itself before validation.
+    let parse_error = Args::try_parse_from([
+        "ironclaw_stress",
+        "--backend",
+        "libsql",
+        "--scenario",
+        "api-user-capacity",
+        "--api-base-url",
+        "http://127.0.0.1:4216",
+        "--api-scripted-tool",
+        "not_a_script",
+        "--mock-llm-bind",
+        "127.0.0.1:19090",
+    ])
+    .expect_err("unknown script key is rejected at parse time");
+    assert!(parse_error.to_string().contains("possible values"));
+
+    // A valid key without the required sidecar is rejected by validation
+    // before any workload starts.
+    let mut missing_sidecar = args;
+    missing_sidecar.mock_llm_bind = None;
+    let error = validate_args(&missing_sidecar).expect_err("sidecar is required");
+    assert!(error.contains("--api-scripted-tool requires --mock-llm-bind"));
+
+    // Scripted verdicts are read from the timeline, so assistant polling is
+    // required for the flow to complete.
+    let mut no_polling = scripted_api_args();
+    no_polling.api_wait_for_assistant = false;
+    let error = validate_args(&no_polling).expect_err("assistant polling is required");
+    assert!(error.contains("--api-scripted-tool requires --api-wait-for-assistant"));
+}
+
+#[test]
+fn scripted_api_hot_writers_require_script_key() {
+    let mut args = test_args();
+    args.scenario = Scenario::ApiUserCapacity;
+    args.api_base_url = Some("http://127.0.0.1:4216".to_string());
+    args.api_hot_writers = 2;
+
+    let error = validate_args(&args).expect_err("hot writers need a script");
+
+    assert!(error.contains("--api-hot-writers requires --api-scripted-tool"));
+}
+
+#[test]
+fn scripted_api_accepts_valid_configuration() {
+    let args = scripted_api_args();
+    validate_args(&args).expect("valid scripted api configuration should pass validation");
+}
+
 #[test]
 fn prewarm_dispatches_secret_and_process_local_scenarios() {
     assert_eq!(
@@ -1370,6 +1522,9 @@ fn test_args() -> Args {
         api_poll_interval_ms: 250,
         api_request_timeout_ms: 10_000,
         api_setup_concurrency: 16,
+        api_scripted_tool: None,
+        api_scripted_doc_sizes: vec![4096, 32768, 131072, 1048576],
+        api_hot_writers: 0,
         api_threads_per_user: 1,
         thread_list_untitled: false,
         api_background_users: 0,