mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-03 08:06:01 +08:00
30ae2d50f6
* feat(stress): scripted tool-call workload with durable write read-back (#7360) Phase 1 of issue #7360: teach the stress harness to drive real builtin and memory tool calls through the production capability path and verify their durable side effects. The api-user-capacity mock LLM sidecar learns a deterministic scripted state machine: the driver embeds an `ironclaw-stress-tool` marker in the user message, the sidecar emits the scripted tool call for a tool advertised in the request, the server executes it through the real capability host, and the driver verifies the read-back verdict in the final assistant message. Verdicts: confirmed / contended (same-user CAS race, counted) / leak (cross-user isolation, hard failure) / missing (write lost, hard failure) / undisclosed (tool never advertised). Scripts: write_file_roundtrip (write_file + read_file of a unique workspace path), memory_roundtrip / memory_grow / memory_mixed (ironclaw.memory.write/replace-append + read of the shared stress/shared.md target — every run doubles as a same-relative-path isolation check). --api-scripted-doc-sizes cycles 4 KiB..1 MiB documents with per-size buckets and submit-to-tool-visible / submit-to-finalize stage latencies; --api-hot-writers spawns concurrent same-user writers for hot-document CAS contention. Gated tools are exercised through the per-user Tools auto-approve setting enabled during setup via the production settings API. Wired as a nightly leg in the hosted-single-tenant Postgres job (the existing server stays up; the leg rebinds the mock sidecar on the same port). Unit coverage: marker parsing, per-op step sequencing, tool-name resolution (encoded/dotted/bare), verdict computation incl. leak precedence, disclosure fallback, timeline helpers, per-size summary buckets, and flag validation. * fix(stress): hot writers on distinct user threads, size floor, CI server lifecycle Review fixes for the #7360 Phase 1 scripted workload: - Hot writers now run on distinct threads of the first user instead of sharing one thread, so concurrent operations exercise real per-user memory-document CAS contention rather than per-thread turn serialization. setup_users creates and records one extra thread per hot writer for user 0; run_hot_writer picks its own thread. - Scripted document sizes are floored at 4 KiB (the token-dominated region below is meaningless and the issue's workloads start there); enforced in marker parsing and --api-scripted-doc-sizes validation. - The CI scripted leg runs inside the server's run block so the trap does not kill the server before it starts; artifacts upload together. - Wire-shape tests: mock_tool_call_response deserializes as the rig OpenAI CompletionResponse (stringified arguments, finish_reason tool_calls) and streaming tool-call chunks carry indexed delta tool_calls. * fix(stress): hot-writer client action ids collide with the primary writer A hot writer and the first user's regular writer shared the same user label and operation index, so their client_action_id values were identical and the server rejected the second submit with a 409 duplicate conflict. Include the scripted op prefix (h{k}-) in the operation ref so concurrent writers always submit distinct action ids. Found by a full local E2E run of the scripted leg against a real hosted-single-tenant server: after the fix, memory_roundtrip with one hot writer runs 9/9 clean (6 confirmed + 3 contended, 0 leaks) and memory_grow runs 8/8 confirmed. * fix(stress): address coderabbit review — verdict integrity, op-scoped tool counts, typed script key (#7382) - compute_verdict: verdict comes from read steps only (write echoes can no longer mask missing/contended) - timeline tool evidence: count tool results by sequence above the op's baseline instead of subtracting page-limited absolute counts - timeline verdict match: delimit prefix by trailing space so op 1 cannot terminate on op 10's message; parse_result_verdict aligns on substring - background users namespace markers as b{index} so cross-cohort leaks cannot read back as their own token - hot writers drain in a dedicated JoinSet (no run_virtual_user refills) - fraction chunks derive from cumulative boundaries so split writes persist exactly the configured size (regression test at 4097) - ScriptKey derives clap::ValueEnum: CLI, marker wire format, and parsing share one string mapping; --api-hot-writers rejects write_file_roundtrip - parse_marker bounds identity grammar; poisoned mutex recovery; sorted stage latencies; single conversation parse per completion request; CLI-level scripted validation test and doc-size bound coverage * test(stress): cover --api-wait-for-assistant gate in CLI-level scripted test (#7382)