6f1ae709d5
* test(tool-search): add large-catalog baseline * feat(tool-search): return and use bounded signatures * feat(tool-search): add fair discovery benchmark arms * test(tool-discovery): add live benchmark harness * feat(tool-discovery): default to namespace summaries * ci(tool-discovery): classify benchmark harness * feat(tool-discovery): use semantic namespaces * fix(tool-search): harden discovery benchmark and mode wiring * docs(tool-search): record corrected benchmark verdict * fix(tool-search): address review feedback
Real-model tool-discovery benchmark
This runner exercises the shipping ironclaw serve binary and configured live
model against deterministic 100-, 500-, and 1,000-tool catalogs. Twenty local
MCP integrations use stable semantic identities such as github, gmail, and
google-calendar; distractors are distributed as evenly as the fixed relevance
corpus permits. They provide read-only synthetic side effects and record the
exact tools and arguments invoked. IronClaw still performs normal authorization,
approval, hooks, safety, and MCP dispatch.
cargo build -p ironclaw
# Requires NEARAI_API_KEY or LIVE_OPENAI_COMPATIBLE_API_KEY.
export NEARAI_API_KEY=...
python3 scripts/tool_discovery_benchmark/run_benchmark.py \
--output-dir /tmp/ironclaw-tool-discovery-benchmark
The default matrix runs all five disclosure arms, all three catalog sizes, all
required scenario classes, and four repetitions (one cold plus three warm).
Use repeated --arm, --tool-count, or --task flags only for diagnosis.
Each observation is appended and synced as soon as it completes, and an interrupted run resumes by stable observation id. Scoring checks required call order and arguments, detects forbidden attempts in model traces, and measures latency to the first correct tool rather than the first tool of any kind.
The output contains per-observation JSONL, aggregate JSON, model traces, browser diagnostics, and server logs. Provider token/cache fields are retained only when the provider reports them. Zero or unavailable usage must not be replaced with estimates derived from JSON bytes.