lywsvip/ironclaw
1
0
Fork 0
mirror of https://github.com/nearai/ironclaw.git synced 2026-09-03 08:06:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

refactor(rules): split agent-evidence into prompt + code rule

Browse Source 
agent-evidence.md mixed two concerns: runtime agent instruction (what
the LLM should do when concluding a turn) and code-enforcement rules
(what the dispatcher, engine, and tools must implement). Rules under
.claude/rules/ only guide Claude Code when editing the repo — the
runtime agent never reads them.

Splits the two:

- crates/ironclaw_engine/prompts/codeact_postamble.md — new section
  "Evidence before claiming side effects". Sits next to the existing
  "FINAL() answer quality" guidance; loaded via include_str! in
  executor/prompt.rs (no Rust change needed).
- .claude/rules/tool-evidence.md — renamed from agent-evidence.md,
  keeps only the code invariants (engine v2 side-effect gate,
  empty-fast ToolError::EmptyResult, external-effect tools must read
  back, setup UI round-trip).

Prompt tests pass unchanged; the postamble addition is pure text.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Illia Polosukhin
2026-04-20 06:21:28 +00:00
parent 9bc9edb440
commit 9d7b43a9f3
3 changed files with 62 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
.claude/rules/agent-evidence.md
View File

@@ -1,39 +0,0 @@
---
paths:
- "src/agent/**"
- "src/tools/**"
- "crates/ironclaw_engine/**"
---
# Agent Claims and Evidence
The most dangerous user-visible bug class in IronClaw is **claim/evidence drift**: the agent's final text says "message sent" / "file attached" / "tool installed" with no corresponding side effect. LLM output is narration, not proof.
## Side-Effect Claims Must Cite Tool Evidence
When the user request implies a side effect (send, save, install, schedule, post, write, delete), the agent's final turn must either:
1. Reference a completed tool call whose `ToolOutput` contains a provider-issued identifier (`message_id`, `bytes_written`, `external_id`, `job_id`, `created_at`), **or**
2. Explicitly state that the action was not performed and why.
Engine v2 detects side-effect intent on the user turn and must reject a model-final turn that has no matching successful tool call. Plain-text "I've sent your message" without a tool call is a failure mode to flag, not behavior to trust. References: #2544, #2580, #2582, #2541, #2447.
## Empty-Fast Tool Outputs Are Errors
A tool that completes in `< 1ms` *and* returns empty content is almost always a silent failure. The dispatcher treats `duration < 1ms && content.is_empty()` as `ToolError::EmptyResult` unless the tool explicitly opts in via a sentinel (idempotent ack, documented no-op confirmation).
`ActionRecord` must always capture byte count and timing so this check is auditable at review time. The UI must suppress the success checkmark when output bytes are zero. Reference: #2545.
## External-Effect Tools Must Read Back
Tools whose side effect is visible only to an external system (Telegram send, Slack post, file write, extension install, OAuth completion) MUST read back the effect before returning success:
- `telegram_send` → capture and return `message_id` from the API response; error if the response lacks one.
- `file_write` → re-stat and return the actual byte count; error on mismatch.
- `extension_install` → call `extensions_list` and assert the new extension is present and active.
- OAuth completion → perform a minimal authenticated read against the provider before declaring success.
A tool without a read-back path is claim-only and must mark its output `unverified: true` so downstream layers can warn. References: #2411 Telegram token Save, #2543 Linear MCP OAuth, #2586 Slack Install.
## Setup UI Actions Round-Trip
Save / Install / Connect buttons in the setup UI must issue a read-back verification immediately after the write succeeds and render the read-back value (or explicit error) to the user — not a local optimistic checkmark. Install actions must dispatch through `ToolDispatcher::dispatch` and surface the resulting `ActionRecord`. A UI success state with no corresponding backend read-back is the same bug class as agent claim drift. References: #2411, #2534, #2543, #2586.

35
.claude/rules/tool-evidence.md Normal file
View File

@@ -0,0 +1,35 @@
---
paths:
- "src/agent/**"
- "src/tools/**"
- "src/channels/web/**"
- "crates/ironclaw_engine/**"
---
# Tool Evidence and Side-Effect Verification
The most dangerous user-visible bug class is **claim/evidence drift**: the agent narrates "message sent" / "file attached" / "tool installed" with no corresponding side effect. The agent-facing half of this rule lives in `crates/ironclaw_engine/prompts/codeact_postamble.md` ("Evidence before claiming side effects"). The code invariants that make the rule enforceable live here.
## Engine v2 Side-Effect Gate
Engine v2 classifies user turns for side-effect intent (send / save / install / schedule / post / write / delete). A model-final turn that lacks at least one successful tool call matching the intent must be rejected before it reaches the user — surface "action not performed" instead of the agent's narration. Reference: #2544, #2580, #2582, #2541, #2447.
## Empty-Fast Outputs Are Errors
A tool that completes in `< 1 ms` **and** returns empty content is almost always a silent failure. The dispatcher treats `duration < 1ms && content.is_empty()` as `ToolError::EmptyResult` unless the tool opts in via a documented no-op sentinel (idempotent ack).
`ActionRecord` must always capture byte count and timing so this check is auditable at review time. The UI must suppress the success checkmark when output bytes are zero. Reference: #2545.
## External-Effect Tools Must Read Back
A tool whose side effect is visible only to an external system (Telegram send, Slack post, file write, extension install, OAuth completion) MUST read back the effect before returning success:
- `telegram_send` → capture and return `message_id` from the API response; error if the response lacks one.
- `file_write` → re-stat and return the actual byte count; error on mismatch.
- `extension_install` → call `extensions_list` and assert the new extension is present and active.
- OAuth completion → perform a minimal authenticated read against the provider before declaring success.
A tool without a read-back path is claim-only and must mark its output `unverified: true` so downstream layers can warn. References: #2411 Telegram token Save, #2543 Linear MCP OAuth, #2586 Slack Install.
## Setup UI Round-Trip
Save / Install / Connect buttons in the setup UI must issue a read-back verification immediately after the write succeeds and render the read-back value (or explicit error) to the user — not a local optimistic checkmark. Install actions dispatch through `ToolDispatcher::dispatch` and surface the resulting `ActionRecord`. A UI success state with no corresponding backend read-back is the same bug class as agent claim drift. References: #2411, #2534, #2543, #2586.

27
crates/ironclaw_engine/prompts/codeact_postamble.md
View File

@@ -76,3 +76,30 @@ proposals with their APY, gain, and cost — not a count. Build up the answer
string with real data from tool results (`proposal["rationale"]`,
`proposal["projected_annual_gain_usd"]`, etc.), then call `FINAL()` once
with the complete Markdown.
## Evidence before claiming side effects
If the user asked for a side effect — send, save, install, schedule, post,
write, delete — your `FINAL()` answer must either:
1. Cite the specific result of a successful tool call (the `message_id`,
`bytes_written`, `external_id`, `job_id`, `created_at` the tool
returned), or
2. Say plainly that the action was not performed, and why.
Never narrate "I've sent your message" / "I attached the file" / "I
installed the tool" without the corresponding tool result in the same
response. Your text is narration, not proof. A user who reads "I sent
it" while nothing arrived is the worst failure mode this system has.
If a tool call returned empty output or completed in under a millisecond,
treat that as failure — the side effect did not happen. Retry or report
the failure, do not claim success.
```repl
result = await telegram_send(chat_id=chat, text=body)
if not result or not result.get("message_id"):
FINAL(f"Tried to send, but Telegram didn't confirm delivery: {result}")
else:
FINAL(f"Sent (message_id={result['message_id']}).")
```