mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-02 23:56:24 +08:00
e3df3ec4ae
* chore: gitignore live test fixture containing recorded credentials
The github_dev_workflow live test records HTTP exchanges including
the github_token Bearer header. GitHub push protection correctly
blocks this. The fixture is only useful locally for replay; the
test skips gracefully without it.
* test: add live test for github developer workflow
Adds tests/e2e_github_dev_workflow.rs — a multi-turn live/replay test that
drives the developer-assistant + github-workflow skills end-to-end against
a synthetic nearai/ironclaw repository:
1. Setup — installs the wf-* mission set (excluding
wf-staging-review per the implement-but-don't-
auto-merge autonomy contract)
2. Issue opened — synthetic github.issue.opened webhook payload
3. Maintainer LGTM — pr.comment.created from a maintainer
4. PR review — non-maintainer review comment
5. CI failure — failing check_run
6. Approval — maintainer approval; asserts NO merge call ever
fires across the whole session
7. Digest — status report referencing the issue/PR
Webhook payloads are injected via TestRig::send_message with a
[GITHUB WEBHOOK] frame that matches what a real webhook→channel
adapter would emit. The mission OnSystemEvent firing path is covered
separately by mission.rs unit tests; this test exercises skill
behavior given the right inputs.
Adds two helpers to tests/support/live_harness.rs:
- trace_contains_tool_call(name, needle)
- assert_trace_contains_tool_call(name, needle, ctx)
Both scan ToolStarted.detail and ToolResult.preview for case-insensitive
substring matches, so behavior tests can assert *what the agent
actually called* without scraping the recorded trace JSON.
Drive-by cleanups from the extension-lifecycle merge:
- thread_ops.rs: drop orphaned RecordingStatusChannel + helper that
came from a dropped extension-lifecycle test variant
- bridge/router.rs: clippy needless_borrow on PendingGate args
- skills/mod.rs: SkillManifest no longer has metadata field; add
requires: GatingRequirements::default() to test fixture
- cargo fmt fallout in recording.rs / live_mission.rs / trace_llm.rs
The test is #[ignore]-tagged (live tier) and skips gracefully in replay
mode until tests/fixtures/llm_traces/live/github_dev_workflow_full_loop.json
is recorded with IRONCLAW_LIVE_TEST=1. Compile coverage is automatic
via the existing test matrix; live execution follows the same pattern
as e2e_live_personas.rs (manual recording + commit fixture).
cargo check --features libsql --tests: clean
cargo clippy --features libsql --tests --test e2e_github_dev_workflow -- -D warnings: clean
cargo test --features libsql --test e2e_github_dev_workflow -- --ignored: passes (skips, fixture missing)
* test(harness): add pre-seed secrets + diagnostic activity dump
Three additions to make the github_dev_workflow live test runnable:
1. **TestRigBuilder::with_secret(name, value)** — pre-seed credentials
in the SecretsStore before the agent starts. The kernel pre-flight
auth gate fires when a skill with a credential spec activates (e.g.
the github skill needs github_token); without a stored credential
the agent gets stuck in 'Authentication required' mode and can't
make progress. Tests inject a fake/dummy value so the gate is
satisfied — the test isn't actually hitting the credentialed API.
Implementation: AppComponents.secrets_store is captured during
build_all() and any pre-seeded (name, value) pairs are written via
secrets_store.create() with user_id = config.owner_id. Already-exists
errors are silenced so the helper is idempotent on seeded DBs.
2. **LiveTestHarnessBuilder::with_secret** — forwards to
TestRigBuilder::with_secret. Plumbed through both build_live and
build_replay so the same fixture works in both modes.
3. **dump_activity helper in e2e_github_dev_workflow.rs** — formats
captured StatusUpdate stream (skill activations + every tool
started/completed/result) to stderr. Used as a pre-assertion
diagnostic so failing live runs surface the agent's actual tool
sequence instead of an opaque panic on a workspace check.
Test relaxations from running this against the real LLM:
- verify_setup_landed accepts either developer-assistant OR
github-workflow as the active skill (the deterministic selector
picks based on keyword scoring + token budget; both routes are
valid since github-workflow owns the mission templates)
- final required-skills check drops developer-assistant in favor of
github-workflow + github (the orchestrator persona is optional)
- setup turn now pre-seeds github_token via with_secret
cargo check --features libsql --tests: clean
* test: rewrite github_dev_workflow as fully real live integration
Pivots the test from synthetic webhook simulation to a real end-to-end
integration test against the real nearai/ironclaw repo. Per project
owner: 'fully real live tests doing useful work on github repo... test
everything like it's live while recording all interactions to debug
what doesn't work and improve that'.
## Why the rewrite
The previous synthetic-event version injected fake GitHub payloads as
channel messages. With a real github_token in scope, the agent
attempted to fetch the fake issue 99001, got a 404, and helpfully
created 3 real issues + 3 real comments on nearai/ironclaw to
"reconcile" the discrepancy. The synthetic approach didn't surface
realistic failure modes anyway (auth gates, payload format mismatches,
rate limits), so we go all-in on real artifacts.
## New flow (2 turns + real artifact lifecycle)
1. Setup turn — agent installs the wf-* mission set for nearai/ironclaw
2. Test (NOT the agent) creates a real issue via direct REST API with
the title "[live-test {timestamp}] Add /metrics Prometheus endpoint"
and a real feature-request body.
3. Triage turn — test asks agent to triage issue #N. Agent reads via
github skill, generates a plan, posts a real comment back.
4. Verification — test polls api.github.com/issues/N/comments and
asserts at least one new comment exists since baseline. Comment
bodies are logged to stderr for human review (the most useful
debug output for iterating on skill quality).
5. Cleanup — std::panic::catch_unwind wraps the body so cleanup runs
regardless of pass/fail. Closes the issue with a final "live test
complete" comment. If cleanup itself fails, the issue URL is
printed for manual recovery.
## Test infrastructure additions
- TestRig.get_secret(name) — read decrypted secrets back from the
rig's SecretsStore. Required so the test can read the github_token
the harness pre-seeded via with_secrets(["github_token"]).
- TestRig captures secrets_store + owner_id from AppComponents during
build (needed for get_secret).
- github_api submodule inside the test file — direct REST helpers for
create_issue, list_issue_comments, post_issue_comment, close_issue.
Uses reqwest directly so the test has guaranteed GitHub access
regardless of skill selection / tool gating.
## Recording
- LLM trace fixture: tests/fixtures/llm_traces/live/github_dev_workflow_full_loop.json (65K)
- Session log: github_dev_workflow_full_loop.log (5.9K)
- Both committed so future runs can replay deterministically without
hitting real GitHub.
## What's NOT covered yet
Dropped from the previous version (can be added back as follow-ups):
- PR creation flow (agent opens a real PR with a real branch + real
code change)
- CI failure simulation (would need a real failing CI run)
- Mission OnSystemEvent firing via real webhooks (needs an HTTP
server registered as a GitHub webhook)
- Maintainer approval flow
This first version validates the most valuable slice: setup → react
to real issue → produce real comment → cleanup. If the agent's
comment quality is good, we expand from here.
cargo check --features libsql --tests: clean
cargo clippy --features libsql --tests --test e2e_github_dev_workflow -- -D warnings: clean
Live recording: passed in 85.9s
- Created issue #2185
- Agent posted 2 comments (full plan + follow-up)
- Closed issue #2185
* feat(skills): one-time setup-marker exclusion + rename persona skills to *-setup
The persona orchestrator skills (developer-assistant, ceo-assistant,
trader-assistant, content-creator-assistant) are pure first-time
onboarding flows — their entire body is Steps 1-N of workspace setup,
mission registration, and calibration memory writes. After those steps
run successfully, there is nothing left for the skill to do, but the
deterministic selector kept evaluating them on every conversation
turn, burning ~3000 tokens of activation budget for work already
completed and risking partial re-runs of setup steps.
This commit makes setup skills opt-in to one-time activation:
## Mechanism: setup_marker exclusion
New optional field on ActivationCriteria:
activation:
setup_marker: commitments/.developer-setup-complete
Before scoring, the selector caller (Agent::select_active_skills)
collects every distinct setup_marker referenced by loaded skills,
checks the workspace for each via Workspace::exists(), and passes
the set of satisfied markers into prefilter_skills. Any skill whose
marker is in the satisfied set is excluded from scoring entirely
(returns None from the filter map, skipping the score_skill call).
The selector check is opt-in: skills without a setup_marker are
unaffected. Reactive operational skills (commitment-triage,
decision-capture, github, github-workflow, etc.) keep activating
on every matching message as before.
Tests:
- 4 unit tests in crates/ironclaw_skills/src/selector.rs covering
marker present/absent, marker mismatch, and skill-without-marker
unaffected paths
- All 152 ironclaw_skills tests pass
- Live e2e_github_dev_workflow run on real nearai/ironclaw passes
(issue #2186 created, comment posted, closed) in 88s
## Rename: *-assistant → *-setup
Per project owner: 'rename persona skills to -setup skills to make
it explicit they are called once'. The -assistant suffix obscured
the lifecycle — these are not always-on assistants, they are
one-time onboarding wizards.
Renamed directories (via git mv) and updated SKILL.md `name:`
fields:
- skills/ceo-assistant → skills/ceo-setup
- skills/content-creator-assistant → skills/content-creator-setup
- skills/developer-assistant → skills/developer-setup
- skills/trader-assistant → skills/trader-setup
All four now declare `setup_marker: commitments/.<name>-setup-complete`
and have a new final 'Step N: Mark setup complete' instructing the
agent to write the marker via memory_write after confirming setup
with the user. Different personas have different markers so they
remain independently triggerable in separate workspaces.
Cross-references updated:
- tests/e2e_live_personas.rs (4 persona test invocations)
- tests/e2e_github_dev_workflow.rs (doc comments)
- tests/e2e/LIVE_TOOL_FAILURES.md (1 reference)
- crates/ironclaw_skills/src/types.rs (doc comment example)
## Bump: SKILLS_MAX_CONTEXT_TOKENS default 4000 → 6000
The previous default was so tight that a setup skill (3000 tokens)
plus its companion github-workflow (2000) plus github (2000) would
overflow at 7000. Reactive operational skills like
commitment-triage, decision-capture, tech-debt-tracker often got
budget-evicted. With setup skills now excluded after onboarding,
the freed budget plus the bump to 6000 lets the most useful
combinations fit comfortably (e.g. github-workflow + github +
product-prioritization is now active in the live recording, where
previously product-prioritization would have been evicted).
## Plumbing changes
- ActivationCriteria gains pub setup_marker: Option<String>
(#[serde(default)], so existing skills are unaffected)
- prefilter_skills signature gains
&satisfied_setup_markers: &HashSet<String> (caller passes empty
set to disable filtering — used by all existing tests via the
prefilter_no_markers wrapper)
- Agent::select_active_skills is now async — it needs to
Workspace::exists() each marker. dispatcher.rs caller updated
to .await. Snapshots the skill list under the read lock then
drops the guard before any await to avoid holding a poisonable
RwLock across an await point.
cargo check --features libsql --tests: clean
cargo clippy --features libsql --tests --all-targets -- -D warnings: clean
cargo test -p ironclaw_skills: 152 passed
Live e2e_github_dev_workflow run: passes (88s)
* feat(skills): chain-load companions + v2 marker exclusion + commitment-setup marker
Three orthogonal follow-ups to the skill lifecycle work.
## 1. Chain-loading via requires.skills (v1 Rust + v2 Python)
When a parent skill is selected by the scorer, its requires.skills
companions are now automatically loaded, bypassing the score filter.
Persona/bundle skills like developer-setup can finally work as
designed: the orchestrator declares which operational skills it
delegates to, and selecting the orchestrator pulls them all in.
- **v1 Rust** (crates/ironclaw_skills/src/selector.rs): extracted
skill_token_cost() and try_select() helpers used by both the
scored-selection loop and the new chain-loading pass. Companions
consume the same budget and respect max_candidates. Non-transitive
(depth 1 only) to keep behavior predictable.
- **v2 Python** (crates/ironclaw_engine/orchestrator/default.py):
select_skills() gains an inline chain-loading pass that mirrors
the Rust logic. Uses a name-indexed lookup built from the skill
list passed in by handle_list_skills. No closure-over-outer-var
tricks that Monty would reject — the inner try-add is inlined.
7 chain-load unit tests in selector.rs covering: pulls in
companions, skipped when parent not selected, respects budget,
skips companion with satisfied marker, non-transitive (depth 2
not pulled), missing companion silent, dedup across parents.
## 2. v2 setup_marker exclusion
The v2 engine's Python orchestrator handles skill selection via
handle_list_skills (Rust) -> select_skills (Python). Since
handle_list_skills already has the full project doc list in scope,
we filter there: any skill whose metadata.activation.setup_marker
is in the set of existing doc titles gets excluded before the
Python orchestrator ever sees it. Zero extra store calls — we
reuse the existing list_memory_docs_with_shared result to build
an O(1) title set.
This is the v2 parity of the v1 satisfied_setup_markers parameter
threaded through prefilter_skills. Both paths now implement the
same rule: a one-time setup skill whose marker file has been
written has finished its job and should not keep burning
activation budget.
## 3. commitment-setup gets a setup_marker
commitment-setup writes commitments/README.md as its first step,
so the marker is automatically set after a successful first run.
Added:
activation:
setup_marker: commitments/README.md
To re-trigger (e.g. migrate to a new schema), delete README.md
first. project-setup was NOT given a marker — it's per-repo,
invoked repeatedly, not a singleton (each call creates a new
projects/<owner>-<repo>/project.md).
## 4. Lifecycle integration test
tests/skill_setup_marker_lifecycle.rs drives a real agent turn
through the v1 selector pipeline (Agent::select_active_skills ->
Workspace::exists -> prefilter_skills) to verify that a setup
skill:
Phase 1: activates on the first matching message (marker absent)
Phase 2: marker file is written via workspace.write()
Phase 3: is excluded on the second matching message
The test asserts on the captured LLM system prompt content (via
rig.captured_llm_requests) rather than on StatusUpdate events so
it's agnostic to v1/v2 path differences in how skill activations
are announced. The skill's body contains a distinctive marker
string (LIFECYCLE-TEST-SKILL-BODY-MARKER-Z7Q) — if the skill was
selected, that string appears in the system prompt; if excluded,
it doesn't.
Cover matrix after this commit:
- v1 selector: 35 unit tests + 4 setup-marker tests + 7 chain-load tests
- v2 handle_list_skills marker exclusion: 1 integration test (lifecycle)
plus structural verification via cargo check (the filter uses the
existing list_memory_docs API, no new store calls to test)
- v2 Python select_skills chain-load: covered by the v1 unit tests
through shared semantic contract (both paths mirror the same
algorithm); a direct Python-level test would require spinning up
the Monty interpreter which is out of scope for this session.
Verification:
cargo test -p ironclaw_skills --lib: 159 passed
cargo test -p ironclaw_engine: 304 passed
cargo test --features libsql --test skill_setup_marker_lifecycle: 1 passed
cargo clippy --features libsql --tests --all-targets -- -D warnings: clean
* feat(skills): carry requires through v1→v2 migration + chain-load test
V2SkillMetadata was missing the `requires` field entirely, so the
v1→v2 skill migration silently dropped `requires.skills` and the
chain-loading code I added to the v2 Python orchestrator in the
previous commit was effectively dead code — it always read an empty
companion list.
This was caught while writing an end-to-end chain-load test: the v1
test (through the Rust selector) passes, the v2 test (through the
Python orchestrator) was failing in a way that only made sense if
the companion metadata never reached Python. Inspection confirmed
`V2SkillMetadata` had no `requires` field, only `activation`.
## Fix
1. `V2SkillMetadata` gains `pub requires: GatingRequirements` with
`#[serde(default)]` for backwards compatibility (legacy
MemoryDocs in existing databases deserialize with an empty
`requires`).
2. `src/bridge/skill_migration.rs::v1_skill_to_memory_doc` now
copies `skill.manifest.requires.clone()` into the new field.
3. Four other explicit `V2SkillMetadata { ... }` literal
constructions updated with `requires: Default::default()`:
- `crates/ironclaw_engine/src/memory/skill_tracker.rs` (test helper)
- `crates/ironclaw_engine/src/runtime/mission.rs` (test helper)
- `crates/ironclaw_skills/src/v2.rs` (serde roundtrip test)
- `tests/engine_v2_skill_codeact.rs` (test fixture)
## New test: tests/skill_chain_load_lifecycle.rs
End-to-end lifecycle test for chain-loading. Writes three skills to
a tempdir:
- `parent-setup-test` — scored by a distinctive keyword, declares
two companions via `requires.skills`
- `companion-one-test` / `companion-two-test` — zero-scoring on
their own (keywords deliberately don't match)
Each skill body carries a distinctive marker string
(`CHAIN-LOAD-PARENT-BODY-J4V`, `CHAIN-LOAD-COMPANION-ONE-K5W`,
`CHAIN-LOAD-COMPANION-TWO-L6X`) that the test greps for in the
captured LLM system prompt via `rig.captured_llm_requests()`. If a
marker is present, the skill was injected into the prompt; if
absent, it wasn't.
Two test variants:
- **v1** (default rig, Rust selector path): **PASSES**. Proves the
chain-loading pass in `prefilter_skills` correctly pulls in both
companions despite their zero individual scores.
- **v2** (with_engine_v2, Python orchestrator path):
**`#[ignore]`d** with a detailed explanation. The v2 engine runs
a Python orchestrator that makes multiple LLM calls per user
message, but the default TestRig uses a single-turn TraceLlm that
exhausts after the first call — observing skill injection through
the v2 path needs a multi-turn TraceLlm harness or a dedicated v2
skill test rig. The structural wiring for v2 chain-loading
(V2SkillMetadata.requires + skill_migration copy + Python
select_skills chain-load pass) compiles and passes the 304-test
engine suite, so this is a test-harness gap, not a code gap.
When the multi-turn harness exists, flipping `#[ignore]` on the v2
test will exercise the full path.
Verification:
cargo test -p ironclaw_skills --lib: 159 passed
cargo test -p ironclaw_engine --lib: 304 passed
cargo test --features libsql --test skill_chain_load_lifecycle
-- --test-threads=1: 1 passed, 1 ignored
cargo test --features libsql --test skill_setup_marker_lifecycle
-- --test-threads=1: 1 passed
cargo clippy --features libsql --tests --all-targets -- -D warnings: clean
Also includes an updated fixture recording from the last live
`e2e_github_dev_workflow` run (issue #2204, agent posted 2 comments,
cleanup closed it). No functional difference; committed for
completeness since the fixture was modified on disk by the live run
and the test is hermetic in replay mode.
* fix: adapt thread_ops test to staging's test helper API
Use make_test_agent_with_status_channel instead of removed
make_thread_ops_test_agent, StdMutex instead of TokioMutex,
and fix String comparison direction.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style: cargo fmt
* fix: remove dead try_add function and stale comments in Python orchestrator
Addresses PR #2268 review feedback: the try_add closure was defined but
never called since the logic was inlined for Monty compatibility.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reconcile test harness after staging merge
Restore our branch's test helpers (SessionTurn, finish_turns_strict,
with_skills_dir, loaded_skill_names, active_skill_names, etc.) that
staging removed, while incorporating staging's new features
(record_trace, with_no_trace_recording, secrets_store/owner_id
accessors). Bridge the API gap with finish_turns_simple for tests
using staging's (String, Vec<String>) tuple convention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address PR #2268 review feedback
- live_harness: replace panic with graceful TestMode::Skipped when
record_trace=false in replay mode; update e2e_live callers to check
mode() != Live instead of == Replay
- test_rig: match SecretError::NotFound explicitly in get_secret(),
return None silently instead of logging expected misses
- test_rig: replace brittle "already exists" string matching in
pre-seed loop with get_decrypted existence check before create
- default.py: align max_context_tokens fallback from 1000 to 2000
to match Rust ActivationCriteria default (both parent and companion)
- e2e_builtin_tool_coverage: fix routine_create_list using hardcoded
"test-user" instead of rig.owner_id() (broke when .with_skills()
changed channel user to config owner_id)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address PR #2268 review feedback (round 2)
1. Fix memory_write `path:` → `target:` in all 4 setup skill completion
markers (developer, ceo, content-creator, trader). The `memory_write`
tool reads `target`, not `path`, so markers were never written to the
correct location.
2. Add setup_marker validation in enforce_limits(): max 256 chars, reject
`..` path traversal. Prevents untrusted skills from abusing markers.
3. Fix v2 Python skill budget: default 4000 → 6000 to match v1 Rust
config. Also port the approx_tokens > declared * 2 sanity check from
Rust to prevent budget bypass via low max_context_tokens declarations.
4. Reorder developer-setup companion skills to put github/github-workflow
first (critical for setup) and fix misleading budget comment in config.
5. Move AssertUnwindSafe cleanup guard in e2e GitHub test to wrap
everything after create_issue, preventing orphaned issues on panic.
6. Scope workspace in select_active_skills to the requesting user_id so
multi-user channels check the correct user's setup marker state.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove duplicate skills_dir field from LiveTestHarnessBuilder
Both sides of the merge added the same field, resulting in a duplicate
declaration that failed compilation in test targets.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: address CI failures and Copilot review feedback
1. Fix formatting (cargo fmt).
2. Filter existing_titles to non-Skill docs in v2 orchestrator so setup
markers don't collide with skill doc titles of the same name.
3. Fix stale doc comment in types.rs (commitments/README.md →
commitments/.developer-setup-complete).
4. Fix misleading comment on v2 requires field — the full
GatingRequirements struct is preserved, not just the companion list.
5. Match SecretError::NotFound explicitly in test_rig pre-seed loop
instead of catching all errors — other errors (DB, crypto) now
surface instead of triggering a blind create.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
46 lines
742 B
Plaintext
46 lines
742 B
Plaintext
|
|
.env
|
|
.env.local
|
|
.env.*
|
|
!.env.example
|
|
|
|
# Claude Code worktrees and lock files
|
|
.claude/worktrees/
|
|
.claude/scheduled_tasks.lock
|
|
|
|
# Sidecar tool data
|
|
.sidecar/
|
|
.todos/
|
|
|
|
target/
|
|
|
|
# Python
|
|
__pycache__/
|
|
*.pyc
|
|
/tests/e2e/.venv/
|
|
|
|
# Benchmark results (local runs, not committed)
|
|
bench-results/
|
|
|
|
# Coverage reports (local runs, not committed)
|
|
/coverage/
|
|
|
|
# WASM build artifacts (loaded from disk, not bundled)
|
|
*.wasm
|
|
|
|
# Traces
|
|
trace_*.json
|
|
|
|
# Local Claude Code settings (machine-specific, should not be committed)
|
|
.claude/settings.local.json
|
|
.worktrees/
|
|
|
|
# Python cache
|
|
__pycache__/
|
|
*.pyc
|
|
*.pyo
|
|
*.pyd
|
|
engine_trace_*.json
|
|
tests/fixtures/llm_traces/live/github_dev_workflow_full_loop.json
|
|
tests/fixtures/llm_traces/live/github_dev_workflow_full_loop.log
|