Files
ironclaw/tests/support/test_rig.rs
firat.sertgoz 08693aa3cc feat(skills): activation feedback pipeline + install idempotence (#2530)
* feat(events): SkillActivated carries activation feedback notes

Add an optional `feedback: Vec<String>` field to the SkillActivated
event so the engine and selector can surface human-readable activation
notes (chain-load reasons, marker exclusions, scoring summaries) to the
UI. Wire the field through the StatusUpdate, the SSE bridge, and the
gateway's activity timeline; serialize-skip empty vectors so the wire
format stays backwards compatible.

* fix(skills): skill_install never prompts when skill is already loaded

When the LLM force-activates a persona via `/ceo-setup` it sometimes
follows up with a redundant `skill_install("ceo-setup")` call. The
`execute` path was already idempotent (returns `already_installed`
without touching the catalog), but `requires_approval` still gated
the call behind a confirmation prompt — pure friction on a guaranteed
no-op.

Mirror the idempotent shortcut in `requires_approval`: when a skill
with the requested name is already loaded (bundled, user, workspace,
or previously installed), return `ApprovalRequirement::Never`. The
shortcut wins even when `install_dependencies=true` because the
top-level execute is still a no-op (companions get reconciled by their
own activation paths). Regression test covers all three cases.

* fix(skills): preserve approval for dependency installs

* fix(events): include feedback in AppEvent::SkillActivated all-variants list

The variant-enumeration constructor in event.rs:501 was missed when
the new `feedback` field was added to AppEvent::SkillActivated, breaking
the build with E0063. All three Clippy CI jobs failed on this.

Regression: covered by `cargo build --all-features`, which fails to
compile if any variant in this list is constructed with missing fields.

* feat(skills): wire up v1 feedback producer for SkillActivated

The `SkillActivated` event carried an empty `feedback` field because
nothing populated it. This adds the producer end of the pipeline.

**Selector:**
- `prefilter_skills` now returns `SelectionOutcome { selected, notes }`.
- `try_select` returns a reason enum (`Selected`, `BudgetFull`,
  `CandidateLimit`, `MarkerSatisfied`, `AlreadySelected`) so callers
  can render distinct notes instead of opaque "skipped".
- Notes generated for:
  - `<companion>: chain-loaded from <parent>`
  - `<companion>: chain-load skipped (budget full)`
  - `<companion>: chain-load skipped (max active skills reached)`
  - `<companion>: chain-load skipped (setup already complete)`
  - `<skill>: skipped (skill context budget exhausted)` for parents
    that scored but didn't fit.

**Agent loop:**
- `select_active_skills` returns the notes alongside selected skills
  and prepends a `<skill>: force-activated via /mention` note for each
  explicit mention.

**Dispatcher:**
- Emits `StatusUpdate::SkillActivated { skill_names, feedback }` via
  `channels.send_status` whenever something activated or notes exist
  (so "nothing loaded because budget exhausted" surfaces too).
- Silent when nothing activated and no notes — no UI noise.

**Stale comment:**
- Router's v2-bridge comment no longer claims v1 callers populate
  feedback "directly on `StatusUpdate`"; the v1 dispatcher now emits
  its own event, and v2 remains empty until the Python orchestrator
  is updated.

Regression: existing selector test `test_chain_load_respects_budget`,
`test_chain_load_skips_companion_with_satisfied_marker`, and
`test_chain_load_is_non_transitive` now also assert that the
corresponding note is in `outcome.notes`. The 42 selector tests and
503 agent-module tests all pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Illia Polosukhin <ilblackdragon@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 22:19:32 +09:00

1507 lines
61 KiB
Rust

//! TestRig -- a builder for wiring a real Agent with a replay LLM and test channel.
//!
//! Constructs a full `Agent` with real tools but a `TraceLlm` (or custom LLM)
//! and a `TestChannel`, runs the agent in a background tokio task, and provides
//! methods to inject messages, wait for responses, and inspect tool calls.
#![allow(dead_code)] // Public API consumed by later test modules (Task 4+).
use std::sync::Arc;
use std::time::{Duration, Instant};
use ironclaw::agent::{Agent, AgentDeps};
use ironclaw::app::{AppBuilder, AppBuilderFlags};
use ironclaw::channels::web::log_layer::LogBroadcaster;
use ironclaw::channels::{OutgoingResponse, StatusUpdate};
use ironclaw::config::Config;
use ironclaw::db::Database;
use ironclaw::llm::{LlmProvider, SessionConfig, SessionManager};
use ironclaw::tools::Tool;
use crate::support::instrumented_llm::InstrumentedLlm;
use crate::support::metrics::{ToolInvocation, TraceMetrics};
use crate::support::test_channel::{CapturedEvent, TestChannel, TestChannelHandle};
use crate::support::trace_llm::{LlmTrace, TraceLlm};
use ironclaw::llm::recording::{HttpExchange, HttpInterceptor, ReplayingHttpInterceptor};
// ---------------------------------------------------------------------------
// TestRig
// ---------------------------------------------------------------------------
/// Substring unique to the static bootstrap greeting (GREETING.md).
/// Used to transparently filter per-user bootstrap greetings from the
/// response stream so tests don't need to account for them manually.
const BOOTSTRAP_GREETING_MARKER: &str = "always-on chief of staff";
/// Configuration for selectively seeding `secrets` rows into a fresh test
/// rig database from an existing libSQL file.
///
/// Live tests use this to pull *only* the credentials they need (e.g. a
/// Google OAuth token) out of the developer's real `~/.ironclaw/ironclaw.db`
/// without cloning the rest of the database. Memory, history, secrets the
/// test didn't ask for — none of it crosses the boundary. The destination
/// DB starts empty, the listed secret rows are inserted under the test
/// rig's owner user, and the test must seed any other state itself.
#[derive(Clone, Debug)]
pub struct SeededSecretsConfig {
/// Path to the source libSQL file (typically `~/.ironclaw/ironclaw.db`).
pub source_path: std::path::PathBuf,
/// User ID to filter the source rows on (typically the developer's
/// owner_id from the live config).
pub source_user_id: String,
/// Names of the secrets to copy. Names not present in the source are
/// logged as warnings and silently skipped — the test will fail fast
/// on its own missing-credential path if a required name was wrong.
pub names: Vec<String>,
}
/// Open the source libSQL database read-only and copy the listed secret
/// rows into the destination database under `owner_user_id`. The source
/// is *only* read; the destination must already have the `secrets` table
/// (i.e. migrations have run on it).
///
/// This intentionally avoids `std::fs::copy` of the underlying SQLite
/// file: copying the whole DB would also pull in workspace memory,
/// conversation history, and *every other* secret row, which is the
/// regression we are explicitly fixing.
#[cfg(feature = "libsql")]
async fn seed_secrets_into(
dest_db: &libsql::Database,
config: &SeededSecretsConfig,
owner_user_id: &str,
) -> Result<(), String> {
if config.names.is_empty() {
return Ok(());
}
if !config.source_path.exists() {
return Err(format!(
"source libSQL DB not found: {}",
config.source_path.display()
));
}
eprintln!(
"[TestRig] Seeding {} secret(s) from {} (user_id={}) → temp DB (owner={})",
config.names.len(),
config.source_path.display(),
config.source_user_id,
owner_user_id,
);
// Open the source via a separate libSQL connection. We never write to
// it. The source process (the developer's running ironclaw) can keep
// running concurrently — libSQL's WAL mode permits a reader from
// another connection.
let src_db = libsql::Builder::new_local(&config.source_path)
.build()
.await
.map_err(|e| format!("open source libSQL DB: {e}"))?;
let src_conn = src_db
.connect()
.map_err(|e| format!("connect to source libSQL DB: {e}"))?;
// Build a parameterized IN-clause: `?, ?, ?...`. We bind names + the
// owner separately to keep this injection-safe even though the input
// is technically test-controlled.
let placeholders = std::iter::repeat_n("?", config.names.len())
.collect::<Vec<_>>()
.join(",");
let select_sql = format!(
"SELECT name, encrypted_value, key_salt, provider, expires_at \
FROM secrets \
WHERE user_id = ? AND name IN ({placeholders})"
);
let mut params: Vec<libsql::Value> = Vec::with_capacity(1 + config.names.len());
params.push(libsql::Value::Text(config.source_user_id.clone()));
for name in &config.names {
params.push(libsql::Value::Text(name.clone()));
}
let mut rows = src_conn
.query(&select_sql, params)
.await
.map_err(|e| format!("query source secrets: {e}"))?;
let dest_conn = dest_db
.connect()
.map_err(|e| format!("connect to destination libSQL DB: {e}"))?;
let mut copied: Vec<String> = Vec::new();
while let Some(row) = rows
.next()
.await
.map_err(|e| format!("iterate source secrets: {e}"))?
{
let name: String = row.get(0).map_err(|e| format!("read secrets.name: {e}"))?;
let encrypted_value: Vec<u8> = row
.get(1)
.map_err(|e| format!("read secrets.encrypted_value: {e}"))?;
let key_salt: Vec<u8> = row
.get(2)
.map_err(|e| format!("read secrets.key_salt: {e}"))?;
let provider: Option<String> = row
.get(3)
.map_err(|e| format!("read secrets.provider: {e}"))?;
let expires_at: Option<String> = row
.get(4)
.map_err(|e| format!("read secrets.expires_at: {e}"))?;
let id = uuid::Uuid::new_v4().to_string();
dest_conn
.execute(
"INSERT INTO secrets \
(id, user_id, name, encrypted_value, key_salt, provider, expires_at) \
VALUES (?, ?, ?, ?, ?, ?, ?)",
libsql::params![
id,
owner_user_id.to_string(),
name.clone(),
encrypted_value,
key_salt,
provider,
expires_at,
],
)
.await
.map_err(|e| format!("insert seeded secret '{name}' into destination: {e}"))?;
copied.push(name);
}
let missing: Vec<&String> = config
.names
.iter()
.filter(|n| !copied.contains(n))
.collect();
if !missing.is_empty() {
eprintln!(
"[TestRig] WARNING: requested secrets not found in source: {:?}",
missing
);
}
eprintln!("[TestRig] Seeded secrets: {:?}", copied);
Ok(())
}
/// A running test agent with methods to inject messages and inspect results.
pub struct TestRig {
/// The test channel for sending messages and reading captures.
channel: Arc<TestChannel>,
/// Instrumented LLM for collecting token/call metrics.
instrumented_llm: Arc<InstrumentedLlm>,
/// When the rig was created (for wall-time measurement).
start_time: Instant,
/// Maximum tool-call iterations per agentic loop (for count-based limit detection).
max_tool_iterations: usize,
/// Handle to the background agent task (wrapped in Option so Drop can take it).
agent_handle: Option<tokio::task::JoinHandle<()>>,
/// Database handle for direct queries in tests.
#[cfg(feature = "libsql")]
db: Arc<dyn Database>,
/// Workspace handle for direct memory operations in tests.
#[cfg(feature = "libsql")]
workspace: Option<Arc<ironclaw::workspace::Workspace>>,
/// The underlying TraceLlm for inspecting captured requests.
#[cfg(feature = "libsql")]
trace_llm: Option<Arc<TraceLlm>>,
/// Extension manager for direct extension operations in tests.
#[cfg(feature = "libsql")]
extension_manager: Option<Arc<ironclaw::extensions::ExtensionManager>>,
/// Skill registry (if skills are enabled) for direct inspection in tests.
#[cfg(feature = "libsql")]
skill_registry: Option<Arc<std::sync::RwLock<ironclaw_skills::SkillRegistry>>>,
/// Session manager for direct session/thread access in tests.
#[cfg(feature = "libsql")]
session_manager: Arc<ironclaw::agent::SessionManager>,
/// Secrets store for tests that need to read pre-seeded credentials
/// (e.g. live tests that issue direct REST calls to the same backend
/// the agent is talking to). Pulled from `AppComponents.secrets_store`
/// during build.
#[cfg(feature = "libsql")]
secrets_store: Option<Arc<dyn ironclaw::secrets::SecretsStore + Send + Sync>>,
/// Owner ID used by the rig — needed by `get_secret` to look up
/// per-user secret rows.
#[cfg(feature = "libsql")]
owner_id: String,
/// Temp directory guard -- keeps the libSQL database file alive.
#[cfg(feature = "libsql")]
_temp_dir: tempfile::TempDir,
/// How many bootstrap greetings to keep in `wait_for_responses`.
/// 0 for normal tests (filter all greetings), 1 for `.with_bootstrap()`
/// tests (keep the startup greeting, filter per-user duplicates).
#[cfg(feature = "libsql")]
bootstrap_greetings_to_keep: usize,
}
impl TestRig {
/// Inject a user message into the agent.
pub async fn send_message(&self, content: &str) {
self.channel.send_message(content).await;
}
/// Inject a raw `IncomingMessage` (for tests that need attachments, etc.).
pub async fn send_incoming(&self, msg: ironclaw::channels::IncomingMessage) {
self.channel.send_incoming(msg).await;
}
/// Return all message lists that were sent to the LLM provider.
///
/// Only available when the rig was built with a `TraceLlm` (i.e., via `.with_trace()`).
pub fn captured_llm_requests(&self) -> Vec<Vec<ironclaw::llm::ChatMessage>> {
self.trace_llm
.as_ref()
.map(|t| t.captured_requests())
.unwrap_or_default()
}
/// Return the extension manager for direct extension operations in tests.
pub fn extension_manager(&self) -> Option<&Arc<ironclaw::extensions::ExtensionManager>> {
self.extension_manager.as_ref()
}
/// Return the session manager for direct session/thread access in tests.
#[cfg(feature = "libsql")]
pub fn session_manager(&self) -> &Arc<ironclaw::agent::SessionManager> {
&self.session_manager
}
/// Read the decrypted value of a pre-seeded secret by name.
///
/// Returns `None` if the rig has no SecretsStore wired (non-libsql
/// configurations) or if the secret doesn't exist for this rig's
/// owner_id. Used by live tests that need to issue direct REST
/// calls to the same backend the agent is talking to (e.g. setting
/// up a real GitHub issue before the agent runs against it).
///
/// Note: this returns the secret in plaintext. Live tests should
/// only call this for credentials that were pre-seeded via
/// `with_secret` or `with_secrets`, never for arbitrary secrets the
/// rig may have inherited from a real DB.
#[cfg(feature = "libsql")]
pub async fn get_secret(&self, name: &str) -> Option<String> {
let store = self.secrets_store.as_ref()?;
match store.get_decrypted(&self.owner_id, name).await {
Ok(decrypted) => Some(decrypted.expose().to_string()),
Err(e) => {
// NotFound is expected for optional secrets — only log real errors
if !matches!(e, ironclaw::secrets::SecretError::NotFound(_)) {
eprintln!(
"[TestRig] get_secret('{name}') for owner '{}' failed: {e}",
self.owner_id
);
}
None
}
}
}
/// Get the secrets store for direct credential manipulation.
#[cfg(feature = "libsql")]
pub fn secrets_store(&self) -> Option<&Arc<dyn ironclaw::secrets::SecretsStore + Send + Sync>> {
self.secrets_store.as_ref()
}
/// The owner identity resolved from `Config::owner_id`.
#[cfg(feature = "libsql")]
pub fn owner_id(&self) -> &str {
&self.owner_id
}
/// Wait until at least `n` non-bootstrap responses have been captured, or
/// `timeout` elapses.
///
/// Per-user bootstrap greetings (fired when `tenant_ctx` creates a workspace
/// for a non-owner user) are transparently filtered from the response stream.
/// For `.with_bootstrap()` tests, the startup greeting is kept (1 allowed)
/// while additional per-user greetings are still filtered.
pub async fn wait_for_responses(&self, n: usize, timeout: Duration) -> Vec<OutgoingResponse> {
let deadline = tokio::time::Instant::now() + timeout;
let mut interval = Duration::from_millis(50);
let max_interval = Duration::from_millis(500);
loop {
let filtered = self.filter_responses(self.channel.captured_responses_async().await);
if filtered.len() >= n {
return filtered;
}
if tokio::time::Instant::now() >= deadline {
return filtered;
}
tokio::time::sleep(interval).await;
interval = (interval * 2).min(max_interval);
}
}
/// Filter bootstrap greetings from the response stream.
///
/// Keeps up to `bootstrap_greetings_to_keep` greeting responses (0 for
/// normal tests, 1 for `.with_bootstrap()` tests) and drops the rest.
fn filter_responses(&self, responses: Vec<OutgoingResponse>) -> Vec<OutgoingResponse> {
let mut greetings_kept = 0usize;
responses
.into_iter()
.filter(|r| {
if r.content.contains(BOOTSTRAP_GREETING_MARKER) {
greetings_kept += 1;
greetings_kept <= self.bootstrap_greetings_to_keep
} else {
true
}
})
.collect()
}
/// Return the names of all `ToolStarted` events captured so far.
pub fn tool_calls_started(&self) -> Vec<String> {
self.channel.tool_calls_started()
}
/// Return the filtered list of captured responses so far.
///
/// Mirrors the bootstrap-greeting filtering used by `wait_for_responses`.
pub async fn captured_responses(&self) -> Vec<OutgoingResponse> {
self.filter_responses(self.channel.captured_responses_async().await)
}
/// Return `(name, success)` for all `ToolCompleted` events captured so far.
pub fn tool_calls_completed(&self) -> Vec<(String, bool)> {
self.channel.tool_calls_completed()
}
/// Return `(name, preview)` for all `ToolResult` events captured so far.
pub fn tool_results(&self) -> Vec<(String, String)> {
self.channel.tool_results()
}
/// Return `(name, duration_ms)` for all completed tools with timing data.
pub fn tool_timings(&self) -> Vec<(String, u64)> {
self.channel.tool_timings()
}
/// Wait until a `Status("Done")` event has been captured, or `timeout` elapses.
pub async fn wait_for_done(&self, timeout: Duration) -> bool {
self.channel.wait_for_done(timeout).await
}
/// Return a snapshot of all captured status events.
pub fn captured_status_events(&self) -> Vec<StatusUpdate> {
self.channel.captured_status_events()
}
/// Return the names of skills loaded into the registry, if skills are
/// enabled. Useful for verifying the registry discovered the SKILL.md
/// files from `with_skills_dir()`.
pub fn loaded_skill_names(&self) -> Vec<String> {
self.skill_registry
.as_ref()
.and_then(|r| {
r.read()
.ok()
.map(|g| g.skills().iter().map(|s| s.name().to_string()).collect())
})
.unwrap_or_default()
}
/// Return the names of skills that were activated during this session,
/// extracted from `SkillActivated` status events.
pub fn active_skill_names(&self) -> Vec<String> {
self.captured_status_events()
.iter()
.filter_map(|event| match event {
StatusUpdate::SkillActivated { skill_names, .. } => Some(skill_names.clone()),
_ => None,
})
.flatten()
.collect()
}
/// Return the ordered log of captured outbound events.
pub fn captured_events(&self) -> Vec<CapturedEvent> {
self.channel.captured_events()
}
/// Clear all captured responses and status events.
pub async fn clear(&self) {
self.channel.clear().await;
}
/// Number of LLM calls made so far.
pub fn llm_call_count(&self) -> u32 {
self.instrumented_llm.call_count()
}
/// Total input tokens across all LLM calls.
pub fn total_input_tokens(&self) -> u32 {
self.instrumented_llm.total_input_tokens()
}
/// Total output tokens across all LLM calls.
pub fn total_output_tokens(&self) -> u32 {
self.instrumented_llm.total_output_tokens()
}
/// Estimated total cost in USD.
pub fn estimated_cost_usd(&self) -> f64 {
self.instrumented_llm.estimated_cost_usd()
}
/// Wall-clock time since rig creation.
pub fn elapsed_ms(&self) -> u64 {
self.start_time.elapsed().as_millis() as u64
}
/// Collect a complete `TraceMetrics` snapshot from all captured data.
///
/// Call this after `wait_for_responses()` to get the full metrics for the
/// scenario. The `turns` count is based on the number of captured responses.
pub async fn collect_metrics(&self) -> TraceMetrics {
let completed = self.tool_calls_completed();
// Build ToolInvocation records from ToolStarted/ToolCompleted pairs,
// matching each completion with its captured timing data.
let timings = self.tool_timings();
let mut timing_iter_by_name: std::collections::HashMap<&str, Vec<u64>> =
std::collections::HashMap::new();
for (name, ms) in &timings {
timing_iter_by_name
.entry(name.as_str())
.or_default()
.push(*ms);
}
let tool_invocations: Vec<ToolInvocation> = completed
.iter()
.map(|(name, success)| {
let duration_ms = timing_iter_by_name
.get_mut(name.as_str())
.and_then(|v| {
if v.is_empty() {
None
} else {
Some(v.remove(0))
}
})
.unwrap_or(0);
ToolInvocation {
name: name.clone(),
duration_ms,
success: *success,
}
})
.collect();
// Detect if iteration limit was hit by comparing completed tool-call count
// against the configured max_tool_iterations threshold.
let hit_iteration_limit = completed.len() >= self.max_tool_iterations;
// Count turns as the number of captured responses.
let responses = self.channel.captured_responses();
let turns = responses.len() as u32;
TraceMetrics {
wall_time_ms: self.elapsed_ms(),
llm_calls: self.instrumented_llm.call_count(),
input_tokens: self.instrumented_llm.total_input_tokens(),
output_tokens: self.instrumented_llm.total_output_tokens(),
estimated_cost_usd: self.instrumented_llm.estimated_cost_usd(),
tool_calls: tool_invocations,
turns,
hit_iteration_limit,
hit_timeout: false, // Caller can set this based on wait_for_responses result.
}
}
/// Run a complete multi-turn trace, injecting user messages from the trace
/// and waiting for responses after each turn.
///
/// Returns a `Vec` of response lists, one per turn. Status events and tool
/// call data accumulate across all turns (no clearing between turns), so
/// post-run assertions like `tool_calls_started()` reflect the whole trace.
pub async fn run_trace(
&self,
trace: &LlmTrace,
timeout: Duration,
) -> Vec<Vec<OutgoingResponse>> {
let mut all_responses: Vec<Vec<OutgoingResponse>> = Vec::new();
let mut total_responses = 0usize;
for turn in &trace.turns {
self.send_message(&turn.user_input).await;
let responses = self.wait_for_responses(total_responses + 1, timeout).await;
// Extract only the new responses from this turn.
let turn_responses: Vec<OutgoingResponse> =
responses.into_iter().skip(total_responses).collect();
total_responses += turn_responses.len();
all_responses.push(turn_responses);
}
all_responses
}
/// Run a trace, then verify all declarative `expects` (top-level and per-turn).
///
/// Returns the per-turn response lists for additional manual assertions.
pub async fn run_and_verify_trace(
&self,
trace: &LlmTrace,
timeout: Duration,
) -> Vec<Vec<OutgoingResponse>> {
use crate::support::assertions::verify_expects;
let all_responses = self.run_trace(trace, timeout).await;
// Verify top-level expects against all accumulated data.
if !trace.expects.is_empty() {
let all_response_strings: Vec<String> = all_responses
.iter()
.flat_map(|turn| turn.iter().map(|r| r.content.clone()))
.collect();
let started = self.tool_calls_started();
let completed = self.tool_calls_completed();
let mut results = self.tool_results();
for status in self.channel.captured_status_events() {
if let ironclaw::channels::StatusUpdate::ToolCompleted {
name,
success: false,
error,
parameters,
..
} = status
{
let detail = format!(
"error={}; params={}",
error.unwrap_or_else(|| "unknown".to_string()),
parameters.unwrap_or_else(|| "{}".to_string())
);
results.push((name, detail));
}
}
verify_expects(
&trace.expects,
&all_response_strings,
&started,
&completed,
&results,
"top-level",
);
}
all_responses
}
/// Verify top-level `expects` from a trace against already-captured data.
///
/// Call this after `send_message()` + `wait_for_responses()` for flat-format
/// traces. For multi-turn traces, use `run_and_verify_trace()` instead.
pub fn verify_trace_expects(&self, trace: &LlmTrace, responses: &[OutgoingResponse]) {
use crate::support::assertions::verify_expects;
if trace.expects.is_empty() {
return;
}
let response_strings: Vec<String> = responses.iter().map(|r| r.content.clone()).collect();
let started = self.tool_calls_started();
let completed = self.tool_calls_completed();
let mut results = self.tool_results();
for status in self.channel.captured_status_events() {
if let ironclaw::channels::StatusUpdate::ToolCompleted {
name,
success: false,
error,
parameters,
..
} = status
{
let detail = format!(
"error={}; params={}",
error.unwrap_or_else(|| "unknown".to_string()),
parameters.unwrap_or_else(|| "{}".to_string())
);
results.push((name, detail));
}
}
verify_expects(
&trace.expects,
&response_strings,
&started,
&completed,
&results,
"top-level",
);
}
/// Signal the channel to shut down and abort the background agent task.
pub fn shutdown(mut self) {
self.channel.signal_shutdown();
if let Some(handle) = self.agent_handle.take() {
handle.abort();
}
}
}
impl Drop for TestRig {
fn drop(&mut self) {
if let Some(handle) = self.agent_handle.take()
&& !handle.is_finished()
{
handle.abort();
}
}
}
// ---------------------------------------------------------------------------
// TestRigBuilder
// ---------------------------------------------------------------------------
/// Specification for loading a real WASM tool in the test rig.
pub struct WasmToolSpec {
pub name: String,
pub wasm_path: std::path::PathBuf,
pub capabilities_path: Option<std::path::PathBuf>,
}
/// Builder for constructing a `TestRig`.
pub struct TestRigBuilder {
trace: Option<LlmTrace>,
llm: Option<Arc<dyn LlmProvider>>,
config_override: Option<Config>,
max_tool_iterations: usize,
injection_check: bool,
auto_approve_tools: Option<bool>,
enable_skills: bool,
skills_dir: Option<std::path::PathBuf>,
enable_routines: bool,
http_exchanges: Vec<HttpExchange>,
http_interceptor_override: Option<Arc<dyn HttpInterceptor>>,
extra_tools: Vec<Arc<dyn Tool>>,
wasm_tools: Vec<WasmToolSpec>,
keep_bootstrap: bool,
engine_v2: bool,
channel_name_override: Option<String>,
seeded_secrets: Option<SeededSecretsConfig>,
/// Pre-seed the SecretsStore with `(name, value)` pairs before the
/// agent starts. Used by live tests that need a credential to *exist*
/// (so the kernel pre-flight auth gate stays out of the way) but
/// don't actually call the credentialed API.
pre_seed_secrets: Vec<(String, String)>,
}
impl TestRigBuilder {
/// Create a new builder with defaults.
pub fn new() -> Self {
Self {
trace: None,
llm: None,
config_override: None,
max_tool_iterations: 10,
injection_check: false,
auto_approve_tools: Some(true),
enable_skills: false,
skills_dir: None,
enable_routines: false,
http_exchanges: Vec::new(),
http_interceptor_override: None,
extra_tools: Vec::new(),
wasm_tools: Vec::new(),
keep_bootstrap: false,
engine_v2: false,
channel_name_override: None,
seeded_secrets: None,
pre_seed_secrets: Vec::new(),
}
}
/// Pre-seed a secret in the SecretsStore before the agent starts.
///
/// This is for tests that need a credential to *exist* so the
/// kernel-level pre-flight auth gate (which fires when a skill with
/// a credential spec activates) doesn't block the conversation. The
/// value can be any non-empty string — the test isn't actually
/// hitting the credentialed API, the credential just needs to be
/// present in the store under the test's owner_id.
///
/// Note: only takes effect when the rig has a working `SecretsStore`
/// (i.e., the libSQL backend with `with_database_and_handles()`,
/// which is the standard rig setup).
pub fn with_secret(mut self, name: impl Into<String>, value: impl Into<String>) -> Self {
self.pre_seed_secrets.push((name.into(), value.into()));
self
}
/// Override the test channel name (default: "test", or "gateway" when
/// `.with_bootstrap()` is set). Use this when you need the channel name to
/// match a real-world channel (e.g. "gateway") so that downstream features
/// keyed on the channel name (assistant conversations, mission notify
/// channels) behave the same as in production.
pub fn with_channel_name(mut self, name: impl Into<String>) -> Self {
self.channel_name_override = Some(name.into());
self
}
/// Selectively seed `secrets` rows from an existing libSQL file into
/// the test rig's fresh temp database.
///
/// Live tests use this to pull just the credentials they need (e.g.
/// `google_oauth_token`) out of the developer's real
/// `~/.ironclaw/ironclaw.db` so OAuth-backed flows work end-to-end —
/// without cloning conversation history, workspace memory, or any
/// secret the test didn't ask for. The destination DB starts empty;
/// the listed rows are inserted under the test rig's owner user; any
/// other state the test relies on must be seeded by the test itself.
///
/// Names that don't exist in the source are logged as warnings and
/// silently skipped — the test will fail fast on its own missing
/// credential path if a required name was wrong.
pub fn with_seeded_secrets(
mut self,
source_path: std::path::PathBuf,
source_user_id: impl Into<String>,
names: impl IntoIterator<Item = impl Into<String>>,
) -> Self {
self.seeded_secrets = Some(SeededSecretsConfig {
source_path,
source_user_id: source_user_id.into(),
names: names.into_iter().map(Into::into).collect(),
});
self
}
/// Load a real WASM tool binary into the test rig.
///
/// The tool will be compiled, registered, and wired with the same HTTP
/// interceptor used for `with_http_exchanges()`, so `http_exchanges` in
/// the trace can specify expected requests/responses for WASM tool HTTP calls.
///
/// If the WASM binary does not exist at build time, the tool is silently
/// skipped (logged as a warning). Tests should use `#[ignore]` or check
/// for the binary in a preamble if the tool is required.
pub fn with_wasm_tool(
mut self,
name: impl Into<String>,
wasm_path: impl Into<std::path::PathBuf>,
capabilities_path: Option<std::path::PathBuf>,
) -> Self {
self.wasm_tools.push(WasmToolSpec {
name: name.into(),
wasm_path: wasm_path.into(),
capabilities_path,
});
self
}
/// Set the LLM trace to replay.
pub fn with_trace(mut self, trace: LlmTrace) -> Self {
self.trace = Some(trace);
self
}
/// Override the LLM provider directly (takes precedence over trace).
pub fn with_llm(mut self, llm: Arc<dyn LlmProvider>) -> Self {
self.llm = Some(llm);
self
}
/// Override the Config to mirror real binary behavior.
///
/// When set, uses this config instead of `Config::for_testing()`.
/// The database path is still overridden to use a temp libSQL file,
/// but agent settings (`allow_local_tools`, `engine_v2`, etc.) are
/// preserved from the provided config. Post-build forcing of
/// `allow_local_tools = true` is skipped so the test matches the
/// real binary's tool availability.
pub fn with_config(mut self, config: Config) -> Self {
self.config_override = Some(config);
self
}
/// Set the maximum number of tool iterations per agentic loop invocation.
pub fn with_max_tool_iterations(mut self, n: usize) -> Self {
self.max_tool_iterations = n;
self
}
/// Register additional custom tools (e.g. stub tools for testing).
pub fn with_extra_tools(mut self, tools: Vec<Arc<dyn Tool>>) -> Self {
self.extra_tools = tools;
self
}
/// Enable prompt injection detection in the safety layer.
///
/// When enabled, tool outputs are scanned for injection patterns
/// (e.g., "ignore previous instructions", special tokens like `<|endoftext|>`)
/// and critical patterns are escaped before reaching the LLM.
pub fn with_injection_check(mut self, enable: bool) -> Self {
self.injection_check = enable;
self
}
/// Override agent-level automatic approval of `UnlessAutoApproved` tools.
pub fn with_auto_approve_tools(mut self, enable: bool) -> Self {
self.auto_approve_tools = Some(enable);
self
}
/// Enable skill discovery and registration for this test rig.
pub fn with_skills(mut self) -> Self {
self.enable_skills = true;
self
}
/// Set a custom skills directory so the test rig loads skill files
/// from a real path (e.g. the repo's `skills/` directory) instead of
/// an empty temp directory. Implies `with_skills()`.
pub fn with_skills_dir(mut self, dir: std::path::PathBuf) -> Self {
self.enable_skills = true;
self.skills_dir = Some(dir);
self
}
/// Enable the routines system so the scheduler is wired with a `RoutineEngine`,
/// allowing routine jobs to actually execute. Routine tools are always registered
/// but require the engine to dispatch jobs.
pub fn with_routines(mut self) -> Self {
self.enable_routines = true;
self
}
/// Keep `bootstrap_pending` so the proactive greeting fires on startup.
pub fn with_bootstrap(mut self) -> Self {
self.keep_bootstrap = true;
self
}
/// Route messages through the engine v2 pipeline instead of the v1 agentic loop.
pub fn with_engine_v2(mut self) -> Self {
self.engine_v2 = true;
self
}
/// Add pre-recorded HTTP exchanges for the `ReplayingHttpInterceptor`.
///
/// When set, all `http` tool calls will return these responses in order
/// instead of making real network requests.
pub fn with_http_exchanges(mut self, exchanges: Vec<HttpExchange>) -> Self {
self.http_exchanges = exchanges;
self
}
/// Override the HTTP interceptor directly.
///
/// When set, this interceptor is used instead of constructing a
/// `ReplayingHttpInterceptor` from trace http_exchanges or
/// `with_http_exchanges()`. Useful for live-mode recording where a
/// `RecordingHttpInterceptor` captures real HTTP traffic.
pub fn with_http_interceptor(mut self, interceptor: Arc<dyn HttpInterceptor>) -> Self {
self.http_interceptor_override = Some(interceptor);
self
}
/// Build the test rig, creating a real agent and spawning it in the background.
///
/// Uses `AppBuilder::build_all()` to get the same component set as the real
/// binary, with only the LLM swapped for TraceLlm.
///
/// Requires the `libsql` feature for the embedded test database.
#[cfg(feature = "libsql")]
pub async fn build(self) -> TestRig {
use ironclaw::channels::ChannelManager;
use ironclaw::db::libsql::LibSqlBackend;
// Destructure self up front to avoid partial-move issues.
let TestRigBuilder {
trace,
llm,
config_override,
max_tool_iterations,
injection_check,
auto_approve_tools,
enable_skills,
skills_dir,
enable_routines,
http_exchanges: explicit_http_exchanges,
http_interceptor_override,
extra_tools,
wasm_tools,
keep_bootstrap,
engine_v2,
channel_name_override,
seeded_secrets,
pre_seed_secrets,
} = self;
// 1. Create temp dir + fresh libSQL database + run migrations.
//
// The destination DB is always created empty. If the test asked
// for specific secrets via `with_seeded_secrets(...)`, we copy
// *only* those rows out of the source DB after migrations have
// run — never the whole file. Memory, history, conversations,
// and unrequested secrets stay isolated in the source.
let temp_dir = tempfile::tempdir().expect("failed to create temp dir");
let db_path = temp_dir.path().join("test_rig.db");
let backend = LibSqlBackend::new_local(&db_path)
.await
.expect("failed to create test LibSqlBackend");
backend
.run_migrations()
.await
.expect("failed to run migrations");
// Build the backend-specific handles so AppBuilder can wire the
// secrets store. `with_database()` alone leaves handles=None,
// which silently disables `SecretsStore` and breaks every test
// that needs OAuth/encrypted credentials. `with_database_and_handles()`
// is the right pairing.
let db_handles = ironclaw::db::DatabaseHandles {
#[cfg(feature = "libsql")]
libsql_db: Some(backend.shared_db()),
#[cfg(feature = "postgres")]
pg_pool: None,
};
let db: Arc<dyn ironclaw::db::Database> = Arc::new(backend);
// 2. Build Config.
let has_config_override = config_override.is_some();
let has_skills_dir_override = skills_dir.is_some();
let skills_dir = skills_dir.unwrap_or_else(|| temp_dir.path().join("skills"));
let installed_skills_dir = temp_dir.path().join("installed_skills");
// Only create the tempdir skills dir if we're using it (i.e. no override).
// Do not try to create the override path — callers are responsible for
// providing an existing directory.
if !has_skills_dir_override {
let _ = std::fs::create_dir_all(&skills_dir);
}
let _ = std::fs::create_dir_all(&installed_skills_dir);
let mut config = if let Some(mut cfg) = config_override {
// Override database to use temp libSQL, but preserve agent/llm settings.
cfg.database.backend = ironclaw::config::DatabaseBackend::LibSql;
cfg.database.libsql_path = Some(db_path);
cfg.skills.local_dir = skills_dir.clone();
cfg.skills.installed_dir = installed_skills_dir.clone();
cfg
} else {
Config::for_testing(db_path, skills_dir.clone(), installed_skills_dir.clone())
};
config.agent.max_tool_iterations = max_tool_iterations;
config.safety.injection_check_enabled = injection_check;
config.skills.enabled = enable_skills;
if let Some(v) = auto_approve_tools {
config.agent.auto_approve_tools = v;
}
// 2b. Selectively seed `secrets` rows from the source DB if the
// test asked for it. We seed *after* migrations have run (so the
// schema exists in the destination) and *under the owner_user_id
// from the active config* so production credential lookups
// (`SELECT ... WHERE user_id = owner`) hit the seeded rows. The
// source DB is opened read-only via a separate libSQL connection;
// nothing else (memory, conversations, other secrets) crosses the
// boundary.
#[cfg(feature = "libsql")]
if let Some(ref ss) = seeded_secrets {
let owner = config.owner_id.clone();
let dest_handle = db_handles
.libsql_db
.as_ref()
.expect("libsql backend handle is required to seed secrets");
seed_secrets_into(dest_handle.as_ref(), ss, &owner)
.await
.expect("failed to seed live-test secrets into temp DB");
}
// 3. Create SessionManager + LogBroadcaster.
let session = Arc::new(SessionManager::new(SessionConfig::default()));
let log_broadcaster = Arc::new(LogBroadcaster::new());
// 4. Create TraceLlm + InstrumentedLlm, extract HTTP exchanges for replay.
let trace_http_exchanges = trace
.as_ref()
.map(|t| t.http_exchanges.clone())
.unwrap_or_default();
let mut trace_llm_ref: Option<Arc<TraceLlm>> = None;
let base_llm: Arc<dyn LlmProvider> = if let Some(llm) = llm {
llm
} else if let Some(trace) = trace {
let tlm = Arc::new(TraceLlm::from_trace(trace));
trace_llm_ref = Some(Arc::clone(&tlm));
tlm
} else {
let trace = LlmTrace::single_turn(
"test-rig-default",
"(default)",
vec![crate::support::trace_llm::TraceStep {
request_hint: None,
response: crate::support::trace_llm::TraceResponse::Text {
content: "Hello from test rig!".to_string(),
input_tokens: 10,
output_tokens: 5,
},
expected_tool_results: Vec::new(),
}],
);
let tlm = Arc::new(TraceLlm::from_trace(trace));
trace_llm_ref = Some(Arc::clone(&tlm));
tlm
};
let instrumented = Arc::new(InstrumentedLlm::new(base_llm));
let llm: Arc<dyn LlmProvider> = Arc::clone(&instrumented) as Arc<dyn LlmProvider>;
// 5. Build AppComponents via AppBuilder with injected DB and LLM.
let mut builder = AppBuilder::new(
config,
AppBuilderFlags::default(),
None,
session,
log_broadcaster,
);
builder.with_database_and_handles(Arc::clone(&db), db_handles);
builder.with_llm(llm);
let mut components = builder
.build_all()
.await
.expect("AppBuilder::build_all() failed in test rig");
// Clear the *owner* workspace bootstrap flag so tests don't get an
// unexpected proactive greeting on startup (unless the test explicitly
// wants to test the bootstrap flow via `.with_bootstrap()`).
//
// Per-user bootstrap greetings (fired when `tenant_ctx` creates a
// workspace for a non-owner user like "test-user") are allowed to
// happen naturally. They are transparently filtered from the response
// stream by `wait_for_responses` so tests don't need to account for
// them in response counting.
if !keep_bootstrap && let Some(ref ws) = components.workspace {
ws.take_bootstrap_pending();
}
// AppBuilder may re-resolve config from env/TOML and override test defaults.
// When a config override was provided, preserve its agent settings to mirror
// the real binary. Otherwise force deterministic test defaults.
if has_config_override {
if let Some(v) = auto_approve_tools {
components.config.agent.auto_approve_tools = v;
}
// allow_local_tools comes from the provided config.
// engine_v2: honour the builder's explicit override if set.
if engine_v2 {
components.config.agent.engine_v2 = true;
}
} else {
components.config.agent.auto_approve_tools = auto_approve_tools.unwrap_or(true);
components.config.agent.allow_local_tools = true;
components.config.agent.engine_v2 = engine_v2;
}
// Reset engine v2 global state so each test gets a clean engine instance.
if components.config.agent.engine_v2 {
ironclaw::bridge::reset_engine_state().await;
}
let scheduler_slot: ironclaw::tools::builtin::SchedulerSlot =
Arc::new(tokio::sync::RwLock::new(None));
// Build HTTP interceptor once — shared by both AgentDeps and WASM tools.
// Direct override takes priority (e.g. RecordingHttpInterceptor for live tests).
let http_interceptor: Option<Arc<dyn HttpInterceptor>> = if let Some(override_interceptor) =
http_interceptor_override
{
Some(override_interceptor)
} else {
let exchanges = if explicit_http_exchanges.is_empty() {
trace_http_exchanges
} else {
explicit_http_exchanges
};
if exchanges.is_empty() {
None
} else {
Some(Arc::new(ReplayingHttpInterceptor::new(exchanges)) as Arc<dyn HttpInterceptor>)
}
};
// 6. Register job tools, routine tools, and extra tools.
{
// Register filesystem/shell dev tools. When using a config override
// (real-binary parity mode), respect the allow_local_tools flag.
// Otherwise always register them for test convenience.
if !has_config_override || components.config.agent.allow_local_tools {
components.tools.register_dev_tools();
}
components.tools.register_job_tools(
Arc::clone(&components.context_manager),
Some(scheduler_slot.clone()),
None,
components.db.clone(),
None,
None,
None,
None,
);
// Routine tools: create a RoutineEngine with the LLM and workspace.
if let (Some(db_arc), Some(ws)) = (&components.db, &components.workspace) {
use ironclaw::agent::routine_engine::RoutineEngine;
use ironclaw::config::RoutineConfig;
let routine_config = RoutineConfig::default();
let (notify_tx, _notify_rx) = tokio::sync::mpsc::channel(16);
let engine = Arc::new(RoutineEngine::new(
routine_config,
ironclaw::tenant::SystemScope::new(Arc::clone(db_arc)),
components.llm.clone(),
Arc::clone(ws),
notify_tx,
None,
None,
components.tools.clone(),
components.safety.clone(),
ironclaw::agent::routine_engine::SandboxReadiness::DisabledByConfig,
));
components
.tools
.register_routine_tools(Arc::clone(db_arc), engine);
}
// Skills tools: rebuild the registry against the test's tempdir.
//
// `AppBuilder::init_database()` re-resolves `config` from
// DB/TOML/env, which clobbers `config.skills.local_dir` back
// to the default (`~/.ironclaw/skills/`). Any registry
// `build_all()` already constructed therefore points at the
// user's real skills dir, not the tempdir the test laid
// down. Rebuild here from the in-scope `skills_dir` /
// `installed_skills_dir`, actually run discovery, and write
// the paths back onto `components.config` so downstream
// consumers (AgentDeps::skills_config) see the same dirs.
if enable_skills {
components.config.skills.local_dir = skills_dir.clone();
components.config.skills.installed_dir = installed_skills_dir.clone();
let mut registry = ironclaw_skills::SkillRegistry::new(skills_dir.clone())
.with_installed_dir(installed_skills_dir.clone());
let _loaded = registry.discover_all().await;
let registry = Arc::new(std::sync::RwLock::new(registry));
let catalog = ironclaw_skills::catalog::shared_catalog();
components
.tools
.register_skill_tools(Arc::clone(&registry), Arc::clone(&catalog));
components.skill_registry = Some(registry);
components.skill_catalog = Some(catalog);
}
// Register any extra test-specific tools.
for tool in extra_tools {
components.tools.register(tool).await;
}
// Register WASM tools with the shared HTTP interceptor.
if !wasm_tools.is_empty() {
use ironclaw::tools::wasm::{
Capabilities, CapabilitiesFile, WasmRuntimeConfig, WasmToolRuntime,
WasmToolWrapper,
};
let runtime = Arc::new(
WasmToolRuntime::new(WasmRuntimeConfig::default())
.expect("create WASM runtime for test rig"),
);
for spec in wasm_tools {
if !spec.wasm_path.exists() {
tracing::warn!(
name = %spec.name,
path = %spec.wasm_path.display(),
"WASM tool binary not found, skipping"
);
continue;
}
let wasm_bytes = tokio::fs::read(&spec.wasm_path)
.await
.unwrap_or_else(|e| panic!("read {}: {e}", spec.wasm_path.display()));
let (capabilities, description) =
if let Some(cap_path) = &spec.capabilities_path {
if cap_path.exists() {
let cap_bytes = tokio::fs::read(cap_path)
.await
.unwrap_or_else(|e| panic!("read {}: {e}", cap_path.display()));
let cap_file = CapabilitiesFile::from_bytes(&cap_bytes)
.expect("parse capabilities.json");
(cap_file.to_capabilities(), cap_file.description.clone())
} else {
(Capabilities::default(), None)
}
} else {
(Capabilities::default(), None)
};
let prepared = runtime
.prepare(&spec.name, &wasm_bytes, None)
.await
.unwrap_or_else(|e| panic!("prepare WASM tool '{}': {e}", spec.name));
let mut wrapper =
WasmToolWrapper::new(Arc::clone(&runtime), prepared, capabilities);
if let Some(desc) = description {
wrapper = wrapper.with_description(desc);
}
if let Some(interceptor) = &http_interceptor {
wrapper = wrapper.with_http_interceptor(Arc::clone(interceptor));
}
components.tools.register(Arc::new(wrapper)).await;
}
}
}
// Save references for test accessors.
let db_ref = components.db.clone().expect("test rig requires a database");
let workspace_ref = components.workspace.clone();
let ext_mgr_ref = components.extension_manager.clone();
let skill_registry_ref = components.skill_registry.clone();
let session_manager_ref = Arc::new(ironclaw::agent::SessionManager::new());
// Pre-seed credentials BEFORE the agent starts. This lets live
// tests inject a fake `github_token` (or similar) so the kernel
// pre-flight auth gate doesn't block the conversation when a
// skill with a credential spec activates. The value is opaque —
// tests aren't actually hitting the credentialed API, the secret
// just needs to exist under the test's owner_id.
if !pre_seed_secrets.is_empty() {
if let Some(ref secrets_store) = components.secrets_store {
use ironclaw::secrets::CreateSecretParams;
let owner_id = components.config.owner_id.clone();
for (name, value) in &pre_seed_secrets {
let params = CreateSecretParams::new(name.clone(), value.clone());
// Only create if truly missing — other errors (DB, crypto)
// should surface rather than triggering a blind create.
match secrets_store.get_decrypted(&owner_id, name).await {
Ok(_) => {} // already seeded — skip
Err(ironclaw::secrets::SecretError::NotFound(_)) => {
if let Err(e) = secrets_store.create(&owner_id, params).await {
eprintln!(
"[TestRig] WARNING: failed to pre-seed secret '{name}' for \
user '{owner_id}': {e}"
);
}
}
Err(e) => {
eprintln!(
"[TestRig] WARNING: unexpected error checking secret '{name}': {e}"
);
}
}
}
} else {
eprintln!(
"[TestRig] WARNING: pre_seed_secrets requested but no SecretsStore is \
wired (need libsql backend with handles)"
);
}
}
// Capture handles tests need to read back state via the same
// SecretsStore the agent will use. Done before AgentDeps moves
// values out of `components`. The owner_id is required for any
// secret lookup since secret rows are keyed by user.
let secrets_store_ref = components.secrets_store.clone();
let owner_id_ref = components.config.owner_id.clone();
// 7. Construct AgentDeps from AppComponents (mirrors main.rs).
let deps = AgentDeps {
owner_id: components.config.owner_id.clone(),
store: components.db,
settings_store: components.settings_store,
llm: components.llm,
cheap_llm: components.cheap_llm,
safety: components.safety,
tools: components.tools,
workspace: components.workspace,
extension_manager: components.extension_manager,
skill_registry: components.skill_registry,
skill_catalog: components.skill_catalog,
skills_config: components.config.skills.clone(),
hooks: components.hooks,
auth_manager: None,
cost_guard: components.cost_guard,
sse_tx: None,
http_interceptor,
transcription: None,
document_extraction: None,
sandbox_readiness: ironclaw::agent::routine_engine::SandboxReadiness::DisabledByConfig,
builder: None,
llm_backend: "nearai".to_string(),
tenant_rates: std::sync::Arc::new(ironclaw::tenant::TenantRateRegistry::new(4, 3)),
};
// 7. Create TestChannel and ChannelManager.
// When testing bootstrap, the channel must be named "gateway" because
// the bootstrap greeting targets only the gateway channel. An explicit
// override (via `with_channel_name`) takes precedence so tests can
// mirror real-world channel naming for features keyed on the channel
// name (e.g. mission notifications routed back to the source channel).
//
// Channel user_id selection: align the channel user identity with the
// config's owner_id when one of the following is true:
// 1. The rig has live-seeded secrets — production credential
// lookups (`secrets WHERE user_id = ?`) must hit the rows we
// just inserted, not the hardcoded `"test-user"`.
// 2. Skills are enabled — engine v2 resolves the thread's project
// from the channel user_id; if the test user is not the owner,
// `resolve_user_project` creates a fresh per-user project with
// no skills migrated to it, and skill activation silently fails.
// For all other tests we keep the historical `"test-user"` default
// so existing tests don't change behaviour.
let channel_user_id = if seeded_secrets.is_some() || enable_skills {
components.config.owner_id.clone()
} else {
"test-user".to_string()
};
let test_channel = if let Some(ref name) = channel_name_override {
Arc::new(TestChannel::with_user_id(channel_user_id).with_name(name.clone()))
} else if keep_bootstrap {
Arc::new(TestChannel::with_user_id(channel_user_id).with_name("gateway"))
} else {
Arc::new(TestChannel::with_user_id(channel_user_id))
};
let handle = TestChannelHandle::new(Arc::clone(&test_channel));
let channel_manager = ChannelManager::new();
channel_manager.add(Box::new(handle)).await;
let channels = Arc::new(channel_manager);
// 7b. Register message tool so routines can send messages to channels.
deps.tools
.register_message_tools(Arc::clone(&channels), deps.extension_manager.clone())
.await;
// 8. Create Agent.
let routine_config = if enable_routines {
Some(ironclaw::config::RoutineConfig {
enabled: true,
cron_check_interval_secs: 60,
max_concurrent_routines: 3,
default_cooldown_secs: 300,
max_lightweight_tokens: 4096,
lightweight_tools_enabled: true,
lightweight_max_iterations: 3,
})
} else {
None
};
let agent = Agent::new(
components.config.agent.clone(),
deps,
channels,
None, // heartbeat_config
None, // hygiene_config
routine_config,
Some(Arc::clone(&components.context_manager)),
Some(Arc::clone(&session_manager_ref)),
);
// Match main.rs: fill the scheduler slot once Agent::new has created it.
*scheduler_slot.write().await = Some(agent.scheduler());
// 9. Spawn agent in background task.
let agent_handle = tokio::spawn(async move {
if let Err(e) = agent.run().await {
eprintln!("[TestRig] Agent exited with error: {e}");
}
});
// 10. Wait for the agent to call channel.start() (up to 5 seconds).
if let Some(rx) = test_channel.take_ready_rx().await {
let _ = tokio::time::timeout(Duration::from_secs(5), rx).await;
}
TestRig {
channel: test_channel,
instrumented_llm: instrumented,
start_time: Instant::now(),
max_tool_iterations,
agent_handle: Some(agent_handle),
db: db_ref,
workspace: workspace_ref,
trace_llm: trace_llm_ref,
extension_manager: ext_mgr_ref,
skill_registry: skill_registry_ref,
session_manager: session_manager_ref,
secrets_store: secrets_store_ref,
owner_id: owner_id_ref,
_temp_dir: temp_dir,
bootstrap_greetings_to_keep: if keep_bootstrap { 1 } else { 0 },
}
}
}
impl Default for TestRigBuilder {
fn default() -> Self {
Self::new()
}
}
impl TestRig {
/// Get the database handle for direct queries.
#[cfg(feature = "libsql")]
pub fn database(&self) -> &Arc<dyn Database> {
&self.db
}
/// Get the workspace handle for direct memory operations.
#[cfg(feature = "libsql")]
pub fn workspace(&self) -> Option<&Arc<ironclaw::workspace::Workspace>> {
self.workspace.as_ref()
}
/// Get the underlying TraceLlm for inspecting captured requests.
#[cfg(feature = "libsql")]
pub fn trace_llm(&self) -> Option<&Arc<TraceLlm>> {
self.trace_llm.as_ref()
}
/// Check if any captured status events contain safety/injection warnings.
pub fn has_safety_warnings(&self) -> bool {
self.captured_status_events().iter().any(|s| {
matches!(s, StatusUpdate::Status(msg) if msg.contains("sanitiz") || msg.contains("inject") || msg.contains("warning"))
})
}
}
// ---------------------------------------------------------------------------
// Convenience: run a recorded trace fixture end-to-end
// ---------------------------------------------------------------------------
/// Load a recorded trace fixture, build a rig, run and verify expects, then shut down.
///
/// `filename` is relative to `tests/fixtures/llm_traces/recorded/`.
#[cfg(feature = "libsql")]
pub async fn run_recorded_trace(filename: &str) {
let path = format!(
"{}/tests/fixtures/llm_traces/recorded/{filename}",
env!("CARGO_MANIFEST_DIR")
);
let trace = LlmTrace::from_file(&path)
.unwrap_or_else(|e| panic!("failed to load trace {filename}: {e}"));
let rig = TestRigBuilder::new()
.with_trace(trace.clone())
.build()
.await;
rig.run_and_verify_trace(&trace, Duration::from_secs(30))
.await;
rig.shutdown();
}
/// Like [`run_recorded_trace`] but routes through the engine v2 pipeline.
#[cfg(feature = "libsql")]
pub async fn run_recorded_trace_v2(filename: &str) {
let path = format!(
"{}/tests/fixtures/llm_traces/recorded/{filename}",
env!("CARGO_MANIFEST_DIR")
);
let trace = LlmTrace::from_file(&path)
.unwrap_or_else(|e| panic!("failed to load trace {filename}: {e}"));
let rig = TestRigBuilder::new()
.with_engine_v2()
.with_trace(trace.clone())
.build()
.await;
rig.run_and_verify_trace(&trace, Duration::from_secs(30))
.await;
rig.shutdown();
}