mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-03 08:06:01 +08:00
* feat(events): SkillActivated carries activation feedback notes
Add an optional `feedback: Vec<String>` field to the SkillActivated
event so the engine and selector can surface human-readable activation
notes (chain-load reasons, marker exclusions, scoring summaries) to the
UI. Wire the field through the StatusUpdate, the SSE bridge, and the
gateway's activity timeline; serialize-skip empty vectors so the wire
format stays backwards compatible.
* fix(skills): skill_install never prompts when skill is already loaded
When the LLM force-activates a persona via `/ceo-setup` it sometimes
follows up with a redundant `skill_install("ceo-setup")` call. The
`execute` path was already idempotent (returns `already_installed`
without touching the catalog), but `requires_approval` still gated
the call behind a confirmation prompt — pure friction on a guaranteed
no-op.
Mirror the idempotent shortcut in `requires_approval`: when a skill
with the requested name is already loaded (bundled, user, workspace,
or previously installed), return `ApprovalRequirement::Never`. The
shortcut wins even when `install_dependencies=true` because the
top-level execute is still a no-op (companions get reconciled by their
own activation paths). Regression test covers all three cases.
* fix(skills): preserve approval for dependency installs
* fix(events): include feedback in AppEvent::SkillActivated all-variants list
The variant-enumeration constructor in event.rs:501 was missed when
the new `feedback` field was added to AppEvent::SkillActivated, breaking
the build with E0063. All three Clippy CI jobs failed on this.
Regression: covered by `cargo build --all-features`, which fails to
compile if any variant in this list is constructed with missing fields.
* feat(skills): wire up v1 feedback producer for SkillActivated
The `SkillActivated` event carried an empty `feedback` field because
nothing populated it. This adds the producer end of the pipeline.
**Selector:**
- `prefilter_skills` now returns `SelectionOutcome { selected, notes }`.
- `try_select` returns a reason enum (`Selected`, `BudgetFull`,
`CandidateLimit`, `MarkerSatisfied`, `AlreadySelected`) so callers
can render distinct notes instead of opaque "skipped".
- Notes generated for:
- `<companion>: chain-loaded from <parent>`
- `<companion>: chain-load skipped (budget full)`
- `<companion>: chain-load skipped (max active skills reached)`
- `<companion>: chain-load skipped (setup already complete)`
- `<skill>: skipped (skill context budget exhausted)` for parents
that scored but didn't fit.
**Agent loop:**
- `select_active_skills` returns the notes alongside selected skills
and prepends a `<skill>: force-activated via /mention` note for each
explicit mention.
**Dispatcher:**
- Emits `StatusUpdate::SkillActivated { skill_names, feedback }` via
`channels.send_status` whenever something activated or notes exist
(so "nothing loaded because budget exhausted" surfaces too).
- Silent when nothing activated and no notes — no UI noise.
**Stale comment:**
- Router's v2-bridge comment no longer claims v1 callers populate
feedback "directly on `StatusUpdate`"; the v1 dispatcher now emits
its own event, and v2 remains empty until the Python orchestrator
is updated.
Regression: existing selector test `test_chain_load_respects_budget`,
`test_chain_load_skips_companion_with_satisfied_marker`, and
`test_chain_load_is_non_transitive` now also assert that the
corresponding note is in `outcome.notes`. The 42 selector tests and
503 agent-module tests all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Illia Polosukhin <ilblackdragon@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1507 lines
61 KiB
Rust
1507 lines
61 KiB
Rust
//! TestRig -- a builder for wiring a real Agent with a replay LLM and test channel.
|
|
//!
|
|
//! Constructs a full `Agent` with real tools but a `TraceLlm` (or custom LLM)
|
|
//! and a `TestChannel`, runs the agent in a background tokio task, and provides
|
|
//! methods to inject messages, wait for responses, and inspect tool calls.
|
|
|
|
#![allow(dead_code)] // Public API consumed by later test modules (Task 4+).
|
|
|
|
use std::sync::Arc;
|
|
use std::time::{Duration, Instant};
|
|
|
|
use ironclaw::agent::{Agent, AgentDeps};
|
|
use ironclaw::app::{AppBuilder, AppBuilderFlags};
|
|
use ironclaw::channels::web::log_layer::LogBroadcaster;
|
|
use ironclaw::channels::{OutgoingResponse, StatusUpdate};
|
|
use ironclaw::config::Config;
|
|
use ironclaw::db::Database;
|
|
use ironclaw::llm::{LlmProvider, SessionConfig, SessionManager};
|
|
use ironclaw::tools::Tool;
|
|
|
|
use crate::support::instrumented_llm::InstrumentedLlm;
|
|
use crate::support::metrics::{ToolInvocation, TraceMetrics};
|
|
use crate::support::test_channel::{CapturedEvent, TestChannel, TestChannelHandle};
|
|
use crate::support::trace_llm::{LlmTrace, TraceLlm};
|
|
|
|
use ironclaw::llm::recording::{HttpExchange, HttpInterceptor, ReplayingHttpInterceptor};
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// TestRig
|
|
// ---------------------------------------------------------------------------
|
|
|
|
/// Substring unique to the static bootstrap greeting (GREETING.md).
|
|
/// Used to transparently filter per-user bootstrap greetings from the
|
|
/// response stream so tests don't need to account for them manually.
|
|
const BOOTSTRAP_GREETING_MARKER: &str = "always-on chief of staff";
|
|
|
|
/// Configuration for selectively seeding `secrets` rows into a fresh test
|
|
/// rig database from an existing libSQL file.
|
|
///
|
|
/// Live tests use this to pull *only* the credentials they need (e.g. a
|
|
/// Google OAuth token) out of the developer's real `~/.ironclaw/ironclaw.db`
|
|
/// without cloning the rest of the database. Memory, history, secrets the
|
|
/// test didn't ask for — none of it crosses the boundary. The destination
|
|
/// DB starts empty, the listed secret rows are inserted under the test
|
|
/// rig's owner user, and the test must seed any other state itself.
|
|
#[derive(Clone, Debug)]
|
|
pub struct SeededSecretsConfig {
|
|
/// Path to the source libSQL file (typically `~/.ironclaw/ironclaw.db`).
|
|
pub source_path: std::path::PathBuf,
|
|
/// User ID to filter the source rows on (typically the developer's
|
|
/// owner_id from the live config).
|
|
pub source_user_id: String,
|
|
/// Names of the secrets to copy. Names not present in the source are
|
|
/// logged as warnings and silently skipped — the test will fail fast
|
|
/// on its own missing-credential path if a required name was wrong.
|
|
pub names: Vec<String>,
|
|
}
|
|
|
|
/// Open the source libSQL database read-only and copy the listed secret
|
|
/// rows into the destination database under `owner_user_id`. The source
|
|
/// is *only* read; the destination must already have the `secrets` table
|
|
/// (i.e. migrations have run on it).
|
|
///
|
|
/// This intentionally avoids `std::fs::copy` of the underlying SQLite
|
|
/// file: copying the whole DB would also pull in workspace memory,
|
|
/// conversation history, and *every other* secret row, which is the
|
|
/// regression we are explicitly fixing.
|
|
#[cfg(feature = "libsql")]
|
|
async fn seed_secrets_into(
|
|
dest_db: &libsql::Database,
|
|
config: &SeededSecretsConfig,
|
|
owner_user_id: &str,
|
|
) -> Result<(), String> {
|
|
if config.names.is_empty() {
|
|
return Ok(());
|
|
}
|
|
if !config.source_path.exists() {
|
|
return Err(format!(
|
|
"source libSQL DB not found: {}",
|
|
config.source_path.display()
|
|
));
|
|
}
|
|
eprintln!(
|
|
"[TestRig] Seeding {} secret(s) from {} (user_id={}) → temp DB (owner={})",
|
|
config.names.len(),
|
|
config.source_path.display(),
|
|
config.source_user_id,
|
|
owner_user_id,
|
|
);
|
|
|
|
// Open the source via a separate libSQL connection. We never write to
|
|
// it. The source process (the developer's running ironclaw) can keep
|
|
// running concurrently — libSQL's WAL mode permits a reader from
|
|
// another connection.
|
|
let src_db = libsql::Builder::new_local(&config.source_path)
|
|
.build()
|
|
.await
|
|
.map_err(|e| format!("open source libSQL DB: {e}"))?;
|
|
let src_conn = src_db
|
|
.connect()
|
|
.map_err(|e| format!("connect to source libSQL DB: {e}"))?;
|
|
|
|
// Build a parameterized IN-clause: `?, ?, ?...`. We bind names + the
|
|
// owner separately to keep this injection-safe even though the input
|
|
// is technically test-controlled.
|
|
let placeholders = std::iter::repeat_n("?", config.names.len())
|
|
.collect::<Vec<_>>()
|
|
.join(",");
|
|
let select_sql = format!(
|
|
"SELECT name, encrypted_value, key_salt, provider, expires_at \
|
|
FROM secrets \
|
|
WHERE user_id = ? AND name IN ({placeholders})"
|
|
);
|
|
let mut params: Vec<libsql::Value> = Vec::with_capacity(1 + config.names.len());
|
|
params.push(libsql::Value::Text(config.source_user_id.clone()));
|
|
for name in &config.names {
|
|
params.push(libsql::Value::Text(name.clone()));
|
|
}
|
|
|
|
let mut rows = src_conn
|
|
.query(&select_sql, params)
|
|
.await
|
|
.map_err(|e| format!("query source secrets: {e}"))?;
|
|
|
|
let dest_conn = dest_db
|
|
.connect()
|
|
.map_err(|e| format!("connect to destination libSQL DB: {e}"))?;
|
|
|
|
let mut copied: Vec<String> = Vec::new();
|
|
while let Some(row) = rows
|
|
.next()
|
|
.await
|
|
.map_err(|e| format!("iterate source secrets: {e}"))?
|
|
{
|
|
let name: String = row.get(0).map_err(|e| format!("read secrets.name: {e}"))?;
|
|
let encrypted_value: Vec<u8> = row
|
|
.get(1)
|
|
.map_err(|e| format!("read secrets.encrypted_value: {e}"))?;
|
|
let key_salt: Vec<u8> = row
|
|
.get(2)
|
|
.map_err(|e| format!("read secrets.key_salt: {e}"))?;
|
|
let provider: Option<String> = row
|
|
.get(3)
|
|
.map_err(|e| format!("read secrets.provider: {e}"))?;
|
|
let expires_at: Option<String> = row
|
|
.get(4)
|
|
.map_err(|e| format!("read secrets.expires_at: {e}"))?;
|
|
|
|
let id = uuid::Uuid::new_v4().to_string();
|
|
dest_conn
|
|
.execute(
|
|
"INSERT INTO secrets \
|
|
(id, user_id, name, encrypted_value, key_salt, provider, expires_at) \
|
|
VALUES (?, ?, ?, ?, ?, ?, ?)",
|
|
libsql::params![
|
|
id,
|
|
owner_user_id.to_string(),
|
|
name.clone(),
|
|
encrypted_value,
|
|
key_salt,
|
|
provider,
|
|
expires_at,
|
|
],
|
|
)
|
|
.await
|
|
.map_err(|e| format!("insert seeded secret '{name}' into destination: {e}"))?;
|
|
copied.push(name);
|
|
}
|
|
|
|
let missing: Vec<&String> = config
|
|
.names
|
|
.iter()
|
|
.filter(|n| !copied.contains(n))
|
|
.collect();
|
|
if !missing.is_empty() {
|
|
eprintln!(
|
|
"[TestRig] WARNING: requested secrets not found in source: {:?}",
|
|
missing
|
|
);
|
|
}
|
|
eprintln!("[TestRig] Seeded secrets: {:?}", copied);
|
|
Ok(())
|
|
}
|
|
|
|
/// A running test agent with methods to inject messages and inspect results.
|
|
pub struct TestRig {
|
|
/// The test channel for sending messages and reading captures.
|
|
channel: Arc<TestChannel>,
|
|
/// Instrumented LLM for collecting token/call metrics.
|
|
instrumented_llm: Arc<InstrumentedLlm>,
|
|
/// When the rig was created (for wall-time measurement).
|
|
start_time: Instant,
|
|
/// Maximum tool-call iterations per agentic loop (for count-based limit detection).
|
|
max_tool_iterations: usize,
|
|
/// Handle to the background agent task (wrapped in Option so Drop can take it).
|
|
agent_handle: Option<tokio::task::JoinHandle<()>>,
|
|
/// Database handle for direct queries in tests.
|
|
#[cfg(feature = "libsql")]
|
|
db: Arc<dyn Database>,
|
|
/// Workspace handle for direct memory operations in tests.
|
|
#[cfg(feature = "libsql")]
|
|
workspace: Option<Arc<ironclaw::workspace::Workspace>>,
|
|
/// The underlying TraceLlm for inspecting captured requests.
|
|
#[cfg(feature = "libsql")]
|
|
trace_llm: Option<Arc<TraceLlm>>,
|
|
/// Extension manager for direct extension operations in tests.
|
|
#[cfg(feature = "libsql")]
|
|
extension_manager: Option<Arc<ironclaw::extensions::ExtensionManager>>,
|
|
/// Skill registry (if skills are enabled) for direct inspection in tests.
|
|
#[cfg(feature = "libsql")]
|
|
skill_registry: Option<Arc<std::sync::RwLock<ironclaw_skills::SkillRegistry>>>,
|
|
/// Session manager for direct session/thread access in tests.
|
|
#[cfg(feature = "libsql")]
|
|
session_manager: Arc<ironclaw::agent::SessionManager>,
|
|
/// Secrets store for tests that need to read pre-seeded credentials
|
|
/// (e.g. live tests that issue direct REST calls to the same backend
|
|
/// the agent is talking to). Pulled from `AppComponents.secrets_store`
|
|
/// during build.
|
|
#[cfg(feature = "libsql")]
|
|
secrets_store: Option<Arc<dyn ironclaw::secrets::SecretsStore + Send + Sync>>,
|
|
/// Owner ID used by the rig — needed by `get_secret` to look up
|
|
/// per-user secret rows.
|
|
#[cfg(feature = "libsql")]
|
|
owner_id: String,
|
|
/// Temp directory guard -- keeps the libSQL database file alive.
|
|
#[cfg(feature = "libsql")]
|
|
_temp_dir: tempfile::TempDir,
|
|
/// How many bootstrap greetings to keep in `wait_for_responses`.
|
|
/// 0 for normal tests (filter all greetings), 1 for `.with_bootstrap()`
|
|
/// tests (keep the startup greeting, filter per-user duplicates).
|
|
#[cfg(feature = "libsql")]
|
|
bootstrap_greetings_to_keep: usize,
|
|
}
|
|
|
|
impl TestRig {
|
|
/// Inject a user message into the agent.
|
|
pub async fn send_message(&self, content: &str) {
|
|
self.channel.send_message(content).await;
|
|
}
|
|
|
|
/// Inject a raw `IncomingMessage` (for tests that need attachments, etc.).
|
|
pub async fn send_incoming(&self, msg: ironclaw::channels::IncomingMessage) {
|
|
self.channel.send_incoming(msg).await;
|
|
}
|
|
|
|
/// Return all message lists that were sent to the LLM provider.
|
|
///
|
|
/// Only available when the rig was built with a `TraceLlm` (i.e., via `.with_trace()`).
|
|
pub fn captured_llm_requests(&self) -> Vec<Vec<ironclaw::llm::ChatMessage>> {
|
|
self.trace_llm
|
|
.as_ref()
|
|
.map(|t| t.captured_requests())
|
|
.unwrap_or_default()
|
|
}
|
|
|
|
/// Return the extension manager for direct extension operations in tests.
|
|
pub fn extension_manager(&self) -> Option<&Arc<ironclaw::extensions::ExtensionManager>> {
|
|
self.extension_manager.as_ref()
|
|
}
|
|
|
|
/// Return the session manager for direct session/thread access in tests.
|
|
#[cfg(feature = "libsql")]
|
|
pub fn session_manager(&self) -> &Arc<ironclaw::agent::SessionManager> {
|
|
&self.session_manager
|
|
}
|
|
|
|
/// Read the decrypted value of a pre-seeded secret by name.
|
|
///
|
|
/// Returns `None` if the rig has no SecretsStore wired (non-libsql
|
|
/// configurations) or if the secret doesn't exist for this rig's
|
|
/// owner_id. Used by live tests that need to issue direct REST
|
|
/// calls to the same backend the agent is talking to (e.g. setting
|
|
/// up a real GitHub issue before the agent runs against it).
|
|
///
|
|
/// Note: this returns the secret in plaintext. Live tests should
|
|
/// only call this for credentials that were pre-seeded via
|
|
/// `with_secret` or `with_secrets`, never for arbitrary secrets the
|
|
/// rig may have inherited from a real DB.
|
|
#[cfg(feature = "libsql")]
|
|
pub async fn get_secret(&self, name: &str) -> Option<String> {
|
|
let store = self.secrets_store.as_ref()?;
|
|
match store.get_decrypted(&self.owner_id, name).await {
|
|
Ok(decrypted) => Some(decrypted.expose().to_string()),
|
|
Err(e) => {
|
|
// NotFound is expected for optional secrets — only log real errors
|
|
if !matches!(e, ironclaw::secrets::SecretError::NotFound(_)) {
|
|
eprintln!(
|
|
"[TestRig] get_secret('{name}') for owner '{}' failed: {e}",
|
|
self.owner_id
|
|
);
|
|
}
|
|
None
|
|
}
|
|
}
|
|
}
|
|
|
|
/// Get the secrets store for direct credential manipulation.
|
|
#[cfg(feature = "libsql")]
|
|
pub fn secrets_store(&self) -> Option<&Arc<dyn ironclaw::secrets::SecretsStore + Send + Sync>> {
|
|
self.secrets_store.as_ref()
|
|
}
|
|
|
|
/// The owner identity resolved from `Config::owner_id`.
|
|
#[cfg(feature = "libsql")]
|
|
pub fn owner_id(&self) -> &str {
|
|
&self.owner_id
|
|
}
|
|
|
|
/// Wait until at least `n` non-bootstrap responses have been captured, or
|
|
/// `timeout` elapses.
|
|
///
|
|
/// Per-user bootstrap greetings (fired when `tenant_ctx` creates a workspace
|
|
/// for a non-owner user) are transparently filtered from the response stream.
|
|
/// For `.with_bootstrap()` tests, the startup greeting is kept (1 allowed)
|
|
/// while additional per-user greetings are still filtered.
|
|
pub async fn wait_for_responses(&self, n: usize, timeout: Duration) -> Vec<OutgoingResponse> {
|
|
let deadline = tokio::time::Instant::now() + timeout;
|
|
let mut interval = Duration::from_millis(50);
|
|
let max_interval = Duration::from_millis(500);
|
|
loop {
|
|
let filtered = self.filter_responses(self.channel.captured_responses_async().await);
|
|
if filtered.len() >= n {
|
|
return filtered;
|
|
}
|
|
if tokio::time::Instant::now() >= deadline {
|
|
return filtered;
|
|
}
|
|
tokio::time::sleep(interval).await;
|
|
interval = (interval * 2).min(max_interval);
|
|
}
|
|
}
|
|
|
|
/// Filter bootstrap greetings from the response stream.
|
|
///
|
|
/// Keeps up to `bootstrap_greetings_to_keep` greeting responses (0 for
|
|
/// normal tests, 1 for `.with_bootstrap()` tests) and drops the rest.
|
|
fn filter_responses(&self, responses: Vec<OutgoingResponse>) -> Vec<OutgoingResponse> {
|
|
let mut greetings_kept = 0usize;
|
|
responses
|
|
.into_iter()
|
|
.filter(|r| {
|
|
if r.content.contains(BOOTSTRAP_GREETING_MARKER) {
|
|
greetings_kept += 1;
|
|
greetings_kept <= self.bootstrap_greetings_to_keep
|
|
} else {
|
|
true
|
|
}
|
|
})
|
|
.collect()
|
|
}
|
|
|
|
/// Return the names of all `ToolStarted` events captured so far.
|
|
pub fn tool_calls_started(&self) -> Vec<String> {
|
|
self.channel.tool_calls_started()
|
|
}
|
|
|
|
/// Return the filtered list of captured responses so far.
|
|
///
|
|
/// Mirrors the bootstrap-greeting filtering used by `wait_for_responses`.
|
|
pub async fn captured_responses(&self) -> Vec<OutgoingResponse> {
|
|
self.filter_responses(self.channel.captured_responses_async().await)
|
|
}
|
|
|
|
/// Return `(name, success)` for all `ToolCompleted` events captured so far.
|
|
pub fn tool_calls_completed(&self) -> Vec<(String, bool)> {
|
|
self.channel.tool_calls_completed()
|
|
}
|
|
|
|
/// Return `(name, preview)` for all `ToolResult` events captured so far.
|
|
pub fn tool_results(&self) -> Vec<(String, String)> {
|
|
self.channel.tool_results()
|
|
}
|
|
|
|
/// Return `(name, duration_ms)` for all completed tools with timing data.
|
|
pub fn tool_timings(&self) -> Vec<(String, u64)> {
|
|
self.channel.tool_timings()
|
|
}
|
|
|
|
/// Wait until a `Status("Done")` event has been captured, or `timeout` elapses.
|
|
pub async fn wait_for_done(&self, timeout: Duration) -> bool {
|
|
self.channel.wait_for_done(timeout).await
|
|
}
|
|
|
|
/// Return a snapshot of all captured status events.
|
|
pub fn captured_status_events(&self) -> Vec<StatusUpdate> {
|
|
self.channel.captured_status_events()
|
|
}
|
|
|
|
/// Return the names of skills loaded into the registry, if skills are
|
|
/// enabled. Useful for verifying the registry discovered the SKILL.md
|
|
/// files from `with_skills_dir()`.
|
|
pub fn loaded_skill_names(&self) -> Vec<String> {
|
|
self.skill_registry
|
|
.as_ref()
|
|
.and_then(|r| {
|
|
r.read()
|
|
.ok()
|
|
.map(|g| g.skills().iter().map(|s| s.name().to_string()).collect())
|
|
})
|
|
.unwrap_or_default()
|
|
}
|
|
|
|
/// Return the names of skills that were activated during this session,
|
|
/// extracted from `SkillActivated` status events.
|
|
pub fn active_skill_names(&self) -> Vec<String> {
|
|
self.captured_status_events()
|
|
.iter()
|
|
.filter_map(|event| match event {
|
|
StatusUpdate::SkillActivated { skill_names, .. } => Some(skill_names.clone()),
|
|
_ => None,
|
|
})
|
|
.flatten()
|
|
.collect()
|
|
}
|
|
|
|
/// Return the ordered log of captured outbound events.
|
|
pub fn captured_events(&self) -> Vec<CapturedEvent> {
|
|
self.channel.captured_events()
|
|
}
|
|
|
|
/// Clear all captured responses and status events.
|
|
pub async fn clear(&self) {
|
|
self.channel.clear().await;
|
|
}
|
|
|
|
/// Number of LLM calls made so far.
|
|
pub fn llm_call_count(&self) -> u32 {
|
|
self.instrumented_llm.call_count()
|
|
}
|
|
|
|
/// Total input tokens across all LLM calls.
|
|
pub fn total_input_tokens(&self) -> u32 {
|
|
self.instrumented_llm.total_input_tokens()
|
|
}
|
|
|
|
/// Total output tokens across all LLM calls.
|
|
pub fn total_output_tokens(&self) -> u32 {
|
|
self.instrumented_llm.total_output_tokens()
|
|
}
|
|
|
|
/// Estimated total cost in USD.
|
|
pub fn estimated_cost_usd(&self) -> f64 {
|
|
self.instrumented_llm.estimated_cost_usd()
|
|
}
|
|
|
|
/// Wall-clock time since rig creation.
|
|
pub fn elapsed_ms(&self) -> u64 {
|
|
self.start_time.elapsed().as_millis() as u64
|
|
}
|
|
|
|
/// Collect a complete `TraceMetrics` snapshot from all captured data.
|
|
///
|
|
/// Call this after `wait_for_responses()` to get the full metrics for the
|
|
/// scenario. The `turns` count is based on the number of captured responses.
|
|
pub async fn collect_metrics(&self) -> TraceMetrics {
|
|
let completed = self.tool_calls_completed();
|
|
|
|
// Build ToolInvocation records from ToolStarted/ToolCompleted pairs,
|
|
// matching each completion with its captured timing data.
|
|
let timings = self.tool_timings();
|
|
let mut timing_iter_by_name: std::collections::HashMap<&str, Vec<u64>> =
|
|
std::collections::HashMap::new();
|
|
for (name, ms) in &timings {
|
|
timing_iter_by_name
|
|
.entry(name.as_str())
|
|
.or_default()
|
|
.push(*ms);
|
|
}
|
|
|
|
let tool_invocations: Vec<ToolInvocation> = completed
|
|
.iter()
|
|
.map(|(name, success)| {
|
|
let duration_ms = timing_iter_by_name
|
|
.get_mut(name.as_str())
|
|
.and_then(|v| {
|
|
if v.is_empty() {
|
|
None
|
|
} else {
|
|
Some(v.remove(0))
|
|
}
|
|
})
|
|
.unwrap_or(0);
|
|
ToolInvocation {
|
|
name: name.clone(),
|
|
duration_ms,
|
|
success: *success,
|
|
}
|
|
})
|
|
.collect();
|
|
|
|
// Detect if iteration limit was hit by comparing completed tool-call count
|
|
// against the configured max_tool_iterations threshold.
|
|
let hit_iteration_limit = completed.len() >= self.max_tool_iterations;
|
|
|
|
// Count turns as the number of captured responses.
|
|
let responses = self.channel.captured_responses();
|
|
let turns = responses.len() as u32;
|
|
|
|
TraceMetrics {
|
|
wall_time_ms: self.elapsed_ms(),
|
|
llm_calls: self.instrumented_llm.call_count(),
|
|
input_tokens: self.instrumented_llm.total_input_tokens(),
|
|
output_tokens: self.instrumented_llm.total_output_tokens(),
|
|
estimated_cost_usd: self.instrumented_llm.estimated_cost_usd(),
|
|
tool_calls: tool_invocations,
|
|
turns,
|
|
hit_iteration_limit,
|
|
hit_timeout: false, // Caller can set this based on wait_for_responses result.
|
|
}
|
|
}
|
|
|
|
/// Run a complete multi-turn trace, injecting user messages from the trace
|
|
/// and waiting for responses after each turn.
|
|
///
|
|
/// Returns a `Vec` of response lists, one per turn. Status events and tool
|
|
/// call data accumulate across all turns (no clearing between turns), so
|
|
/// post-run assertions like `tool_calls_started()` reflect the whole trace.
|
|
pub async fn run_trace(
|
|
&self,
|
|
trace: &LlmTrace,
|
|
timeout: Duration,
|
|
) -> Vec<Vec<OutgoingResponse>> {
|
|
let mut all_responses: Vec<Vec<OutgoingResponse>> = Vec::new();
|
|
let mut total_responses = 0usize;
|
|
for turn in &trace.turns {
|
|
self.send_message(&turn.user_input).await;
|
|
let responses = self.wait_for_responses(total_responses + 1, timeout).await;
|
|
// Extract only the new responses from this turn.
|
|
let turn_responses: Vec<OutgoingResponse> =
|
|
responses.into_iter().skip(total_responses).collect();
|
|
total_responses += turn_responses.len();
|
|
all_responses.push(turn_responses);
|
|
}
|
|
all_responses
|
|
}
|
|
|
|
/// Run a trace, then verify all declarative `expects` (top-level and per-turn).
|
|
///
|
|
/// Returns the per-turn response lists for additional manual assertions.
|
|
pub async fn run_and_verify_trace(
|
|
&self,
|
|
trace: &LlmTrace,
|
|
timeout: Duration,
|
|
) -> Vec<Vec<OutgoingResponse>> {
|
|
use crate::support::assertions::verify_expects;
|
|
|
|
let all_responses = self.run_trace(trace, timeout).await;
|
|
|
|
// Verify top-level expects against all accumulated data.
|
|
if !trace.expects.is_empty() {
|
|
let all_response_strings: Vec<String> = all_responses
|
|
.iter()
|
|
.flat_map(|turn| turn.iter().map(|r| r.content.clone()))
|
|
.collect();
|
|
let started = self.tool_calls_started();
|
|
let completed = self.tool_calls_completed();
|
|
let mut results = self.tool_results();
|
|
for status in self.channel.captured_status_events() {
|
|
if let ironclaw::channels::StatusUpdate::ToolCompleted {
|
|
name,
|
|
success: false,
|
|
error,
|
|
parameters,
|
|
..
|
|
} = status
|
|
{
|
|
let detail = format!(
|
|
"error={}; params={}",
|
|
error.unwrap_or_else(|| "unknown".to_string()),
|
|
parameters.unwrap_or_else(|| "{}".to_string())
|
|
);
|
|
results.push((name, detail));
|
|
}
|
|
}
|
|
verify_expects(
|
|
&trace.expects,
|
|
&all_response_strings,
|
|
&started,
|
|
&completed,
|
|
&results,
|
|
"top-level",
|
|
);
|
|
}
|
|
|
|
all_responses
|
|
}
|
|
|
|
/// Verify top-level `expects` from a trace against already-captured data.
|
|
///
|
|
/// Call this after `send_message()` + `wait_for_responses()` for flat-format
|
|
/// traces. For multi-turn traces, use `run_and_verify_trace()` instead.
|
|
pub fn verify_trace_expects(&self, trace: &LlmTrace, responses: &[OutgoingResponse]) {
|
|
use crate::support::assertions::verify_expects;
|
|
|
|
if trace.expects.is_empty() {
|
|
return;
|
|
}
|
|
let response_strings: Vec<String> = responses.iter().map(|r| r.content.clone()).collect();
|
|
let started = self.tool_calls_started();
|
|
let completed = self.tool_calls_completed();
|
|
let mut results = self.tool_results();
|
|
for status in self.channel.captured_status_events() {
|
|
if let ironclaw::channels::StatusUpdate::ToolCompleted {
|
|
name,
|
|
success: false,
|
|
error,
|
|
parameters,
|
|
..
|
|
} = status
|
|
{
|
|
let detail = format!(
|
|
"error={}; params={}",
|
|
error.unwrap_or_else(|| "unknown".to_string()),
|
|
parameters.unwrap_or_else(|| "{}".to_string())
|
|
);
|
|
results.push((name, detail));
|
|
}
|
|
}
|
|
verify_expects(
|
|
&trace.expects,
|
|
&response_strings,
|
|
&started,
|
|
&completed,
|
|
&results,
|
|
"top-level",
|
|
);
|
|
}
|
|
|
|
/// Signal the channel to shut down and abort the background agent task.
|
|
pub fn shutdown(mut self) {
|
|
self.channel.signal_shutdown();
|
|
if let Some(handle) = self.agent_handle.take() {
|
|
handle.abort();
|
|
}
|
|
}
|
|
}
|
|
|
|
impl Drop for TestRig {
|
|
fn drop(&mut self) {
|
|
if let Some(handle) = self.agent_handle.take()
|
|
&& !handle.is_finished()
|
|
{
|
|
handle.abort();
|
|
}
|
|
}
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// TestRigBuilder
|
|
// ---------------------------------------------------------------------------
|
|
|
|
/// Specification for loading a real WASM tool in the test rig.
|
|
pub struct WasmToolSpec {
|
|
pub name: String,
|
|
pub wasm_path: std::path::PathBuf,
|
|
pub capabilities_path: Option<std::path::PathBuf>,
|
|
}
|
|
|
|
/// Builder for constructing a `TestRig`.
|
|
pub struct TestRigBuilder {
|
|
trace: Option<LlmTrace>,
|
|
llm: Option<Arc<dyn LlmProvider>>,
|
|
config_override: Option<Config>,
|
|
max_tool_iterations: usize,
|
|
injection_check: bool,
|
|
auto_approve_tools: Option<bool>,
|
|
enable_skills: bool,
|
|
skills_dir: Option<std::path::PathBuf>,
|
|
enable_routines: bool,
|
|
http_exchanges: Vec<HttpExchange>,
|
|
http_interceptor_override: Option<Arc<dyn HttpInterceptor>>,
|
|
extra_tools: Vec<Arc<dyn Tool>>,
|
|
wasm_tools: Vec<WasmToolSpec>,
|
|
keep_bootstrap: bool,
|
|
engine_v2: bool,
|
|
channel_name_override: Option<String>,
|
|
seeded_secrets: Option<SeededSecretsConfig>,
|
|
/// Pre-seed the SecretsStore with `(name, value)` pairs before the
|
|
/// agent starts. Used by live tests that need a credential to *exist*
|
|
/// (so the kernel pre-flight auth gate stays out of the way) but
|
|
/// don't actually call the credentialed API.
|
|
pre_seed_secrets: Vec<(String, String)>,
|
|
}
|
|
|
|
impl TestRigBuilder {
|
|
/// Create a new builder with defaults.
|
|
pub fn new() -> Self {
|
|
Self {
|
|
trace: None,
|
|
llm: None,
|
|
config_override: None,
|
|
max_tool_iterations: 10,
|
|
injection_check: false,
|
|
auto_approve_tools: Some(true),
|
|
enable_skills: false,
|
|
skills_dir: None,
|
|
enable_routines: false,
|
|
http_exchanges: Vec::new(),
|
|
http_interceptor_override: None,
|
|
extra_tools: Vec::new(),
|
|
wasm_tools: Vec::new(),
|
|
keep_bootstrap: false,
|
|
engine_v2: false,
|
|
channel_name_override: None,
|
|
seeded_secrets: None,
|
|
pre_seed_secrets: Vec::new(),
|
|
}
|
|
}
|
|
|
|
/// Pre-seed a secret in the SecretsStore before the agent starts.
|
|
///
|
|
/// This is for tests that need a credential to *exist* so the
|
|
/// kernel-level pre-flight auth gate (which fires when a skill with
|
|
/// a credential spec activates) doesn't block the conversation. The
|
|
/// value can be any non-empty string — the test isn't actually
|
|
/// hitting the credentialed API, the credential just needs to be
|
|
/// present in the store under the test's owner_id.
|
|
///
|
|
/// Note: only takes effect when the rig has a working `SecretsStore`
|
|
/// (i.e., the libSQL backend with `with_database_and_handles()`,
|
|
/// which is the standard rig setup).
|
|
pub fn with_secret(mut self, name: impl Into<String>, value: impl Into<String>) -> Self {
|
|
self.pre_seed_secrets.push((name.into(), value.into()));
|
|
self
|
|
}
|
|
|
|
/// Override the test channel name (default: "test", or "gateway" when
|
|
/// `.with_bootstrap()` is set). Use this when you need the channel name to
|
|
/// match a real-world channel (e.g. "gateway") so that downstream features
|
|
/// keyed on the channel name (assistant conversations, mission notify
|
|
/// channels) behave the same as in production.
|
|
pub fn with_channel_name(mut self, name: impl Into<String>) -> Self {
|
|
self.channel_name_override = Some(name.into());
|
|
self
|
|
}
|
|
|
|
/// Selectively seed `secrets` rows from an existing libSQL file into
|
|
/// the test rig's fresh temp database.
|
|
///
|
|
/// Live tests use this to pull just the credentials they need (e.g.
|
|
/// `google_oauth_token`) out of the developer's real
|
|
/// `~/.ironclaw/ironclaw.db` so OAuth-backed flows work end-to-end —
|
|
/// without cloning conversation history, workspace memory, or any
|
|
/// secret the test didn't ask for. The destination DB starts empty;
|
|
/// the listed rows are inserted under the test rig's owner user; any
|
|
/// other state the test relies on must be seeded by the test itself.
|
|
///
|
|
/// Names that don't exist in the source are logged as warnings and
|
|
/// silently skipped — the test will fail fast on its own missing
|
|
/// credential path if a required name was wrong.
|
|
pub fn with_seeded_secrets(
|
|
mut self,
|
|
source_path: std::path::PathBuf,
|
|
source_user_id: impl Into<String>,
|
|
names: impl IntoIterator<Item = impl Into<String>>,
|
|
) -> Self {
|
|
self.seeded_secrets = Some(SeededSecretsConfig {
|
|
source_path,
|
|
source_user_id: source_user_id.into(),
|
|
names: names.into_iter().map(Into::into).collect(),
|
|
});
|
|
self
|
|
}
|
|
|
|
/// Load a real WASM tool binary into the test rig.
|
|
///
|
|
/// The tool will be compiled, registered, and wired with the same HTTP
|
|
/// interceptor used for `with_http_exchanges()`, so `http_exchanges` in
|
|
/// the trace can specify expected requests/responses for WASM tool HTTP calls.
|
|
///
|
|
/// If the WASM binary does not exist at build time, the tool is silently
|
|
/// skipped (logged as a warning). Tests should use `#[ignore]` or check
|
|
/// for the binary in a preamble if the tool is required.
|
|
pub fn with_wasm_tool(
|
|
mut self,
|
|
name: impl Into<String>,
|
|
wasm_path: impl Into<std::path::PathBuf>,
|
|
capabilities_path: Option<std::path::PathBuf>,
|
|
) -> Self {
|
|
self.wasm_tools.push(WasmToolSpec {
|
|
name: name.into(),
|
|
wasm_path: wasm_path.into(),
|
|
capabilities_path,
|
|
});
|
|
self
|
|
}
|
|
|
|
/// Set the LLM trace to replay.
|
|
pub fn with_trace(mut self, trace: LlmTrace) -> Self {
|
|
self.trace = Some(trace);
|
|
self
|
|
}
|
|
|
|
/// Override the LLM provider directly (takes precedence over trace).
|
|
pub fn with_llm(mut self, llm: Arc<dyn LlmProvider>) -> Self {
|
|
self.llm = Some(llm);
|
|
self
|
|
}
|
|
|
|
/// Override the Config to mirror real binary behavior.
|
|
///
|
|
/// When set, uses this config instead of `Config::for_testing()`.
|
|
/// The database path is still overridden to use a temp libSQL file,
|
|
/// but agent settings (`allow_local_tools`, `engine_v2`, etc.) are
|
|
/// preserved from the provided config. Post-build forcing of
|
|
/// `allow_local_tools = true` is skipped so the test matches the
|
|
/// real binary's tool availability.
|
|
pub fn with_config(mut self, config: Config) -> Self {
|
|
self.config_override = Some(config);
|
|
self
|
|
}
|
|
|
|
/// Set the maximum number of tool iterations per agentic loop invocation.
|
|
pub fn with_max_tool_iterations(mut self, n: usize) -> Self {
|
|
self.max_tool_iterations = n;
|
|
self
|
|
}
|
|
|
|
/// Register additional custom tools (e.g. stub tools for testing).
|
|
pub fn with_extra_tools(mut self, tools: Vec<Arc<dyn Tool>>) -> Self {
|
|
self.extra_tools = tools;
|
|
self
|
|
}
|
|
|
|
/// Enable prompt injection detection in the safety layer.
|
|
///
|
|
/// When enabled, tool outputs are scanned for injection patterns
|
|
/// (e.g., "ignore previous instructions", special tokens like `<|endoftext|>`)
|
|
/// and critical patterns are escaped before reaching the LLM.
|
|
pub fn with_injection_check(mut self, enable: bool) -> Self {
|
|
self.injection_check = enable;
|
|
self
|
|
}
|
|
|
|
/// Override agent-level automatic approval of `UnlessAutoApproved` tools.
|
|
pub fn with_auto_approve_tools(mut self, enable: bool) -> Self {
|
|
self.auto_approve_tools = Some(enable);
|
|
self
|
|
}
|
|
|
|
/// Enable skill discovery and registration for this test rig.
|
|
pub fn with_skills(mut self) -> Self {
|
|
self.enable_skills = true;
|
|
self
|
|
}
|
|
|
|
/// Set a custom skills directory so the test rig loads skill files
|
|
/// from a real path (e.g. the repo's `skills/` directory) instead of
|
|
/// an empty temp directory. Implies `with_skills()`.
|
|
pub fn with_skills_dir(mut self, dir: std::path::PathBuf) -> Self {
|
|
self.enable_skills = true;
|
|
self.skills_dir = Some(dir);
|
|
self
|
|
}
|
|
|
|
/// Enable the routines system so the scheduler is wired with a `RoutineEngine`,
|
|
/// allowing routine jobs to actually execute. Routine tools are always registered
|
|
/// but require the engine to dispatch jobs.
|
|
pub fn with_routines(mut self) -> Self {
|
|
self.enable_routines = true;
|
|
self
|
|
}
|
|
|
|
/// Keep `bootstrap_pending` so the proactive greeting fires on startup.
|
|
pub fn with_bootstrap(mut self) -> Self {
|
|
self.keep_bootstrap = true;
|
|
self
|
|
}
|
|
|
|
/// Route messages through the engine v2 pipeline instead of the v1 agentic loop.
|
|
pub fn with_engine_v2(mut self) -> Self {
|
|
self.engine_v2 = true;
|
|
self
|
|
}
|
|
|
|
/// Add pre-recorded HTTP exchanges for the `ReplayingHttpInterceptor`.
|
|
///
|
|
/// When set, all `http` tool calls will return these responses in order
|
|
/// instead of making real network requests.
|
|
pub fn with_http_exchanges(mut self, exchanges: Vec<HttpExchange>) -> Self {
|
|
self.http_exchanges = exchanges;
|
|
self
|
|
}
|
|
|
|
/// Override the HTTP interceptor directly.
|
|
///
|
|
/// When set, this interceptor is used instead of constructing a
|
|
/// `ReplayingHttpInterceptor` from trace http_exchanges or
|
|
/// `with_http_exchanges()`. Useful for live-mode recording where a
|
|
/// `RecordingHttpInterceptor` captures real HTTP traffic.
|
|
pub fn with_http_interceptor(mut self, interceptor: Arc<dyn HttpInterceptor>) -> Self {
|
|
self.http_interceptor_override = Some(interceptor);
|
|
self
|
|
}
|
|
|
|
/// Build the test rig, creating a real agent and spawning it in the background.
|
|
///
|
|
/// Uses `AppBuilder::build_all()` to get the same component set as the real
|
|
/// binary, with only the LLM swapped for TraceLlm.
|
|
///
|
|
/// Requires the `libsql` feature for the embedded test database.
|
|
#[cfg(feature = "libsql")]
|
|
pub async fn build(self) -> TestRig {
|
|
use ironclaw::channels::ChannelManager;
|
|
use ironclaw::db::libsql::LibSqlBackend;
|
|
|
|
// Destructure self up front to avoid partial-move issues.
|
|
let TestRigBuilder {
|
|
trace,
|
|
llm,
|
|
config_override,
|
|
max_tool_iterations,
|
|
injection_check,
|
|
auto_approve_tools,
|
|
enable_skills,
|
|
skills_dir,
|
|
enable_routines,
|
|
http_exchanges: explicit_http_exchanges,
|
|
http_interceptor_override,
|
|
extra_tools,
|
|
wasm_tools,
|
|
keep_bootstrap,
|
|
engine_v2,
|
|
channel_name_override,
|
|
seeded_secrets,
|
|
pre_seed_secrets,
|
|
} = self;
|
|
|
|
// 1. Create temp dir + fresh libSQL database + run migrations.
|
|
//
|
|
// The destination DB is always created empty. If the test asked
|
|
// for specific secrets via `with_seeded_secrets(...)`, we copy
|
|
// *only* those rows out of the source DB after migrations have
|
|
// run — never the whole file. Memory, history, conversations,
|
|
// and unrequested secrets stay isolated in the source.
|
|
let temp_dir = tempfile::tempdir().expect("failed to create temp dir");
|
|
let db_path = temp_dir.path().join("test_rig.db");
|
|
|
|
let backend = LibSqlBackend::new_local(&db_path)
|
|
.await
|
|
.expect("failed to create test LibSqlBackend");
|
|
backend
|
|
.run_migrations()
|
|
.await
|
|
.expect("failed to run migrations");
|
|
|
|
// Build the backend-specific handles so AppBuilder can wire the
|
|
// secrets store. `with_database()` alone leaves handles=None,
|
|
// which silently disables `SecretsStore` and breaks every test
|
|
// that needs OAuth/encrypted credentials. `with_database_and_handles()`
|
|
// is the right pairing.
|
|
let db_handles = ironclaw::db::DatabaseHandles {
|
|
#[cfg(feature = "libsql")]
|
|
libsql_db: Some(backend.shared_db()),
|
|
#[cfg(feature = "postgres")]
|
|
pg_pool: None,
|
|
};
|
|
let db: Arc<dyn ironclaw::db::Database> = Arc::new(backend);
|
|
|
|
// 2. Build Config.
|
|
let has_config_override = config_override.is_some();
|
|
let has_skills_dir_override = skills_dir.is_some();
|
|
let skills_dir = skills_dir.unwrap_or_else(|| temp_dir.path().join("skills"));
|
|
let installed_skills_dir = temp_dir.path().join("installed_skills");
|
|
// Only create the tempdir skills dir if we're using it (i.e. no override).
|
|
// Do not try to create the override path — callers are responsible for
|
|
// providing an existing directory.
|
|
if !has_skills_dir_override {
|
|
let _ = std::fs::create_dir_all(&skills_dir);
|
|
}
|
|
let _ = std::fs::create_dir_all(&installed_skills_dir);
|
|
let mut config = if let Some(mut cfg) = config_override {
|
|
// Override database to use temp libSQL, but preserve agent/llm settings.
|
|
cfg.database.backend = ironclaw::config::DatabaseBackend::LibSql;
|
|
cfg.database.libsql_path = Some(db_path);
|
|
cfg.skills.local_dir = skills_dir.clone();
|
|
cfg.skills.installed_dir = installed_skills_dir.clone();
|
|
cfg
|
|
} else {
|
|
Config::for_testing(db_path, skills_dir.clone(), installed_skills_dir.clone())
|
|
};
|
|
config.agent.max_tool_iterations = max_tool_iterations;
|
|
config.safety.injection_check_enabled = injection_check;
|
|
config.skills.enabled = enable_skills;
|
|
if let Some(v) = auto_approve_tools {
|
|
config.agent.auto_approve_tools = v;
|
|
}
|
|
|
|
// 2b. Selectively seed `secrets` rows from the source DB if the
|
|
// test asked for it. We seed *after* migrations have run (so the
|
|
// schema exists in the destination) and *under the owner_user_id
|
|
// from the active config* so production credential lookups
|
|
// (`SELECT ... WHERE user_id = owner`) hit the seeded rows. The
|
|
// source DB is opened read-only via a separate libSQL connection;
|
|
// nothing else (memory, conversations, other secrets) crosses the
|
|
// boundary.
|
|
#[cfg(feature = "libsql")]
|
|
if let Some(ref ss) = seeded_secrets {
|
|
let owner = config.owner_id.clone();
|
|
let dest_handle = db_handles
|
|
.libsql_db
|
|
.as_ref()
|
|
.expect("libsql backend handle is required to seed secrets");
|
|
seed_secrets_into(dest_handle.as_ref(), ss, &owner)
|
|
.await
|
|
.expect("failed to seed live-test secrets into temp DB");
|
|
}
|
|
|
|
// 3. Create SessionManager + LogBroadcaster.
|
|
let session = Arc::new(SessionManager::new(SessionConfig::default()));
|
|
let log_broadcaster = Arc::new(LogBroadcaster::new());
|
|
|
|
// 4. Create TraceLlm + InstrumentedLlm, extract HTTP exchanges for replay.
|
|
let trace_http_exchanges = trace
|
|
.as_ref()
|
|
.map(|t| t.http_exchanges.clone())
|
|
.unwrap_or_default();
|
|
|
|
let mut trace_llm_ref: Option<Arc<TraceLlm>> = None;
|
|
let base_llm: Arc<dyn LlmProvider> = if let Some(llm) = llm {
|
|
llm
|
|
} else if let Some(trace) = trace {
|
|
let tlm = Arc::new(TraceLlm::from_trace(trace));
|
|
trace_llm_ref = Some(Arc::clone(&tlm));
|
|
tlm
|
|
} else {
|
|
let trace = LlmTrace::single_turn(
|
|
"test-rig-default",
|
|
"(default)",
|
|
vec![crate::support::trace_llm::TraceStep {
|
|
request_hint: None,
|
|
response: crate::support::trace_llm::TraceResponse::Text {
|
|
content: "Hello from test rig!".to_string(),
|
|
input_tokens: 10,
|
|
output_tokens: 5,
|
|
},
|
|
expected_tool_results: Vec::new(),
|
|
}],
|
|
);
|
|
let tlm = Arc::new(TraceLlm::from_trace(trace));
|
|
trace_llm_ref = Some(Arc::clone(&tlm));
|
|
tlm
|
|
};
|
|
let instrumented = Arc::new(InstrumentedLlm::new(base_llm));
|
|
let llm: Arc<dyn LlmProvider> = Arc::clone(&instrumented) as Arc<dyn LlmProvider>;
|
|
|
|
// 5. Build AppComponents via AppBuilder with injected DB and LLM.
|
|
let mut builder = AppBuilder::new(
|
|
config,
|
|
AppBuilderFlags::default(),
|
|
None,
|
|
session,
|
|
log_broadcaster,
|
|
);
|
|
builder.with_database_and_handles(Arc::clone(&db), db_handles);
|
|
builder.with_llm(llm);
|
|
let mut components = builder
|
|
.build_all()
|
|
.await
|
|
.expect("AppBuilder::build_all() failed in test rig");
|
|
|
|
// Clear the *owner* workspace bootstrap flag so tests don't get an
|
|
// unexpected proactive greeting on startup (unless the test explicitly
|
|
// wants to test the bootstrap flow via `.with_bootstrap()`).
|
|
//
|
|
// Per-user bootstrap greetings (fired when `tenant_ctx` creates a
|
|
// workspace for a non-owner user like "test-user") are allowed to
|
|
// happen naturally. They are transparently filtered from the response
|
|
// stream by `wait_for_responses` so tests don't need to account for
|
|
// them in response counting.
|
|
if !keep_bootstrap && let Some(ref ws) = components.workspace {
|
|
ws.take_bootstrap_pending();
|
|
}
|
|
|
|
// AppBuilder may re-resolve config from env/TOML and override test defaults.
|
|
// When a config override was provided, preserve its agent settings to mirror
|
|
// the real binary. Otherwise force deterministic test defaults.
|
|
if has_config_override {
|
|
if let Some(v) = auto_approve_tools {
|
|
components.config.agent.auto_approve_tools = v;
|
|
}
|
|
// allow_local_tools comes from the provided config.
|
|
// engine_v2: honour the builder's explicit override if set.
|
|
if engine_v2 {
|
|
components.config.agent.engine_v2 = true;
|
|
}
|
|
} else {
|
|
components.config.agent.auto_approve_tools = auto_approve_tools.unwrap_or(true);
|
|
components.config.agent.allow_local_tools = true;
|
|
components.config.agent.engine_v2 = engine_v2;
|
|
}
|
|
|
|
// Reset engine v2 global state so each test gets a clean engine instance.
|
|
if components.config.agent.engine_v2 {
|
|
ironclaw::bridge::reset_engine_state().await;
|
|
}
|
|
|
|
let scheduler_slot: ironclaw::tools::builtin::SchedulerSlot =
|
|
Arc::new(tokio::sync::RwLock::new(None));
|
|
|
|
// Build HTTP interceptor once — shared by both AgentDeps and WASM tools.
|
|
// Direct override takes priority (e.g. RecordingHttpInterceptor for live tests).
|
|
let http_interceptor: Option<Arc<dyn HttpInterceptor>> = if let Some(override_interceptor) =
|
|
http_interceptor_override
|
|
{
|
|
Some(override_interceptor)
|
|
} else {
|
|
let exchanges = if explicit_http_exchanges.is_empty() {
|
|
trace_http_exchanges
|
|
} else {
|
|
explicit_http_exchanges
|
|
};
|
|
if exchanges.is_empty() {
|
|
None
|
|
} else {
|
|
Some(Arc::new(ReplayingHttpInterceptor::new(exchanges)) as Arc<dyn HttpInterceptor>)
|
|
}
|
|
};
|
|
|
|
// 6. Register job tools, routine tools, and extra tools.
|
|
{
|
|
// Register filesystem/shell dev tools. When using a config override
|
|
// (real-binary parity mode), respect the allow_local_tools flag.
|
|
// Otherwise always register them for test convenience.
|
|
if !has_config_override || components.config.agent.allow_local_tools {
|
|
components.tools.register_dev_tools();
|
|
}
|
|
|
|
components.tools.register_job_tools(
|
|
Arc::clone(&components.context_manager),
|
|
Some(scheduler_slot.clone()),
|
|
None,
|
|
components.db.clone(),
|
|
None,
|
|
None,
|
|
None,
|
|
None,
|
|
);
|
|
|
|
// Routine tools: create a RoutineEngine with the LLM and workspace.
|
|
if let (Some(db_arc), Some(ws)) = (&components.db, &components.workspace) {
|
|
use ironclaw::agent::routine_engine::RoutineEngine;
|
|
use ironclaw::config::RoutineConfig;
|
|
|
|
let routine_config = RoutineConfig::default();
|
|
let (notify_tx, _notify_rx) = tokio::sync::mpsc::channel(16);
|
|
let engine = Arc::new(RoutineEngine::new(
|
|
routine_config,
|
|
ironclaw::tenant::SystemScope::new(Arc::clone(db_arc)),
|
|
components.llm.clone(),
|
|
Arc::clone(ws),
|
|
notify_tx,
|
|
None,
|
|
None,
|
|
components.tools.clone(),
|
|
components.safety.clone(),
|
|
ironclaw::agent::routine_engine::SandboxReadiness::DisabledByConfig,
|
|
));
|
|
components
|
|
.tools
|
|
.register_routine_tools(Arc::clone(db_arc), engine);
|
|
}
|
|
|
|
// Skills tools: rebuild the registry against the test's tempdir.
|
|
//
|
|
// `AppBuilder::init_database()` re-resolves `config` from
|
|
// DB/TOML/env, which clobbers `config.skills.local_dir` back
|
|
// to the default (`~/.ironclaw/skills/`). Any registry
|
|
// `build_all()` already constructed therefore points at the
|
|
// user's real skills dir, not the tempdir the test laid
|
|
// down. Rebuild here from the in-scope `skills_dir` /
|
|
// `installed_skills_dir`, actually run discovery, and write
|
|
// the paths back onto `components.config` so downstream
|
|
// consumers (AgentDeps::skills_config) see the same dirs.
|
|
if enable_skills {
|
|
components.config.skills.local_dir = skills_dir.clone();
|
|
components.config.skills.installed_dir = installed_skills_dir.clone();
|
|
let mut registry = ironclaw_skills::SkillRegistry::new(skills_dir.clone())
|
|
.with_installed_dir(installed_skills_dir.clone());
|
|
let _loaded = registry.discover_all().await;
|
|
let registry = Arc::new(std::sync::RwLock::new(registry));
|
|
let catalog = ironclaw_skills::catalog::shared_catalog();
|
|
components
|
|
.tools
|
|
.register_skill_tools(Arc::clone(®istry), Arc::clone(&catalog));
|
|
components.skill_registry = Some(registry);
|
|
components.skill_catalog = Some(catalog);
|
|
}
|
|
|
|
// Register any extra test-specific tools.
|
|
for tool in extra_tools {
|
|
components.tools.register(tool).await;
|
|
}
|
|
|
|
// Register WASM tools with the shared HTTP interceptor.
|
|
if !wasm_tools.is_empty() {
|
|
use ironclaw::tools::wasm::{
|
|
Capabilities, CapabilitiesFile, WasmRuntimeConfig, WasmToolRuntime,
|
|
WasmToolWrapper,
|
|
};
|
|
|
|
let runtime = Arc::new(
|
|
WasmToolRuntime::new(WasmRuntimeConfig::default())
|
|
.expect("create WASM runtime for test rig"),
|
|
);
|
|
|
|
for spec in wasm_tools {
|
|
if !spec.wasm_path.exists() {
|
|
tracing::warn!(
|
|
name = %spec.name,
|
|
path = %spec.wasm_path.display(),
|
|
"WASM tool binary not found, skipping"
|
|
);
|
|
continue;
|
|
}
|
|
let wasm_bytes = tokio::fs::read(&spec.wasm_path)
|
|
.await
|
|
.unwrap_or_else(|e| panic!("read {}: {e}", spec.wasm_path.display()));
|
|
let (capabilities, description) =
|
|
if let Some(cap_path) = &spec.capabilities_path {
|
|
if cap_path.exists() {
|
|
let cap_bytes = tokio::fs::read(cap_path)
|
|
.await
|
|
.unwrap_or_else(|e| panic!("read {}: {e}", cap_path.display()));
|
|
let cap_file = CapabilitiesFile::from_bytes(&cap_bytes)
|
|
.expect("parse capabilities.json");
|
|
(cap_file.to_capabilities(), cap_file.description.clone())
|
|
} else {
|
|
(Capabilities::default(), None)
|
|
}
|
|
} else {
|
|
(Capabilities::default(), None)
|
|
};
|
|
|
|
let prepared = runtime
|
|
.prepare(&spec.name, &wasm_bytes, None)
|
|
.await
|
|
.unwrap_or_else(|e| panic!("prepare WASM tool '{}': {e}", spec.name));
|
|
let mut wrapper =
|
|
WasmToolWrapper::new(Arc::clone(&runtime), prepared, capabilities);
|
|
if let Some(desc) = description {
|
|
wrapper = wrapper.with_description(desc);
|
|
}
|
|
if let Some(interceptor) = &http_interceptor {
|
|
wrapper = wrapper.with_http_interceptor(Arc::clone(interceptor));
|
|
}
|
|
components.tools.register(Arc::new(wrapper)).await;
|
|
}
|
|
}
|
|
}
|
|
|
|
// Save references for test accessors.
|
|
let db_ref = components.db.clone().expect("test rig requires a database");
|
|
let workspace_ref = components.workspace.clone();
|
|
let ext_mgr_ref = components.extension_manager.clone();
|
|
let skill_registry_ref = components.skill_registry.clone();
|
|
let session_manager_ref = Arc::new(ironclaw::agent::SessionManager::new());
|
|
|
|
// Pre-seed credentials BEFORE the agent starts. This lets live
|
|
// tests inject a fake `github_token` (or similar) so the kernel
|
|
// pre-flight auth gate doesn't block the conversation when a
|
|
// skill with a credential spec activates. The value is opaque —
|
|
// tests aren't actually hitting the credentialed API, the secret
|
|
// just needs to exist under the test's owner_id.
|
|
if !pre_seed_secrets.is_empty() {
|
|
if let Some(ref secrets_store) = components.secrets_store {
|
|
use ironclaw::secrets::CreateSecretParams;
|
|
let owner_id = components.config.owner_id.clone();
|
|
for (name, value) in &pre_seed_secrets {
|
|
let params = CreateSecretParams::new(name.clone(), value.clone());
|
|
// Only create if truly missing — other errors (DB, crypto)
|
|
// should surface rather than triggering a blind create.
|
|
match secrets_store.get_decrypted(&owner_id, name).await {
|
|
Ok(_) => {} // already seeded — skip
|
|
Err(ironclaw::secrets::SecretError::NotFound(_)) => {
|
|
if let Err(e) = secrets_store.create(&owner_id, params).await {
|
|
eprintln!(
|
|
"[TestRig] WARNING: failed to pre-seed secret '{name}' for \
|
|
user '{owner_id}': {e}"
|
|
);
|
|
}
|
|
}
|
|
Err(e) => {
|
|
eprintln!(
|
|
"[TestRig] WARNING: unexpected error checking secret '{name}': {e}"
|
|
);
|
|
}
|
|
}
|
|
}
|
|
} else {
|
|
eprintln!(
|
|
"[TestRig] WARNING: pre_seed_secrets requested but no SecretsStore is \
|
|
wired (need libsql backend with handles)"
|
|
);
|
|
}
|
|
}
|
|
|
|
// Capture handles tests need to read back state via the same
|
|
// SecretsStore the agent will use. Done before AgentDeps moves
|
|
// values out of `components`. The owner_id is required for any
|
|
// secret lookup since secret rows are keyed by user.
|
|
let secrets_store_ref = components.secrets_store.clone();
|
|
let owner_id_ref = components.config.owner_id.clone();
|
|
|
|
// 7. Construct AgentDeps from AppComponents (mirrors main.rs).
|
|
let deps = AgentDeps {
|
|
owner_id: components.config.owner_id.clone(),
|
|
store: components.db,
|
|
settings_store: components.settings_store,
|
|
llm: components.llm,
|
|
cheap_llm: components.cheap_llm,
|
|
safety: components.safety,
|
|
tools: components.tools,
|
|
workspace: components.workspace,
|
|
extension_manager: components.extension_manager,
|
|
skill_registry: components.skill_registry,
|
|
skill_catalog: components.skill_catalog,
|
|
skills_config: components.config.skills.clone(),
|
|
hooks: components.hooks,
|
|
auth_manager: None,
|
|
cost_guard: components.cost_guard,
|
|
sse_tx: None,
|
|
http_interceptor,
|
|
transcription: None,
|
|
document_extraction: None,
|
|
sandbox_readiness: ironclaw::agent::routine_engine::SandboxReadiness::DisabledByConfig,
|
|
builder: None,
|
|
llm_backend: "nearai".to_string(),
|
|
tenant_rates: std::sync::Arc::new(ironclaw::tenant::TenantRateRegistry::new(4, 3)),
|
|
};
|
|
|
|
// 7. Create TestChannel and ChannelManager.
|
|
// When testing bootstrap, the channel must be named "gateway" because
|
|
// the bootstrap greeting targets only the gateway channel. An explicit
|
|
// override (via `with_channel_name`) takes precedence so tests can
|
|
// mirror real-world channel naming for features keyed on the channel
|
|
// name (e.g. mission notifications routed back to the source channel).
|
|
//
|
|
// Channel user_id selection: align the channel user identity with the
|
|
// config's owner_id when one of the following is true:
|
|
// 1. The rig has live-seeded secrets — production credential
|
|
// lookups (`secrets WHERE user_id = ?`) must hit the rows we
|
|
// just inserted, not the hardcoded `"test-user"`.
|
|
// 2. Skills are enabled — engine v2 resolves the thread's project
|
|
// from the channel user_id; if the test user is not the owner,
|
|
// `resolve_user_project` creates a fresh per-user project with
|
|
// no skills migrated to it, and skill activation silently fails.
|
|
// For all other tests we keep the historical `"test-user"` default
|
|
// so existing tests don't change behaviour.
|
|
let channel_user_id = if seeded_secrets.is_some() || enable_skills {
|
|
components.config.owner_id.clone()
|
|
} else {
|
|
"test-user".to_string()
|
|
};
|
|
let test_channel = if let Some(ref name) = channel_name_override {
|
|
Arc::new(TestChannel::with_user_id(channel_user_id).with_name(name.clone()))
|
|
} else if keep_bootstrap {
|
|
Arc::new(TestChannel::with_user_id(channel_user_id).with_name("gateway"))
|
|
} else {
|
|
Arc::new(TestChannel::with_user_id(channel_user_id))
|
|
};
|
|
let handle = TestChannelHandle::new(Arc::clone(&test_channel));
|
|
let channel_manager = ChannelManager::new();
|
|
channel_manager.add(Box::new(handle)).await;
|
|
let channels = Arc::new(channel_manager);
|
|
|
|
// 7b. Register message tool so routines can send messages to channels.
|
|
deps.tools
|
|
.register_message_tools(Arc::clone(&channels), deps.extension_manager.clone())
|
|
.await;
|
|
|
|
// 8. Create Agent.
|
|
let routine_config = if enable_routines {
|
|
Some(ironclaw::config::RoutineConfig {
|
|
enabled: true,
|
|
cron_check_interval_secs: 60,
|
|
max_concurrent_routines: 3,
|
|
default_cooldown_secs: 300,
|
|
max_lightweight_tokens: 4096,
|
|
lightweight_tools_enabled: true,
|
|
lightweight_max_iterations: 3,
|
|
})
|
|
} else {
|
|
None
|
|
};
|
|
let agent = Agent::new(
|
|
components.config.agent.clone(),
|
|
deps,
|
|
channels,
|
|
None, // heartbeat_config
|
|
None, // hygiene_config
|
|
routine_config,
|
|
Some(Arc::clone(&components.context_manager)),
|
|
Some(Arc::clone(&session_manager_ref)),
|
|
);
|
|
|
|
// Match main.rs: fill the scheduler slot once Agent::new has created it.
|
|
*scheduler_slot.write().await = Some(agent.scheduler());
|
|
|
|
// 9. Spawn agent in background task.
|
|
let agent_handle = tokio::spawn(async move {
|
|
if let Err(e) = agent.run().await {
|
|
eprintln!("[TestRig] Agent exited with error: {e}");
|
|
}
|
|
});
|
|
|
|
// 10. Wait for the agent to call channel.start() (up to 5 seconds).
|
|
if let Some(rx) = test_channel.take_ready_rx().await {
|
|
let _ = tokio::time::timeout(Duration::from_secs(5), rx).await;
|
|
}
|
|
|
|
TestRig {
|
|
channel: test_channel,
|
|
instrumented_llm: instrumented,
|
|
start_time: Instant::now(),
|
|
max_tool_iterations,
|
|
agent_handle: Some(agent_handle),
|
|
db: db_ref,
|
|
workspace: workspace_ref,
|
|
trace_llm: trace_llm_ref,
|
|
extension_manager: ext_mgr_ref,
|
|
skill_registry: skill_registry_ref,
|
|
session_manager: session_manager_ref,
|
|
secrets_store: secrets_store_ref,
|
|
owner_id: owner_id_ref,
|
|
_temp_dir: temp_dir,
|
|
bootstrap_greetings_to_keep: if keep_bootstrap { 1 } else { 0 },
|
|
}
|
|
}
|
|
}
|
|
|
|
impl Default for TestRigBuilder {
|
|
fn default() -> Self {
|
|
Self::new()
|
|
}
|
|
}
|
|
|
|
impl TestRig {
|
|
/// Get the database handle for direct queries.
|
|
#[cfg(feature = "libsql")]
|
|
pub fn database(&self) -> &Arc<dyn Database> {
|
|
&self.db
|
|
}
|
|
|
|
/// Get the workspace handle for direct memory operations.
|
|
#[cfg(feature = "libsql")]
|
|
pub fn workspace(&self) -> Option<&Arc<ironclaw::workspace::Workspace>> {
|
|
self.workspace.as_ref()
|
|
}
|
|
|
|
/// Get the underlying TraceLlm for inspecting captured requests.
|
|
#[cfg(feature = "libsql")]
|
|
pub fn trace_llm(&self) -> Option<&Arc<TraceLlm>> {
|
|
self.trace_llm.as_ref()
|
|
}
|
|
|
|
/// Check if any captured status events contain safety/injection warnings.
|
|
pub fn has_safety_warnings(&self) -> bool {
|
|
self.captured_status_events().iter().any(|s| {
|
|
matches!(s, StatusUpdate::Status(msg) if msg.contains("sanitiz") || msg.contains("inject") || msg.contains("warning"))
|
|
})
|
|
}
|
|
}
|
|
|
|
// ---------------------------------------------------------------------------
|
|
// Convenience: run a recorded trace fixture end-to-end
|
|
// ---------------------------------------------------------------------------
|
|
|
|
/// Load a recorded trace fixture, build a rig, run and verify expects, then shut down.
|
|
///
|
|
/// `filename` is relative to `tests/fixtures/llm_traces/recorded/`.
|
|
#[cfg(feature = "libsql")]
|
|
pub async fn run_recorded_trace(filename: &str) {
|
|
let path = format!(
|
|
"{}/tests/fixtures/llm_traces/recorded/{filename}",
|
|
env!("CARGO_MANIFEST_DIR")
|
|
);
|
|
let trace = LlmTrace::from_file(&path)
|
|
.unwrap_or_else(|e| panic!("failed to load trace {filename}: {e}"));
|
|
let rig = TestRigBuilder::new()
|
|
.with_trace(trace.clone())
|
|
.build()
|
|
.await;
|
|
rig.run_and_verify_trace(&trace, Duration::from_secs(30))
|
|
.await;
|
|
rig.shutdown();
|
|
}
|
|
|
|
/// Like [`run_recorded_trace`] but routes through the engine v2 pipeline.
|
|
#[cfg(feature = "libsql")]
|
|
pub async fn run_recorded_trace_v2(filename: &str) {
|
|
let path = format!(
|
|
"{}/tests/fixtures/llm_traces/recorded/{filename}",
|
|
env!("CARGO_MANIFEST_DIR")
|
|
);
|
|
let trace = LlmTrace::from_file(&path)
|
|
.unwrap_or_else(|e| panic!("failed to load trace {filename}: {e}"));
|
|
let rig = TestRigBuilder::new()
|
|
.with_engine_v2()
|
|
.with_trace(trace.clone())
|
|
.build()
|
|
.await;
|
|
rig.run_and_verify_trace(&trace, Duration::from_secs(30))
|
|
.await;
|
|
rig.shutdown();
|
|
}
|