mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-03 08:06:01 +08:00
c5177861c1
* fix(host-runtime): classify HTTP error responses as failures * fix(host-runtime): address review — bound HTTP error diagnostics and add status regression tests (#7330) * docs(host-runtime): correct saved-body retrieval sequence for failed HTTP calls (#7330) * fix(host-runtime): converge failure-diagnostic trim, scrub controls, cover header/base64 branches - share one budget-trim engine (fit_output_to_budget) between the success path and the failure diagnostic; the diagnostic trim now converges in a strict-progress loop instead of an 8-iteration cap, and the headers branch stays reachable when the inline body is empty or absent, so header-heavy 4xx/5xx diagnostics keep status/auth_hint/truncation instead of collapsing to the fallback verdict - scrub DEL/C1 control bytes (U+007F..U+009F) from the serialized diagnostic: serde_json does not escape them, and ModelDiagnostic validation would otherwise replace the whole diagnostic with the fixed fallback sentence at the resolution boundary - move the shaped output by value instead of cloning it per error response - add unit coverage for the empty-body/header trim, base64 alignment on the failure path, and control-char scrubbing; wire the redirect regression test into the architecture-runtime gate * fix(host-runtime): shape errors at diagnostic budget, attach wall clock, pin envelope size - shape 4xx/5xx responses at the 4 KiB diagnostic budget instead of the success inline limit so the discarded success-budget trim pass (and its serializations) no longer runs on the failure path - attach dispatch wall_clock_ms to the OperationFailed usage, matching the sibling first-party dispatches' failure-path accounting - pin the truncation envelope below its reserved budget with a unit test - correct the fit_output_to_budget doc to describe the incremental re-measure loop rather than a fixed three-serialization bound * fix(host-runtime): pin error-status predicate, failure usage, and fallback shape - single is_error_status predicate shared by the dispatch shape-limit selection and classify_status, so the 400..=599 boundary cannot drift - pin failure usage accounting: classify_status unit test asserts egress bytes + wall_clock_ms on the OperationFailed outcome; the 403 integration test asserts egress bytes reach the governor for failed calls - pin the fallback diagnostic payload shape for non-object output and note why the serde-failure branch is unreachable by construction * fix(host-runtime): migrate stale 5xx-success integration test to failed-outcome contract - reborn_integration_http_matcher asserted the pre-change contract that a scripted HTTP 500 surfaces as a successful tool result; the new classification makes it a recoverable OperationFailed outcome, so the test now asserts ToolErrorClass::Failed with the operation_failed kind (run still completes; docstring updated to the contract doc) - pin the post-fit fallback safety valve with an oversized untrimmable key test; correct the governor-accounting comment; dedupe the 400 boundary rationale onto is_error_status * docs(host-runtime): sync matcher guide to failed-outcome contract, document fence interaction - tests/integration/CLAUDE.md .with_status entry now states 4xx/5xx classify as a Failed tool outcome (operation_failed) with sanitized diagnostic context; other statuses remain Completed results - host-runtime contract doc records the loop-host injection-fence interaction: verdict semantics never depend on the fenced diagnostic surviving the observation bound (OperationFailed + safe summary always reach the model) - document the fit_output_to_budget convergence bound (<= 3 passes) * docs(host-runtime): correct serialize_diagnostic guard rationale serde_json serializes every Value string without revalidating UTF-8 (probed: even an unsafe lone-surrogate string serializes Ok), so the serde-failure arm is a pure defensive guard for future Value shapes, not a reachable lone-surrogate path. Correct the doc comment and the fallback test note to state the empirical fact. * fix(host-runtime): keep shape-stage truncation flags in failure-diagnostic envelope The re-inserted truncation envelope carried only the diagnostic-budget trim state, so a 4xx/5xx response whose shape stage had already marked headers or body as truncated (e.g. more than 32 headers) could end up with headers_truncated:true beside an envelope claiming headers:false. OR the surviving keys into the envelope and pin with a regression test; boundary doc now states the full complement (outside 400..=599 stays inspectable, including out-of-spec 600+). * docs(host-runtime): sync support-module doc to failed-outcome contract, fix trim rationale - tests/integration/support/http_matcher.rs module doc still claimed .with_status non-2xx stays Completed; now documents 4xx/5xx as a model-visible Failed operation_failed outcome - bounded_failure_diagnostic doc: head-keeping truncation cuts only the last-sorted keys (status, truncation envelope); auth_hint sorts first and survives * fix(host-runtime): address coderabbitai/ironloopai review — egress cap, fence headroom, saved-body fallback (#7342) - shape failed responses at the caller's response_body_limit again so egress-truncation accounting (body_was_truncated_by_egress) stays correct; the diagnostic display budget is applied separately. Pinned by builtin_http_error_diagnostic_preserves_egress_truncation_flag. - reserve MODEL_DIAGNOSTIC_FENCE_HEADROOM_BYTES so a diagnostic wrapped in the loop-host external-content fence still fits the observation budget; pinned by failure_diagnostic_stays_within_budget_when_fenced. - retain compact saved_body evidence (bounded path prefix + bytes_written) in the fallback verdict instead of dropping the save destination; pinned by failure_diagnostic_fallback_retains_saved_body_evidence. - rename failure_diagnostic_falls_back_on_unserializable_output to the non-object contract it actually exercises (Value::Null serializes). - host-runtime contract: correct the save-mode body-retention wording and document retry_after_ms semantics (None does not permit immediate retry).
218 lines
10 KiB
Bash
Executable File
218 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
# Run the deterministic Rust-side Reborn E2E gate.
|
|
# Usage:
|
|
# scripts/reborn-e2e-rust.sh # all groups
|
|
# scripts/reborn-e2e-rust.sh architecture-boundaries # dependency and protocol boundaries
|
|
# scripts/reborn-e2e-rust.sh architecture-runtime # host runtime and capability spine
|
|
# scripts/reborn-e2e-rust.sh runtimes # dispatcher/runtime/process lanes
|
|
# scripts/reborn-e2e-rust.sh substrates # event/network/secret substrates
|
|
#
|
|
# Extra cargo test args can be passed through CARGO_TEST_ARGS, for example:
|
|
# CARGO_TEST_ARGS='-- --nocapture' scripts/reborn-e2e-rust.sh architecture
|
|
|
|
group="${1:-all}"
|
|
extra_args=${CARGO_TEST_ARGS:-"-- --nocapture"}
|
|
|
|
run_test() {
|
|
local package="$1"
|
|
local test_name="$2"
|
|
echo "::group::cargo test -p ${package} --test ${test_name}"
|
|
# shellcheck disable=SC2086 # extra_args intentionally expands into cargo's trailing args.
|
|
cargo test -p "${package}" --test "${test_name}" ${extra_args}
|
|
echo "::endgroup::"
|
|
}
|
|
|
|
run_test_exact() {
|
|
local package="$1"
|
|
local test_target="$2"
|
|
local test_name="$3"
|
|
local listed
|
|
listed=$(cargo test -p "${package}" --test "${test_target}" "${test_name}" -- --exact --list)
|
|
if ! grep -Fqx "${test_name}: test" <<<"${listed}"; then
|
|
echo "error: exact test selector matched zero tests: ${package}/${test_target} ${test_name}" >&2
|
|
return 1
|
|
fi
|
|
echo "::group::cargo test -p ${package} --test ${test_target} ${test_name} ${extra_args} --exact"
|
|
# shellcheck disable=SC2086 # extra_args intentionally expands into cargo's trailing args.
|
|
cargo test -p "${package}" --test "${test_target}" "${test_name}" ${extra_args} --exact
|
|
echo "::endgroup::"
|
|
}
|
|
|
|
run_lib_test() {
|
|
local package="$1"
|
|
local test_filter="$2"
|
|
echo "::group::cargo test -p ${package} --lib ${test_filter}"
|
|
# shellcheck disable=SC2086 # extra_args intentionally expands into cargo's trailing args.
|
|
cargo test -p "${package}" --lib "${test_filter}" ${extra_args}
|
|
echo "::endgroup::"
|
|
}
|
|
|
|
run_lib_test_exact() {
|
|
local package="$1"
|
|
local test_name="$2"
|
|
local listed
|
|
listed=$(cargo test -p "${package}" --lib "${test_name}" -- --exact --list)
|
|
if ! grep -Fqx "${test_name}: test" <<<"${listed}"; then
|
|
echo "error: exact library test selector matched zero tests: ${package} ${test_name}" >&2
|
|
return 1
|
|
fi
|
|
echo "::group::cargo test -p ${package} --lib ${test_name} ${extra_args} --exact"
|
|
# shellcheck disable=SC2086 # extra_args intentionally expands into cargo's trailing args.
|
|
cargo test -p "${package}" --lib "${test_name}" ${extra_args} --exact
|
|
echo "::endgroup::"
|
|
}
|
|
|
|
run_architecture_boundaries() {
|
|
run_test ironclaw_architecture_tests reborn_dependency_boundaries
|
|
# Pins docs/reborn/contracts/turns-agent-loop.md: terminal model
|
|
# provider authentication and transcript persistence failures remain durable,
|
|
# actionable, redacted, and never issue duplicate model/tool side effects.
|
|
run_test_exact ironclaw_integration_tests reborn_integration_cancel \
|
|
mid_turn_auth_provider_error_reaches_failed_with_credentials_category
|
|
run_test_exact ironclaw_integration_tests reborn_integration_model_recovery \
|
|
transcript_write_failure_stops_without_another_model_or_tool_side_effect
|
|
run_test_exact ironclaw_integration_tests reborn_integration_model_recovery \
|
|
tool_result_transcript_failure_stops_without_duplicate_model_or_tool_side_effect
|
|
# Keep protocol/recovery selectors with the targets already compiled by this
|
|
# lane instead of rebuilding them in the host-runtime lane.
|
|
run_test_exact ironclaw_loop_host llm_gateway \
|
|
gateway_maps_deterministic_provider_response_errors_to_invalid_output
|
|
run_test_exact ironclaw_integration_tests reborn_integration_model_recovery \
|
|
deterministic_provider_response_errors_use_bounded_invalid_output_recovery
|
|
# Pins the retired-taxonomy Telegram identifiers and prevents v1 pairing
|
|
# routes from re-entering the Reborn context.
|
|
run_test ironclaw_architecture_tests telegram_extension_gates
|
|
# Pins docs/reborn/contracts/host-api.md: every recoverable verdict carries
|
|
# an inline model diagnostic, and legacy omissions upgrade explicitly.
|
|
run_lib_test_exact ironclaw_host_api resolution::tests::recoverable_failure_carries_its_model_visible_diagnostic
|
|
run_lib_test_exact ironclaw_loop_contracts host::capability::tests::legacy_capability_failure_without_detail_rehydrates_explicit_fallback
|
|
# Pins docs/reborn/contracts/loop-exit.md: retired diagnostic_ref string/null
|
|
# payloads remain readable but the retired field is never written again.
|
|
run_lib_test_exact ironclaw_turns loop_exit::tests::loop_failed_accepts_retired_diagnostic_ref_but_does_not_serialize_it
|
|
# Pins docs/reborn/contracts/loop-exit.md and turn-runner.md: a rejected
|
|
# checkpoint remains terminal after projection into the process journal and
|
|
# cannot create a retry process.
|
|
run_lib_test_exact ironclaw_turns \
|
|
process_projection::runtime::tests::retry_rejects_checkpoint_rejection_without_creating_a_process
|
|
}
|
|
|
|
run_architecture_runtime() {
|
|
run_test ironclaw_host_runtime host_runtime_contract
|
|
run_test ironclaw_host_runtime host_runtime_services_contract
|
|
# Pins docs/reborn/contracts/host-runtime.md: scoped JSON files expose
|
|
# bounded collection selection and numeric aggregation through the real turn.
|
|
run_test_exact ironclaw_integration_tests reborn_integration_tool_call \
|
|
json_runs_bounded_collection_operations
|
|
run_test ironclaw_host_runtime reborn_e2e_gate
|
|
run_test ironclaw_host_runtime reborn_invoke_vertical_slice
|
|
run_test ironclaw_host_runtime runtime_http_egress_contract
|
|
# Pins docs/reborn/contracts/host-runtime.md: an HTTP 4xx/5xx response is a
|
|
# model-visible failed capability outcome, not transport-level success.
|
|
run_test_exact ironclaw_host_runtime first_party_builtin_tools \
|
|
builtin_http_surfaces_http_error_status_as_failed_outcome
|
|
run_test_exact ironclaw_host_runtime first_party_builtin_tools \
|
|
builtin_http_keeps_redirect_responses_model_visible
|
|
run_test_exact ironclaw_host_runtime first_party_builtin_tools \
|
|
builtin_http_surfaces_server_error_status_as_failed_outcome
|
|
run_test_exact ironclaw_host_runtime first_party_builtin_tools \
|
|
builtin_http_save_surfaces_http_error_status_as_failed_outcome
|
|
run_test_exact ironclaw_host_runtime first_party_builtin_tools \
|
|
builtin_http_classifies_status_range_boundaries
|
|
# Pins docs/reborn/contracts/host-runtime.md: the failure diagnostic is
|
|
# trimmed to the model-visible diagnostic budget and stays valid JSON.
|
|
run_test_exact ironclaw_host_runtime first_party_builtin_tools \
|
|
builtin_http_error_diagnostic_respects_model_diagnostic_budget
|
|
run_test ironclaw_host_runtime builtin_obligation_handler_contract
|
|
run_test ironclaw_host_runtime obligation_services_composition_contract
|
|
run_test ironclaw_host_runtime production_trust_contract
|
|
run_test ironclaw_capabilities capability_boundary_contract
|
|
run_test ironclaw_capabilities capability_host_contract
|
|
run_test ironclaw_capabilities capability_host_dispatcher_integration
|
|
run_test ironclaw_capabilities capability_host_process_integration
|
|
run_test ironclaw_capabilities capability_host_invocation_state_contract
|
|
run_test ironclaw_capabilities capability_host_spawn_contract
|
|
run_test ironclaw_capabilities capability_obligation_handler_contract
|
|
}
|
|
|
|
run_architecture() {
|
|
run_architecture_boundaries
|
|
run_architecture_runtime
|
|
}
|
|
|
|
run_runtimes() {
|
|
# These two suites pin `RuntimeDispatcher` and live with it in
|
|
# `ironclaw_capabilities`.
|
|
run_test ironclaw_capabilities runtime_dispatch_contract
|
|
run_test ironclaw_capabilities runtime_dispatch_event_contract
|
|
# main's runtime_dispatcher_integration / vertical_slice_contract test the
|
|
# retired RuntimeAdapter<F, G> architecture; the ToolResolver/BoundCapabilityAdapter
|
|
# pipeline is pinned by the two dispatch contract suites above.
|
|
run_test ironclaw_wasm wasm_dispatch_integration
|
|
run_test ironclaw_wasm wasm_http_adapter_contract
|
|
run_test ironclaw_wasm wit_tool_runtime_contract
|
|
run_test ironclaw_sandbox script_dispatch_integration
|
|
run_test ironclaw_sandbox script_http_adapter_contract
|
|
run_test ironclaw_sandbox script_runner_contract
|
|
run_test ironclaw_sandbox docker_security
|
|
run_test ironclaw_mcp mcp_adapter_contract
|
|
run_test ironclaw_mcp mcp_dispatch_integration
|
|
# Pins docs/reborn/contracts/trust-boundary-hardening.md through the whole
|
|
# turn: the scrubbed, bounded MCP cause reaches the next model request.
|
|
run_test_exact ironclaw_integration_tests reborn_integration_mcp mcp_tool_call_error_cause_is_scrubbed_and_bounded_in_next_model_request
|
|
run_test ironclaw_processes process_host_contract
|
|
run_test ironclaw_processes process_journal_store_contract
|
|
run_test ironclaw_processes legacy_migration_backend_contract
|
|
run_test ironclaw_processes process_services_contract
|
|
}
|
|
|
|
run_substrates() {
|
|
run_test ironclaw_event_log durable_log_contract
|
|
# Pins docs/reborn/contracts/events.md: runtime snapshot/replay projections
|
|
# preserve nested dispatcher failures without synthesizing child run rows.
|
|
run_test ironclaw_event_projections nested_dispatch_projection_contract
|
|
run_test ironclaw_filesystem catalog_contract
|
|
run_test ironclaw_filesystem filesystem_contract
|
|
run_test ironclaw_network boundary_contract
|
|
run_test ironclaw_network network_http_egress_contract
|
|
run_test ironclaw_network network_policy_contract
|
|
run_test ironclaw_secrets boundary_contract
|
|
run_test ironclaw_secrets secret_store_contract
|
|
run_test ironclaw_resources resource_governor_contract
|
|
run_test ironclaw_approvals approval_store_contract
|
|
run_test ironclaw_approvals approval_resolution_contract
|
|
run_test ironclaw_approvals boundary_contract
|
|
run_test ironclaw_authorization boundary_contract
|
|
run_test ironclaw_authorization capability_access_contract
|
|
run_test ironclaw_authorization capability_lease_contract
|
|
}
|
|
|
|
case "${group}" in
|
|
architecture)
|
|
run_architecture
|
|
;;
|
|
architecture-boundaries)
|
|
run_architecture_boundaries
|
|
;;
|
|
architecture-runtime)
|
|
run_architecture_runtime
|
|
;;
|
|
runtimes)
|
|
run_runtimes
|
|
;;
|
|
substrates)
|
|
run_substrates
|
|
;;
|
|
all)
|
|
run_architecture
|
|
run_runtimes
|
|
run_substrates
|
|
;;
|
|
*)
|
|
echo "unknown Reborn E2E group: ${group}" >&2
|
|
echo "expected one of: architecture, architecture-boundaries, architecture-runtime, runtimes, substrates, all" >&2
|
|
exit 2
|
|
;;
|
|
esac
|