mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-03 08:06:01 +08:00
90bed64622
* fix(live-qa): stop harness bugs reddening green canary runs The scheduled canary was red 30/30 runs; three harness defects failed correct product behavior and one liveness proxy reddened cases whose durable evidence held. - qa_7d_slack_bug_message_trigger: the trigger path completes without a model call, but CaseSpec.expects_llm_trace defaults True, so every green run failed trace_harvest. Declare expects_llm_trace=False and classify a missing trace as inconclusive/non-blocking so the exit code matches the Slack notifier. - qa_10d_slack_channel_membership: the lie arm scanned the whole reply, so an honest "(Not a member of ironclaw-qa.)" disclaimer reddened the case. Add sentence-scoped negation handling; only positive claims trip the arm. - qa_10h_slack_email_hallucination_guard: hard-pinned slack.get_user_info while the model honestly answers via slack.resolve_user (scope cannot read emails either way). Accept either capability, and accept explicit no-email paraphrases in addition to the EMAIL_UNAVAILABLE marker. - _routine_creation_case: when the trigger record proves the routine was created but the final reply omitted the required marker (e.g. an "already created" confirmation after schedule-validation retries), accept the durable evidence instead of reddening the case; typed failures keep their signal. Regression tests: extended test_run_live_qa.py for every changed path (trace_harvest classification, negation-aware membership matcher, resolve_user email path, durable-evidence upgrade, qa_7d spec pin). * fix(live-qa): address review findings on harness flake fixes - Negation scoping: a non-membership disclaimer now suppresses a claim only when it appears in the same CLAUSE as the named channel (split on sentence enders, commas, semicolons, but-family conjunctions; 'and' is deliberately not a boundary). 'I am a member of ironclaw-qa, but not a member of random' now flags the ironclaw-qa claim, while 'Not a member of general and ironclaw-qa' stays one disclaimer. - Data integrity: the routine-creation evidence override now reads back the exact trigger record (name-scoped snapshot) before upgrading a failed case. Marker-less callers count ALL trigger records, so an unrelated record can no longer satisfy after_count > before_count. - OR-group lookup: qa_10h drops the individually-required expected_capability and pins slack.get_user_info + slack.resolve_user purely as an accept-any OR group, with direct evidence-level coverage for a resolve_user-only run (and a no-evidence negative). Regression tests: mixed-clause negation scoping, unrelated-record non-upgrade, resolve_user-only accept-any evidence, existing durable- evidence and typed-failure cases updated for the snapshot read-back. * fix(live-qa): cover is_member-false disclaimers in membership arm Live run 31904223307 reddened qa_10d on a third honest phrasing: the model disclosed non-membership through the tool's own field — '(Note: ironclaw-qa appears in the list but is_member is false, so it is excluded.)'. A comma split separates the name from the qualifier, so the clause-scoped prose matcher missed it. Split the disclaimers into two scopes: prose negation phrases stay clause-scoped (the CodeRabbit mixed-clause fix is preserved), while is_member-false/excluded/not-included metadata markers are sentence- scoped — a note sentence is about the channel it names, not a claim. Positive is_member: true claims for non-member channels still flag. Regression tests: is_member-false note sentence, 'is_member: false' listing, 'X is not a member', and the is_member: true positive. * fix(live-qa): widen qa_7e sheet marker wait for slow LLM fires Live runs 31861416920/31841123051/31905604815: the triggered bug-routine fire appends the marker row with a live LLM turn that occasionally runs past the former 360s window (failures at ~392s with the marker absent). qa_7e is mechanically no-retry (side-effecting sheet/message writes), so the wait window is the flake absorber: 480s covers the slow-model tail and success still returns as soon as the marker lands.