lywsvip/ironclaw
1
0
Fork 0
mirror of https://github.com/nearai/ironclaw.git synced 2026-09-03 08:06:01 +08:00
Code Issues Packages Projects Releases Wiki Activity

test(mutation): add a mutation-audit harness, and fix the run-ordering contract it found (#6674)

Browse Source 
* test(projections): make the run-ordering contract actually detect its own bug

Mutation testing (`cargo mutants` over runtime_projection.rs, 39 mutants)
found `sort_runs_for_projection` could be replaced with a no-op while the whole
suite stayed green — including
`replay_projection_orders_runs_by_recent_activity_descending`, which exists
specifically to pin that ordering.

The test was not wrong, it was too weak to fail. `RuntimeProjectionState::runs`
is a `HashMap`, so `into_values()` yields iteration order, and with only two
invocations that order matches sorted order roughly half the time. The test
passed with the sort deleted by luck, and would have kept passing at whatever
rate `RandomState` happened to produce.

Six invocations now, with UUIDs deliberately not in append order so hash order
cannot trivially coincide with either append or sorted order. Accidental
agreement drops from ~1-in-2 to ~1-in-720. Strengthened in place rather than
added alongside, per the consolidate-don't-proliferate rule.

Also gitignores `mutants.out/`, the cargo-mutants report tree. It is a
regenerable local-audit artifact that currently shows up as untracked.

Verification: re-ran the same 39 mutants against the strengthened suite —
`sort_runs_for_projection` moves from MISSED to caught (20 caught / 2 missed,
was 19 / 3). Crate suite green, fmt and
`clippy -p ironclaw_event_projections --tests -- -D warnings` clean.

Two survivors remain and are deliberately not addressed here:

- `retain_invocations` -> no-op survives. It is the only enforcement of
  `fold_runtime_prefix`'s documented "invocations identified by `touched`" and
  `O(touched)` memory contracts, on a checkpoint that accumulates across pages.
  A regression test for it failed against unmutated code, meaning the intended
  contract is not what I assumed; whether the runs list is page-scoped or
  thread-scoped is a product question. Filed rather than guessed — weakening the
  assertion to match current behavior would defeat the purpose.
- `> -> >=` in `enforce_capability_activity_output_limit` is an equivalent
  mutant, not a coverage gap: at `len == limit` the mutated branch runs
  `select_nth_unstable_by` then a `truncate` that is a no-op, then the same full
  sort, so both paths produce identical output. No test can catch it.

Refs #6524

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(mutation): add a mutation-audit harness with a mechanical acceptance gate

Coverage proves a line ran; it says nothing about whether anything checked the
result. Mutation testing closes that gap by sabotaging one piece of production
code at a time and re-running the suite — a sabotage the suite still passes is
code with no assertion behind it.

This lands the tooling so the audit can be worked at volume. Epic #6524
workstream 11.

- `scripts/mutation-audit.sh` runs a scoped audit and emits a triage queue.
- `scripts/ci/mutation_triage_queue.py` turns the report into a self-contained
  work queue: each survivor with its sabotage diff and enclosing source inlined,
  so nobody has to go hunting. Scores over *viable* mutants, since unviable ones
  failed to compile and carry no signal.
- `scripts/mutation-verify-fix.sh` is the acceptance gate and the reason the
  queue scales: a fix is accepted only when the suite passes on real code AND
  fails with the sabotage applied. The first condition rejects a test reshaped
  to match current behaviour; the second rejects a decorative test. Both failure
  modes were produced while developing this harness, and both are now
  auto-rejected without anyone reviewing a diff.
- `docs/internal/mutation-audit.md` documents the verdict taxonomy
  (`real-gap` / `equivalent-mutant` / `needs-product-decision`) and the staged
  rollout.

Two traps are encoded in the tooling because both produced confidently wrong
answers during development:

1. **An unscoped run silently finds zero mutants.** Without `--package`,
   cargo-mutants scopes to the workspace-root package, which has no lib/bin. The
   result reads as a clean bill of health. The script now refuses to run
   unscoped.
2. **A shared `CARGO_TARGET_DIR` corrupts verdicts in both directions.**
   cargo-mutants copies the tree per job; pointing every copy at one absolute
   target directory lets parallel jobs clobber each other's artifacts, so a job
   can test a binary built from a different mutant's source. Observed, not
   theorised: the same mutant reported MISSED with a shared dir and caught
   without one, on byte-identical source. The dangerous direction is a surviving
   mutant reported as caught, which launders a real hole as covered. Both
   scripts now unset it and say so.

`equivalent-mutant` and `needs-product-decision` are first-class verdicts, not
escape hatches. Roughly a third of the first run's survivors could not be caught
by any test, and one needed a product decision about intended behaviour. Without
those verdicts, queue work degenerates into writing tests that assert current
behaviour — manufacturing the decorative coverage the audit exists to remove.
This is also why a mutation score must never gate CI.

Regression coverage: `scripts/test-mutation-audit.sh`, 15 hermetic cases (no
cargo) pinning each trap above plus empty-report and missing-report handling.
Guardrails are code; a checker that silently does nothing is worse than none.

Validation:
- `./scripts/test-mutation-audit.sh` — 15/15 pass
- `bash -n` clean on all three scripts; `py_compile` clean
- End-to-end against a real crate: the gate ACCEPTS
  `sort_runs_for_projection` (fixed in #6674) and correctly reports the two
  remaining survivors; it REJECTED the same mutant while a shared
  CARGO_TARGET_DIR was set, which is how trap 2 was found.
- Independently confirmed the #6674 fix outside cargo-mutants entirely: applied
  the sabotage by hand and ran the test 20 times — 20/20 caught.

Refs #6524

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(mutation): fix the harness's own guards after review

Addresses five review findings on #6674. Two were functional and one exposed
that a self-test was passing for the wrong reason.

1. **The cargo-mutants presence check ran before argument validation**, so on a
   machine without the tool the unscoped-run guard was unreachable and reported
   "not installed" instead of the usage error. The self-tests asserting that
   guard passed anyway — because the author had cargo-mutants installed. A
   self-test that depends on the developer's environment is the opposite of the
   hermetic guarantee its own header claims. Moved the check after validation
   and added case A2, which runs both scripts under a stub PATH containing no
   cargo at all.

2. **Grep alternation precedence in the baseline-failure check.**
   `^(FAILED|ERROR).*[Uu]nmutated baseline|baseline failed` parses as
   `(^(FAILED|ERROR).*Unmutated baseline)|(baseline failed)`, so any log line
   containing "baseline failed" anywhere tripped it. In a mechanical gate a
   false rejection is a real cost. Grouped the alternation under the prefix;
   verified a stray occurrence no longer matches while a real baseline failure
   still does.

3. `--help` printed through `set -euo pipefail` because the sed range outran the
   comment block. Stops at the last comment line now, like its sibling script.

4. Extracted `_count_nonempty_lines`, removing three copies of the same
   "filter non-blank lines from an optional file" logic and bringing
   `build_queue` back under Ruff's PLR0915 statement limit.

5. `window[:60] + [...]` -> `[*window[:60], ...]` (Ruff RUF005).

Validation: 17/17 self-tests pass (15 before, plus the two new absent-PATH
cases); `bash -n` and `py_compile` clean; `ruff check --select
RUF005,PLR0915,ARG,F401` clean on the queue generator; `--help` output and both
grep-precedence directions checked by hand.

Refs #6524

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(mutation): finish the queue-generator lint cleanup

Follow-up to the review fixes: the `_count_nonempty_lines` extraction alone did
not clear Ruff PLR0915 (it traded two comprehensions for two calls, roughly net
zero — still 60 statements). I pushed 83282bb38 before reading the ruff output
properly, so this completes it.

`build_queue` is now decomposed into `_preamble`, `_entry`, and
`_load_outcomes`, which drops it well under the statement limit and makes each
piece independently readable.

Also parenthesised six implicit string concatenations inside list literals
(ISC004). Not chased for the linter's sake — the repo has no ruff config and its
existing Python scripts report 16 default-ruleset findings — but because inside
a list `"a" "b"` looks like two entries and is one, so a dropped comma silently
merges rows of the verdict table.

`ruff check` is now clean on the file with the default ruleset. 16 self-tests
pass and the generated queue is byte-identical on real report data (20/22 over
viable mutants, both sabotage diffs inlined).

Refs #6524

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
firat.sertgoz
2026-07-27 10:56:00 +03:00
committed by GitHub
parent 1cb8b1a9aa
commit 02edc4e688
7 changed files with 798 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.gitignore vendored
View File

@@ -99,3 +99,9 @@ test-tools/*/wasm-src/target/
# (webui_v2, e2e, tooling, …) and stops the recurring accidental adds
# (#6298/#6305).
node_modules/
# cargo-mutants writes its report tree (caught/missed/timeout lists, per-mutant
# logs, and a copy of each mutated diff) next to the manifest it is run from.
# It is a regenerable build artifact of a local audit, never source.
mutants.out/
mutants.out.old/

58
crates/ironclaw_event_projections/tests/replay_projection_contract.rs
View File

@@ -1829,37 +1829,65 @@ async fn replay_projection_keeps_spawned_process_run_active_until_terminal_proce
#[tokio::test]
async fn replay_projection_orders_runs_by_recent_activity_descending() {
// Six invocations, not two. `RuntimeProjectionState::runs` is a `HashMap`,
// so with two entries its iteration order matches sorted order about half
// the time and this test passes by luck even when the sort is gone —
// mutation testing confirmed `sort_runs_for_projection` could be replaced
// with a no-op while the suite stayed green. Six entries make accidental
// agreement a 1-in-720 coincidence, and the UUIDs below are deliberately
// not in append order so hash order cannot trivially match it either.
let log = Arc::new(InMemoryDurableEventLog::new());
let service = ReplayEventProjectionService::new(Arc::clone(&log));
let thread = ThreadId::new("thread-a").unwrap();
let older_invocation = InvocationId::parse("00000000-0000-4000-8000-000000000001").unwrap();
let newer_invocation = InvocationId::parse("ffffffff-ffff-4fff-8fff-ffffffffffff").unwrap();
let older_scope = scope_for_thread_with_invocation(thread.clone(), older_invocation);
let newer_scope = scope_for_thread_with_invocation(thread, newer_invocation);
let capability = capability_id();
let invocations = [
"00000000-0000-4000-8000-000000000001",
"ffffffff-ffff-4fff-8fff-ffffffffffff",
"7f7f7f7f-7f7f-4f7f-8f7f-7f7f7f7f7f7f",
"11111111-1111-4111-8111-111111111111",
"cccccccc-cccc-4ccc-8ccc-cccccccccccc",
"3a3a3a3a-3a3a-4a3a-8a3a-3a3a3a3a3a3a",
]
.map(|raw| InvocationId::parse(raw).unwrap());
log.append(RuntimeEvent::dispatch_requested(
older_scope.clone(),
capability.clone(),
))
.await
.unwrap();
log.append(RuntimeEvent::dispatch_requested(newer_scope, capability))
let mut scopes = Vec::new();
for invocation in &invocations {
let scope = scope_for_thread_with_invocation(thread.clone(), *invocation);
log.append(RuntimeEvent::dispatch_requested(
scope.clone(),
capability.clone(),
))
.await
.unwrap();
scopes.push(scope);
}
let snapshot = service
.snapshot(ProjectionRequest {
scope: ProjectionScope::from_resource_scope(&older_scope),
scope: ProjectionScope::from_resource_scope(&scopes[0]),
after: None,
limit: 16,
})
.await
.unwrap();
assert_eq!(snapshot.runs.len(), 2);
assert_eq!(snapshot.runs[0].invocation_id, newer_invocation);
assert_eq!(snapshot.runs[1].invocation_id, older_invocation);
assert_eq!(
snapshot.runs.len(),
invocations.len(),
"{:?}",
snapshot.runs
);
// Most recent activity first, so exactly the reverse of append order.
let expected = invocations.iter().rev().copied().collect::<Vec<_>>();
assert_eq!(
snapshot
.runs
.iter()
.map(|run| run.invocation_id)
.collect::<Vec<_>>(),
expected,
"runs must be ordered most-recent-first, not in HashMap iteration order"
);
}
#[tokio::test]

157
docs/internal/mutation-audit.md Normal file
View File

@@ -0,0 +1,157 @@
# Mutation audits — testing the tests
Coverage proves a line *ran*. It says nothing about whether anything *checked
the result*. A test can execute a sorting function and never look at whether it
sorted.
A mutation audit sabotages one piece of production code at a time and re-runs
the suite. A sabotage the suite still passes is code with no assertion behind
it. That is the gap coverage cannot see.
Epic #6524, workstream 11.
## The finding that motivated this
`crates/ironclaw_event_projections/src/runtime_projection.rs` had a test named
`replay_projection_orders_runs_by_recent_activity_descending` — its entire job
was to verify run ordering. Deleting the sort function it was guarding did not
fail it.
The projection stores runs in a `HashMap`, and the test used two invocations.
With two entries, unsorted order matches sorted order about half the time, so
the test was a coin flip that kept landing heads. It was correct in intent and
structurally unable to fail. Reading it tells you nothing; only the sabotage
does. Fixed in PR #6674 by going to six invocations (~1-in-720 rather than
~1-in-2), verified by 20 consecutive hand-sabotaged runs.
## Running one
```bash
cargo install cargo-mutants --locked
# one file (start here — a file is a coffee break, not an overnight job)
./scripts/mutation-audit.sh -p ironclaw_event_projections \
crates/ironclaw_event_projections/src/runtime_projection.rs
# a whole package
./scripts/mutation-audit.sh -p ironclaw_dispatcher
```
Output is `mutants.out/triage-queue.md`: one entry per survivor, with the
sabotage diff and the enclosing source inlined so nobody has to go hunting.
`mutants.out/` is gitignored — it is a regenerable artifact.
Measured cost: 39 mutants over one 449-line file in ~9 minutes (a one-time
baseline build, then ~3s build + ~3s test per mutant via incremental
compilation).
## Never inherit `CARGO_TARGET_DIR`
Both scripts unset it and say so. This is not fussiness — it silently corrupts
results.
cargo-mutants copies the source tree per job. If every copy is redirected at one
shared absolute target directory, parallel jobs clobber each other's compiled
artifacts and a job can run a test binary built from a *different* job's mutated
source. Verdicts then come out wrong in **both** directions: a killed mutant
reported as surviving, and — far worse — a surviving mutant reported as caught,
which launders a real hole as covered.
This was observed, not theorised. The same mutant reported MISSED with a shared
target dir and caught without one, on byte-identical source. If you invoke
`cargo mutants` directly rather than through these scripts, do not set it.
## Triage: every survivor gets exactly one verdict
| verdict | meaning | next step |
|---|---|---|
| `real-gap` | the behaviour is genuinely unasserted | write a test; `scripts/mutation-verify-fix.sh` must accept it |
| `equivalent-mutant` | no test *can* catch it — the change cannot alter observable behaviour | record the reasoning; write no test |
| `needs-product-decision` | the intended contract is unclear | route to an owner; **do not invent one** |
**Score over viable mutants, not all mutants.** Mutants that fail to compile
("unviable") carry no signal. The motivating run was 20 caught / 22 viable, with
17 unviable — reporting 20/39 would understate the suite and send someone
chasing non-problems.
### `equivalent-mutant` is real and common
Roughly a third of survivors in the first run could not be caught by any test.
Worked example from that run — `> ` changed to `>=` in
`enforce_capability_activity_output_limit`:
At `len == limit`, the only input where the two differ, the mutated branch runs
`select_nth_unstable_by`, then a `truncate(limit)` that is a **no-op** because
`len == limit`, then the same full sort. Both paths produce identical output.
Writing a test to "fix" this would assert nothing.
This is why a mutation *score* must never gate CI: it would be a flake
generator, and the pressure to make the number go up produces exactly the
decorative tests the audit exists to find.
### `needs-product-decision` is load-bearing, not an escape hatch
The second survivor in that run was `retain_invocations` — the only enforcement
of `fold_runtime_prefix`'s documented "invocations identified by `touched`" and
`O(touched)` memory contracts, on a checkpoint that accumulates across pages.
A regression test written for it **failed against unmutated code**: today a page
does return earlier pages' invocations. So the model of the seam was wrong, not
the implementation, and whether the runs list is page-scoped or thread-scoped is
a product question.
The right move was to route it, not to reshape the assertion until it went
green. Without this verdict available, whoever works the queue invents an
intended contract and writes a test asserting current behaviour — manufacturing
the decorative coverage the audit is meant to remove. Grade queue work on
correctly routing here, not on closing every entry.
## The acceptance gate
A fix for a `real-gap` survivor is accepted only when both hold:
1. the suite **passes** on unmodified code — rejects a test reshaped to match a
mutant rather than the intended behaviour;
2. the suite **fails** with that sabotage applied — rejects a decorative test.
```bash
./scripts/mutation-verify-fix.sh -p ironclaw_event_projections \
'crates/ironclaw_event_projections/src/runtime_projection.rs:80:5: replace sort_runs_for_projection with ()'
```
Copy the mutant string verbatim from `missed.txt` or the triage queue. Exit 0
accepted, 1 rejected, 2 usage/tooling error.
This criterion is mechanical, which is what lets the queue be worked at volume:
no reviewer has to judge whether a test is real. Both failure modes above were
produced during development of this harness, and both are now auto-rejected.
## Scaling up
Workspace-wide there are ~43,700 generatable mutants
(`cargo mutants --list --workspace | wc -l`). Compute is affordable — sharded
nightly via `--shard k/n`. The cost that matters is triage, so expand a
frontier rather than switching everything on:
1. Modules where a bug already escaped to `main` — that is where the hypothesis
is cheapest to test, and where it first paid off.
2. The invariants workstream 11 names: authorization, approvals, credential
scope, tenant isolation, persistence/CAS, retry classification, trigger
scheduling, idempotency, delivery deduplication, redaction. A file list, not
whole crates.
3. Nightly across all crates, reporting only **newly surviving** mutants against
a committed baseline (`--iterate` skips previously-caught ones). Triage the
delta, not the corpus — the same shape as the coverage ratchet.
4. Never a PR-blocking mutation score. See `equivalent-mutant` above.
## Self-tests
```bash
./scripts/test-mutation-audit.sh
```
Fast and hermetic — no cargo. Pins the failure modes that produced confidently
wrong answers during development: an unscoped run silently finding zero mutants,
an inherited `CARGO_TARGET_DIR`, scoring over all mutants instead of viable
ones, and a missing report reading as an empty queue. Guardrails are code; a
checker that silently does nothing is worse than none.

206
scripts/ci/mutation_triage_queue.py Executable file
View File

@@ -0,0 +1,206 @@
#!/usr/bin/env python3
"""Turn a cargo-mutants report into a self-contained triage queue.
Each surviving mutant becomes one entry carrying the sabotage diff and the
enclosing source, so whoever works the queue never has to go find the code.
That matters at scale: the queue is the unit of work, and an entry that
requires hunting is an entry that gets skipped or guessed at.
Verdict taxonomy is documented in docs/internal/mutation-audit.md. The
`needs-product-decision` verdict is load-bearing, not a cop-out: without it,
whoever works the queue will invent an intended contract and write a test
asserting it, which manufactures exactly the decorative coverage this audit
exists to find.
"""
from __future__ import annotations
import argparse
import json
import re
from pathlib import Path
MUTANT_LINE = re.compile(r"^(?P<file>[^:]+):(?P<line>\d+):(?P<col>\d+): (?P<what>.*)$")
def _enclosing_rust_item(source_path: Path, line_number: int) -> tuple[str, int]:
"""Source of the fn/impl containing `line_number`, plus its start line.
Deliberately simple: scan back to the nearest column-0 or 4-space `fn`, then
forward to the next one. A mutation report is read by a human or an agent,
not parsed, so an approximate window beats a dependency on a Rust parser.
"""
try:
lines = source_path.read_text().splitlines()
except OSError:
return ("<source unavailable>", 0)
start = 0
opener = re.compile(r"^(\s{0,4})(pub(\([^)]*\))?\s+)?(async\s+)?(unsafe\s+)?fn\s")
for index in range(min(line_number, len(lines)) - 1, -1, -1):
if opener.match(lines[index]):
start = index
break
end = len(lines)
for index in range(start + 1, len(lines)):
if opener.match(lines[index]):
end = index
break
window = lines[start:end]
# Keep entries readable; a very long function is a smell worth seeing, but
# the queue should not become a source dump.
if len(window) > 60:
window = [*window[:60], " // … truncated, see the file for the rest"]
return ("\n".join(window), start + 1)
def _count_nonempty_lines(path: Path) -> int:
"""Non-blank lines in one of cargo-mutants' plain-text result files."""
if not path.is_file():
return 0
return len([line for line in path.read_text().splitlines() if line.strip()])
def _diff_for(report_dir: Path, outcomes: dict, mutant: str) -> str:
"""The sabotage cargo-mutants applied, as a diff, when it recorded one."""
for outcome in outcomes.get("outcomes", []):
scenario = outcome.get("scenario")
if not isinstance(scenario, dict):
continue
described = scenario.get("Mutant", {})
if not isinstance(described, dict):
continue
# `name` is already the exact string cargo-mutants writes to
# missed.txt, so match on it rather than rebuilding it from parts.
if str(described.get("name", "")).strip() != mutant.strip():
continue
diff_path = outcome.get("diff_path")
if not diff_path:
continue
candidate = report_dir / diff_path
if candidate.is_file():
return candidate.read_text()
return ""
def _preamble(survivors: int, caught: int, viable: int, unviable: int) -> list[str]:
"""Header block: the counts, and how to read them."""
lines = [
"# Mutation audit triage queue",
"",
(
f"- **{survivors} survivors** to triage "
f"({caught} caught, {viable} viable, {unviable} unviable/ignored)"
),
]
if viable:
lines.append(f"- Caught rate over viable mutants: **{caught}/{viable}**")
lines += [
(
"- Score the caught rate over *viable* mutants only. Unviable "
"mutants failed to compile and carry no signal."
),
"",
(
"Assign every entry exactly one verdict. See "
"`docs/internal/mutation-audit.md`."
),
"",
"| verdict | meaning | next step |",
"|---|---|---|",
(
"| `real-gap` | behaviour genuinely unasserted | write a test, "
"then `scripts/mutation-verify-fix.sh` must accept it |"
),
(
"| `equivalent-mutant` | no test *can* catch it — the change "
"cannot alter observable behaviour | record the reasoning; no test |"
),
(
"| `needs-product-decision` | the intended contract is unclear | "
"route to an owner; **do not invent one** |"
),
"",
"---",
"",
]
return lines
def _entry(index: int, mutant: str, report_dir: Path, outcomes: dict) -> list[str]:
"""One survivor, with enough context to triage without opening the file."""
lines = [
f"## {index}. `{mutant}`",
"",
"- verdict: `TODO`",
"- reasoning: _TODO_",
"",
]
match = MUTANT_LINE.match(mutant)
if match:
source_path = Path(match.group("file"))
body, start_line = _enclosing_rust_item(source_path, int(match.group("line")))
lines += [
f"Enclosing item (`{source_path}:{start_line}`):",
"",
"```rust",
body,
"```",
"",
]
diff = _diff_for(report_dir, outcomes, mutant)
if diff:
lines += ["Sabotage applied:", "", "```diff", diff.strip(), "```", ""]
lines += ["---", ""]
return lines
def _load_outcomes(report_dir: Path) -> dict:
"""cargo-mutants' machine-readable report, when it wrote a usable one."""
outcomes_path = report_dir / "outcomes.json"
if not outcomes_path.is_file():
return {}
try:
return json.loads(outcomes_path.read_text())
except json.JSONDecodeError:
return {}
def build_queue(report_dir: Path) -> str:
missed_path = report_dir / "missed.txt"
if not missed_path.is_file():
raise SystemExit(f"no missed.txt in {report_dir} — was the audit run?")
survivors = [line for line in missed_path.read_text().splitlines() if line.strip()]
outcomes = _load_outcomes(report_dir)
caught = _count_nonempty_lines(report_dir / "caught.txt")
unviable = _count_nonempty_lines(report_dir / "unviable.txt")
out = _preamble(len(survivors), caught, caught + len(survivors), unviable)
for index, mutant in enumerate(survivors, start=1):
out += _entry(index, mutant, report_dir, outcomes)
if not survivors:
out += ["No surviving mutants. Nothing to triage.", ""]
return "\n".join(out)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--report-dir", required=True, type=Path)
parser.add_argument("--output", required=True, type=Path)
args = parser.parse_args()
queue = build_queue(args.report_dir)
args.output.write_text(queue)
print(f"wrote {args.output}")
if __name__ == "__main__":
main()

116
scripts/mutation-audit.sh Executable file
View File

@@ -0,0 +1,116 @@
#!/usr/bin/env bash
# Run a targeted mutation audit and emit a triage-ready queue of survivors.
#
# Mutation testing sabotages one piece of production code at a time and re-runs
# the tests. A sabotage the suite still passes ("MISSED") is code with no
# assertion behind it. Line coverage cannot find these: it proves a line ran,
# not that anything checked the result.
#
# Usage:
# ./scripts/mutation-audit.sh -p ironclaw_event_projections \
# crates/ironclaw_event_projections/src/runtime_projection.rs
# ./scripts/mutation-audit.sh -p ironclaw_dispatcher # whole package
#
# Options (env vars):
# MUT_JOBS=3 Parallel mutants (default: 3)
# MUT_TIMEOUT=300 Per-mutant timeout in seconds (default: 300)
# MUT_OUT=mutants.out Report directory (default: mutants.out)
# MUT_ITERATE=0 Set to 1 to skip mutants caught in a previous run
#
# Output: $MUT_OUT/triage-queue.md — one entry per survivor with its sabotage
# diff and the enclosing source, so a reviewer never has to go hunting.
#
# Requires: cargo-mutants (install: cargo install cargo-mutants --locked)
#
# Deliberately NOT a CI gate. See docs/internal/mutation-audit.md — roughly a
# third of survivors are "equivalent mutants" that no test can catch, so a
# mutation score would be a flake generator. This is a periodic audit whose
# output is a work queue.
set -euo pipefail
MUT_JOBS="${MUT_JOBS:-3}"
MUT_TIMEOUT="${MUT_TIMEOUT:-300}"
MUT_OUT="${MUT_OUT:-mutants.out}"
MUT_ITERATE="${MUT_ITERATE:-0}"
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root"
# See scripts/mutation-verify-fix.sh for the full rationale: a shared
# CARGO_TARGET_DIR lets parallel mutant builds clobber each other's artifacts,
# so a job can test a binary built from a different mutant's source. Verdicts
# then come out wrong in both directions. Never inherit it.
if [ -n "${CARGO_TARGET_DIR:-}" ]; then
echo "note: ignoring CARGO_TARGET_DIR=$CARGO_TARGET_DIR — a shared target" >&2
echo " directory produces wrong mutation verdicts." >&2
unset CARGO_TARGET_DIR
fi
package=""
files=()
while [ $# -gt 0 ]; do
case "$1" in
-p | --package)
package="$2"
shift 2
;;
-h | --help)
sed -n '2,28p' "${BASH_SOURCE[0]}" | sed 's|^# \{0,1\}||'
exit 0
;;
*)
files+=("$1")
shift
;;
esac
done
if [ -z "$package" ]; then
# Without --package, cargo-mutants scopes to the workspace-root package,
# which has no lib or bin of its own and therefore yields zero mutants —
# a silent no-op that reads like "nothing to fix".
echo "error: -p/--package is required (the workspace root has no lib/bin," >&2
echo " so an unscoped run silently finds zero mutants)." >&2
exit 1
fi
# Checked after argument validation on purpose: a usage error must report the
# usage error, not a missing tool. Ordering these the other way round made the
# unscoped-run guard unreachable on any machine without cargo-mutants installed
# — and made this script's self-tests pass only because the author had it.
if ! command -v cargo-mutants >/dev/null 2>&1; then
echo "error: cargo-mutants not installed." >&2
echo " cargo install cargo-mutants --locked" >&2
exit 1
fi
args=(--package "$package" --timeout "$MUT_TIMEOUT" --jobs "$MUT_JOBS" --output "$MUT_OUT")
for file in "${files[@]:-}"; do
[ -n "$file" ] && args+=(-f "$file")
done
[ "$MUT_ITERATE" = "1" ] && args+=(--iterate)
echo "▶ mutation audit: package=$package files=${files[*]:-<all>}"
set +e
cargo mutants "${args[@]}"
mutants_status=$?
set -e
# Exit 2 means surviving mutants were found, which is the expected outcome of an
# audit, not a failure. Anything else is a real error.
if [ "$mutants_status" -ne 0 ] && [ "$mutants_status" -ne 2 ]; then
echo "error: cargo mutants failed with status $mutants_status" >&2
exit "$mutants_status"
fi
python3 "$repo_root/scripts/ci/mutation_triage_queue.py" \
--report-dir "$MUT_OUT" \
--output "$MUT_OUT/triage-queue.md"
echo
echo "▶ triage queue: $MUT_OUT/triage-queue.md"
echo " Assign each survivor one verdict (see docs/internal/mutation-audit.md):"
echo " real-gap · equivalent-mutant · needs-product-decision"
echo " A fix for a real-gap survivor is only accepted once"
echo " scripts/mutation-verify-fix.sh proves it moves MISSED -> caught."

141
scripts/mutation-verify-fix.sh Executable file
View File

@@ -0,0 +1,141 @@
#!/usr/bin/env bash
# Prove that a new test actually kills the mutant it claims to kill.
#
# This is the acceptance gate for mutation-driven test work, and the reason the
# triage queue can be worked at scale without trusting anyone's judgement: the
# criterion is mechanical, not a review opinion.
#
# A fix is accepted only when BOTH hold:
# 1. the suite passes on unmodified code (the test is not just broken)
# 2. the suite fails with the sabotage applied (the test actually checks it)
#
# Condition 2 is what a decorative test cannot satisfy, and condition 1 is what
# a test reshaped to match current behaviour cannot satisfy. Together they
# reject both failure modes without a human reading the diff.
#
# Usage:
# ./scripts/mutation-verify-fix.sh -p <package> \
# 'crates/foo/src/bar.rs:80:5: replace baz with ()'
#
# The mutant string is copied verbatim from mutants.out/missed.txt or the
# triage queue.
#
# Options (env vars):
# MUT_TIMEOUT=300 Per-mutant timeout in seconds (default: 300)
#
# Exit codes: 0 accepted · 1 rejected · 2 usage/tooling error
set -euo pipefail
MUT_TIMEOUT="${MUT_TIMEOUT:-300}"
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root"
# A shared CARGO_TARGET_DIR silently corrupts mutation results and must never
# be inherited here. cargo-mutants copies the source tree per job; if every
# copy is redirected at one absolute target directory, parallel jobs clobber
# each other's compiled artifacts and a job can run a test binary built from a
# different job's mutated source. The verdicts then come out wrong in BOTH
# directions — a killed mutant reported as surviving, and worse, a surviving
# mutant reported as caught. This was observed, not theorised: the same mutant
# reported MISSED with a shared dir and caught without one, on identical source.
# Correctness beats the warm cache for a tool whose only job is to be trusted.
if [ -n "${CARGO_TARGET_DIR:-}" ]; then
echo "note: ignoring CARGO_TARGET_DIR=$CARGO_TARGET_DIR — a shared target" >&2
echo " directory produces wrong mutation verdicts (see comment above)." >&2
unset CARGO_TARGET_DIR
fi
package=""
mutant=""
while [ $# -gt 0 ]; do
case "$1" in
-p | --package)
package="$2"
shift 2
;;
-h | --help)
sed -n '2,26p' "${BASH_SOURCE[0]}" | sed 's|^# \{0,1\}||'
exit 0
;;
*)
mutant="$1"
shift
;;
esac
done
if [ -z "$package" ] || [ -z "$mutant" ]; then
echo "usage: $0 -p <package> '<mutant line from missed.txt>'" >&2
exit 2
fi
if ! command -v cargo-mutants >/dev/null 2>&1; then
echo "error: cargo-mutants not installed (cargo install cargo-mutants --locked)" >&2
exit 2
fi
# The mutant line is "<file>:<line>:<col>: <description>". cargo-mutants
# --file wants just the path, and re-running the whole file is what lets us
# assert this specific mutant flipped to caught.
file="${mutant%%:*}"
if [ ! -f "$file" ]; then
echo "error: mutant does not name an existing file: $file" >&2
exit 2
fi
work="$(mktemp -d "${TMPDIR:-/tmp}/ironclaw-mutation-verify.XXXXXX")"
cleanup() { rm -rf "$work"; }
trap cleanup EXIT
echo "▶ verifying fix for:"
echo " $mutant"
echo
set +e
cargo mutants --package "$package" -f "$file" \
--timeout "$MUT_TIMEOUT" --jobs 3 --output "$work" >"$work/run.log" 2>&1
status=$?
set -e
if [ ! -f "$work/mutants.out/missed.txt" ]; then
echo "REJECTED: cargo mutants produced no report (status $status)." >&2
tail -20 "$work/run.log" >&2
exit 2
fi
# Baseline failure means the suite does not pass on unmodified code, so nothing
# downstream is meaningful — this catches a test written to match a mutant
# rather than the intended behaviour.
if grep -qiE "^(FAILED|ERROR).*([Uu]nmutated baseline|baseline failed)" "$work/run.log"; then
echo "REJECTED: the test suite does not pass on unmodified code." >&2
echo " A regression test must pass before it can prove anything." >&2
exit 1
fi
if grep -Fq "$mutant" "$work/mutants.out/missed.txt"; then
echo "REJECTED: that sabotage still survives — the suite passes with it applied."
echo " The new test does not actually check this behaviour."
echo
echo "Still-surviving mutants in $file:"
sed 's/^/ /' "$work/mutants.out/missed.txt"
exit 1
fi
if ! grep -Fq "$mutant" "$work/mutants.out/caught.txt" 2>/dev/null; then
echo "REJECTED: that mutant is neither missed nor caught in this run." >&2
echo " It is likely unviable now, or the string does not match any" >&2
echo " generated mutant — re-copy it from missed.txt verbatim." >&2
exit 1
fi
echo "ACCEPTED: the sabotage is now caught."
echo " suite passes on real code, and fails once this mutant is applied."
echo
echo "Remaining survivors in $file (for the triage queue):"
if [ -s "$work/mutants.out/missed.txt" ]; then
sed 's/^/ /' "$work/mutants.out/missed.txt"
else
echo " none"
fi

129
scripts/test-mutation-audit.sh Executable file
View File

@@ -0,0 +1,129 @@
#!/usr/bin/env bash
# Self-tests for the mutation-audit harness.
#
# Guardrails are code: a checker that silently does nothing is worse than no
# checker, because it reads as a clean bill of health. These cases pin the
# failure modes that actually bit during development — each one produced a
# wrong, confident answer before it was fixed.
#
# Fast and hermetic: no cargo, no compilation. The expensive end-to-end
# behaviour (a mutant flipping MISSED -> caught) is proven by running
# scripts/mutation-verify-fix.sh against a real crate, not here.
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
audit="$repo_root/scripts/mutation-audit.sh"
verify="$repo_root/scripts/mutation-verify-fix.sh"
queue="$repo_root/scripts/ci/mutation_triage_queue.py"
work="$(mktemp -d "${TMPDIR:-/tmp}/ironclaw-mutation-selftest.XXXXXX")"
cleanup() { rm -rf "$work"; }
trap cleanup EXIT
failures=0
check() {
local label="$1"
shift
if "$@"; then
echo " ok $label"
else
echo " FAIL $label"
failures=$((failures + 1))
fi
}
echo "▶ A. an unscoped audit is refused, not silently empty"
# Without --package, cargo-mutants scopes to the workspace-root package, which
# has no lib/bin, and reports zero mutants. That reads as "nothing to fix".
check "audit without --package exits non-zero" \
bash -c "! '$audit' 2>/dev/null"
check "audit without --package explains why" \
bash -c "'$audit' 2>&1 | grep -q 'silently finds zero mutants'"
echo "▶ A2. usage guards work with cargo-mutants absent from PATH"
# Regression: the cargo-mutants presence check originally ran *before* argument
# validation, so on a machine without the tool the unscoped-run guard was
# unreachable and reported the wrong error. These cases passed anyway because
# the author had cargo-mutants installed — the self-test depended on the
# developer's environment, which is exactly what "hermetic" is supposed to rule
# out. A stub PATH with no cargo at all reproduces a clean machine.
bare_path="$work/bare-bin"
mkdir -p "$bare_path"
for tool in bash sed grep python3 mktemp rm dirname cd; do
src="$(command -v "$tool" 2>/dev/null || true)"
[ -n "$src" ] && ln -sf "$src" "$bare_path/$tool"
done
check "audit still reports the usage error, not a missing tool" \
bash -c "PATH='$bare_path' '$audit' 2>&1 | grep -q 'silently finds zero mutants'"
check "verify still reports the usage error, not a missing tool" \
bash -c "PATH='$bare_path' '$verify' 2>&1 | grep -q 'usage:'"
echo "▶ B. the verify gate refuses incomplete invocations"
check "verify with no args exits non-zero" \
bash -c "! '$verify' 2>/dev/null"
check "verify without --package exits non-zero" \
bash -c "! '$verify' 'crates/x/src/y.rs:1:1: replace f with ()' 2>/dev/null"
check "verify rejects a mutant naming a nonexistent file" \
bash -c "! '$verify' -p some_pkg 'crates/nope/src/gone.rs:1:1: replace f with ()' 2>/dev/null"
echo "▶ C. both scripts refuse to inherit a shared CARGO_TARGET_DIR"
# This one is load-bearing: a shared target dir made the gate report a killed
# mutant as surviving, on identical source. Wrong in both directions.
check "audit warns and unsets CARGO_TARGET_DIR" \
bash -c "CARGO_TARGET_DIR=/tmp/shared-target '$audit' 2>&1 | grep -q 'ignoring CARGO_TARGET_DIR'"
check "verify warns and unsets CARGO_TARGET_DIR" \
bash -c "CARGO_TARGET_DIR=/tmp/shared-target '$verify' 2>&1 | grep -q 'ignoring CARGO_TARGET_DIR'"
echo "▶ D. the triage queue reports survivors and scores over viable mutants"
report="$work/report"
mkdir -p "$report"
cat >"$report/missed.txt" <<'EOF'
crates/demo/src/lib.rs:12:5: replace add with ()
EOF
cat >"$report/caught.txt" <<'EOF'
crates/demo/src/lib.rs:20:5: replace sub with ()
crates/demo/src/lib.rs:24:5: replace mul with ()
EOF
cat >"$report/unviable.txt" <<'EOF'
crates/demo/src/lib.rs:30:5: replace thing -> Self with Default::default()
crates/demo/src/lib.rs:34:5: replace other -> Self with Default::default()
crates/demo/src/lib.rs:38:5: replace more -> Self with Default::default()
EOF
python3 "$queue" --report-dir "$report" --output "$work/queue.md" >/dev/null
check "queue counts survivors" \
grep -q '\*\*1 survivors\*\*' "$work/queue.md"
# The headline number must exclude unviable mutants: they failed to compile and
# say nothing about test strength. Scoring 2/6 instead of 2/3 would understate
# the suite and invite someone to 'fix' non-problems.
check "queue scores over viable mutants only (2/3, not 2/6)" \
grep -q '\*\*2/3\*\*' "$work/queue.md"
check "queue lists the surviving mutant verbatim" \
grep -q 'replace add with ()' "$work/queue.md"
check "queue offers the needs-product-decision verdict" \
grep -q 'needs-product-decision' "$work/queue.md"
check "queue seeds every entry with an unset verdict" \
grep -q 'verdict: .TODO.' "$work/queue.md"
echo "▶ E. an empty survivor list is reported as clean, not as an error"
empty="$work/empty"
mkdir -p "$empty"
: >"$empty/missed.txt"
: >"$empty/caught.txt"
python3 "$queue" --report-dir "$empty" --output "$work/empty.md" >/dev/null
check "queue states there is nothing to triage" \
grep -q 'No surviving mutants' "$work/empty.md"
echo "▶ F. a missing report is a loud error, not an empty queue"
check "queue fails when missed.txt is absent" \
bash -c "! python3 '$queue' --report-dir '$work/absent' --output '$work/x.md' 2>/dev/null"
echo
if [ "$failures" -eq 0 ]; then
echo "all mutation-harness self-tests passed"
else
echo "$failures self-test(s) failed" >&2
exit 1
fi