mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-03 08:06:01 +08:00
test(mutation): fix the harness's own guards after review
Addresses five review findings on #6674. Two were functional and one exposed that a self-test was passing for the wrong reason. 1. **The cargo-mutants presence check ran before argument validation**, so on a machine without the tool the unscoped-run guard was unreachable and reported "not installed" instead of the usage error. The self-tests asserting that guard passed anyway — because the author had cargo-mutants installed. A self-test that depends on the developer's environment is the opposite of the hermetic guarantee its own header claims. Moved the check after validation and added case A2, which runs both scripts under a stub PATH containing no cargo at all. 2. **Grep alternation precedence in the baseline-failure check.** `^(FAILED|ERROR).*[Uu]nmutated baseline|baseline failed` parses as `(^(FAILED|ERROR).*Unmutated baseline)|(baseline failed)`, so any log line containing "baseline failed" anywhere tripped it. In a mechanical gate a false rejection is a real cost. Grouped the alternation under the prefix; verified a stray occurrence no longer matches while a real baseline failure still does. 3. `--help` printed through `set -euo pipefail` because the sed range outran the comment block. Stops at the last comment line now, like its sibling script. 4. Extracted `_count_nonempty_lines`, removing three copies of the same "filter non-blank lines from an optional file" logic and bringing `build_queue` back under Ruff's PLR0915 statement limit. 5. `window[:60] + [...]` -> `[*window[:60], ...]` (Ruff RUF005). Validation: 17/17 self-tests pass (15 before, plus the two new absent-PATH cases); `bash -n` and `py_compile` clean; `ruff check --select RUF005,PLR0915,ARG,F401` clean on the queue generator; `--help` output and both grep-precedence directions checked by hand. Refs #6524 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -52,10 +52,17 @@ def _enclosing_rust_item(source_path: Path, line_number: int) -> tuple[str, int]
|
||||
# Keep entries readable; a very long function is a smell worth seeing, but
|
||||
# the queue should not become a source dump.
|
||||
if len(window) > 60:
|
||||
window = window[:60] + [" // … truncated, see the file for the rest"]
|
||||
window = [*window[:60], " // … truncated, see the file for the rest"]
|
||||
return ("\n".join(window), start + 1)
|
||||
|
||||
|
||||
def _count_nonempty_lines(path: Path) -> int:
|
||||
"""Non-blank lines in one of cargo-mutants' plain-text result files."""
|
||||
if not path.is_file():
|
||||
return 0
|
||||
return len([line for line in path.read_text().splitlines() if line.strip()])
|
||||
|
||||
|
||||
def _diff_for(report_dir: Path, outcomes: dict, mutant: str) -> str:
|
||||
"""The sabotage cargo-mutants applied, as a diff, when it recorded one."""
|
||||
for outcome in outcomes.get("outcomes", []):
|
||||
@@ -93,20 +100,8 @@ def build_queue(report_dir: Path) -> str:
|
||||
except json.JSONDecodeError:
|
||||
outcomes = {}
|
||||
|
||||
caught = len(
|
||||
[
|
||||
line
|
||||
for line in (report_dir / "caught.txt").read_text().splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
) if (report_dir / "caught.txt").is_file() else 0
|
||||
unviable = len(
|
||||
[
|
||||
line
|
||||
for line in (report_dir / "unviable.txt").read_text().splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
) if (report_dir / "unviable.txt").is_file() else 0
|
||||
caught = _count_nonempty_lines(report_dir / "caught.txt")
|
||||
unviable = _count_nonempty_lines(report_dir / "unviable.txt")
|
||||
|
||||
viable = caught + len(survivors)
|
||||
out: list[str] = []
|
||||
|
||||
@@ -47,12 +47,6 @@ if [ -n "${CARGO_TARGET_DIR:-}" ]; then
|
||||
unset CARGO_TARGET_DIR
|
||||
fi
|
||||
|
||||
if ! command -v cargo-mutants >/dev/null 2>&1; then
|
||||
echo "error: cargo-mutants not installed." >&2
|
||||
echo " cargo install cargo-mutants --locked" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
package=""
|
||||
files=()
|
||||
while [ $# -gt 0 ]; do
|
||||
@@ -62,7 +56,7 @@ while [ $# -gt 0 ]; do
|
||||
shift 2
|
||||
;;
|
||||
-h | --help)
|
||||
sed -n '2,30p' "${BASH_SOURCE[0]}" | sed 's|^# \{0,1\}||'
|
||||
sed -n '2,28p' "${BASH_SOURCE[0]}" | sed 's|^# \{0,1\}||'
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
@@ -81,6 +75,16 @@ if [ -z "$package" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Checked after argument validation on purpose: a usage error must report the
|
||||
# usage error, not a missing tool. Ordering these the other way round made the
|
||||
# unscoped-run guard unreachable on any machine without cargo-mutants installed
|
||||
# — and made this script's self-tests pass only because the author had it.
|
||||
if ! command -v cargo-mutants >/dev/null 2>&1; then
|
||||
echo "error: cargo-mutants not installed." >&2
|
||||
echo " cargo install cargo-mutants --locked" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
args=(--package "$package" --timeout "$MUT_TIMEOUT" --jobs "$MUT_JOBS" --output "$MUT_OUT")
|
||||
for file in "${files[@]:-}"; do
|
||||
[ -n "$file" ] && args+=(-f "$file")
|
||||
|
||||
@@ -108,7 +108,7 @@ fi
|
||||
# Baseline failure means the suite does not pass on unmodified code, so nothing
|
||||
# downstream is meaningful — this catches a test written to match a mutant
|
||||
# rather than the intended behaviour.
|
||||
if grep -qiE "^(FAILED|ERROR).*[Uu]nmutated baseline|baseline failed" "$work/run.log"; then
|
||||
if grep -qiE "^(FAILED|ERROR).*([Uu]nmutated baseline|baseline failed)" "$work/run.log"; then
|
||||
echo "REJECTED: the test suite does not pass on unmodified code." >&2
|
||||
echo " A regression test must pass before it can prove anything." >&2
|
||||
exit 1
|
||||
|
||||
@@ -41,6 +41,24 @@ check "audit without --package exits non-zero" \
|
||||
check "audit without --package explains why" \
|
||||
bash -c "'$audit' 2>&1 | grep -q 'silently finds zero mutants'"
|
||||
|
||||
echo "▶ A2. usage guards work with cargo-mutants absent from PATH"
|
||||
# Regression: the cargo-mutants presence check originally ran *before* argument
|
||||
# validation, so on a machine without the tool the unscoped-run guard was
|
||||
# unreachable and reported the wrong error. These cases passed anyway because
|
||||
# the author had cargo-mutants installed — the self-test depended on the
|
||||
# developer's environment, which is exactly what "hermetic" is supposed to rule
|
||||
# out. A stub PATH with no cargo at all reproduces a clean machine.
|
||||
bare_path="$work/bare-bin"
|
||||
mkdir -p "$bare_path"
|
||||
for tool in bash sed grep python3 mktemp rm dirname cd; do
|
||||
src="$(command -v "$tool" 2>/dev/null || true)"
|
||||
[ -n "$src" ] && ln -sf "$src" "$bare_path/$tool"
|
||||
done
|
||||
check "audit still reports the usage error, not a missing tool" \
|
||||
bash -c "PATH='$bare_path' '$audit' 2>&1 | grep -q 'silently finds zero mutants'"
|
||||
check "verify still reports the usage error, not a missing tool" \
|
||||
bash -c "PATH='$bare_path' '$verify' 2>&1 | grep -q 'usage:'"
|
||||
|
||||
echo "▶ B. the verify gate refuses incomplete invocations"
|
||||
check "verify with no args exits non-zero" \
|
||||
bash -c "! '$verify' 2>/dev/null"
|
||||
|
||||
Reference in New Issue
Block a user