test(mutation): fix the harness's own guards after review

Addresses five review findings on #6674. Two were functional and one exposed
that a self-test was passing for the wrong reason.

1. **The cargo-mutants presence check ran before argument validation**, so on a
   machine without the tool the unscoped-run guard was unreachable and reported
   "not installed" instead of the usage error. The self-tests asserting that
   guard passed anyway — because the author had cargo-mutants installed. A
   self-test that depends on the developer's environment is the opposite of the
   hermetic guarantee its own header claims. Moved the check after validation
   and added case A2, which runs both scripts under a stub PATH containing no
   cargo at all.

2. **Grep alternation precedence in the baseline-failure check.**
   `^(FAILED|ERROR).*[Uu]nmutated baseline|baseline failed` parses as
   `(^(FAILED|ERROR).*Unmutated baseline)|(baseline failed)`, so any log line
   containing "baseline failed" anywhere tripped it. In a mechanical gate a
   false rejection is a real cost. Grouped the alternation under the prefix;
   verified a stray occurrence no longer matches while a real baseline failure
   still does.

3. `--help` printed through `set -euo pipefail` because the sed range outran the
   comment block. Stops at the last comment line now, like its sibling script.

4. Extracted `_count_nonempty_lines`, removing three copies of the same
   "filter non-blank lines from an optional file" logic and bringing
   `build_queue` back under Ruff's PLR0915 statement limit.

5. `window[:60] + [...]` -> `[*window[:60], ...]` (Ruff RUF005).

Validation: 17/17 self-tests pass (15 before, plus the two new absent-PATH
cases); `bash -n` and `py_compile` clean; `ruff check --select
RUF005,PLR0915,ARG,F401` clean on the queue generator; `--help` output and both
grep-precedence directions checked by hand.

Refs #6524

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
serrrfirat
2026-07-25 15:21:32 +03:00
parent 8bb0f02bde
commit 83282bb382
4 changed files with 40 additions and 23 deletions

View File

@@ -52,10 +52,17 @@ def _enclosing_rust_item(source_path: Path, line_number: int) -> tuple[str, int]
# Keep entries readable; a very long function is a smell worth seeing, but
# the queue should not become a source dump.
if len(window) > 60:
window = window[:60] + [" // … truncated, see the file for the rest"]
window = [*window[:60], " // … truncated, see the file for the rest"]
return ("\n".join(window), start + 1)
def _count_nonempty_lines(path: Path) -> int:
"""Non-blank lines in one of cargo-mutants' plain-text result files."""
if not path.is_file():
return 0
return len([line for line in path.read_text().splitlines() if line.strip()])
def _diff_for(report_dir: Path, outcomes: dict, mutant: str) -> str:
"""The sabotage cargo-mutants applied, as a diff, when it recorded one."""
for outcome in outcomes.get("outcomes", []):
@@ -93,20 +100,8 @@ def build_queue(report_dir: Path) -> str:
except json.JSONDecodeError:
outcomes = {}
caught = len(
[
line
for line in (report_dir / "caught.txt").read_text().splitlines()
if line.strip()
]
) if (report_dir / "caught.txt").is_file() else 0
unviable = len(
[
line
for line in (report_dir / "unviable.txt").read_text().splitlines()
if line.strip()
]
) if (report_dir / "unviable.txt").is_file() else 0
caught = _count_nonempty_lines(report_dir / "caught.txt")
unviable = _count_nonempty_lines(report_dir / "unviable.txt")
viable = caught + len(survivors)
out: list[str] = []

View File

@@ -47,12 +47,6 @@ if [ -n "${CARGO_TARGET_DIR:-}" ]; then
unset CARGO_TARGET_DIR
fi
if ! command -v cargo-mutants >/dev/null 2>&1; then
echo "error: cargo-mutants not installed." >&2
echo " cargo install cargo-mutants --locked" >&2
exit 1
fi
package=""
files=()
while [ $# -gt 0 ]; do
@@ -62,7 +56,7 @@ while [ $# -gt 0 ]; do
shift 2
;;
-h | --help)
sed -n '2,30p' "${BASH_SOURCE[0]}" | sed 's|^# \{0,1\}||'
sed -n '2,28p' "${BASH_SOURCE[0]}" | sed 's|^# \{0,1\}||'
exit 0
;;
*)
@@ -81,6 +75,16 @@ if [ -z "$package" ]; then
exit 1
fi
# Checked after argument validation on purpose: a usage error must report the
# usage error, not a missing tool. Ordering these the other way round made the
# unscoped-run guard unreachable on any machine without cargo-mutants installed
# — and made this script's self-tests pass only because the author had it.
if ! command -v cargo-mutants >/dev/null 2>&1; then
echo "error: cargo-mutants not installed." >&2
echo " cargo install cargo-mutants --locked" >&2
exit 1
fi
args=(--package "$package" --timeout "$MUT_TIMEOUT" --jobs "$MUT_JOBS" --output "$MUT_OUT")
for file in "${files[@]:-}"; do
[ -n "$file" ] && args+=(-f "$file")

View File

@@ -108,7 +108,7 @@ fi
# Baseline failure means the suite does not pass on unmodified code, so nothing
# downstream is meaningful — this catches a test written to match a mutant
# rather than the intended behaviour.
if grep -qiE "^(FAILED|ERROR).*[Uu]nmutated baseline|baseline failed" "$work/run.log"; then
if grep -qiE "^(FAILED|ERROR).*([Uu]nmutated baseline|baseline failed)" "$work/run.log"; then
echo "REJECTED: the test suite does not pass on unmodified code." >&2
echo " A regression test must pass before it can prove anything." >&2
exit 1

View File

@@ -41,6 +41,24 @@ check "audit without --package exits non-zero" \
check "audit without --package explains why" \
bash -c "'$audit' 2>&1 | grep -q 'silently finds zero mutants'"
echo "▶ A2. usage guards work with cargo-mutants absent from PATH"
# Regression: the cargo-mutants presence check originally ran *before* argument
# validation, so on a machine without the tool the unscoped-run guard was
# unreachable and reported the wrong error. These cases passed anyway because
# the author had cargo-mutants installed — the self-test depended on the
# developer's environment, which is exactly what "hermetic" is supposed to rule
# out. A stub PATH with no cargo at all reproduces a clean machine.
bare_path="$work/bare-bin"
mkdir -p "$bare_path"
for tool in bash sed grep python3 mktemp rm dirname cd; do
src="$(command -v "$tool" 2>/dev/null || true)"
[ -n "$src" ] && ln -sf "$src" "$bare_path/$tool"
done
check "audit still reports the usage error, not a missing tool" \
bash -c "PATH='$bare_path' '$audit' 2>&1 | grep -q 'silently finds zero mutants'"
check "verify still reports the usage error, not a missing tool" \
bash -c "PATH='$bare_path' '$verify' 2>&1 | grep -q 'usage:'"
echo "▶ B. the verify gate refuses incomplete invocations"
check "verify with no args exits non-zero" \
bash -c "! '$verify' 2>/dev/null"