diff --git a/docs/solutions/workflow-learnings/probe-the-instrument-the-way-ci-runs-it.md b/docs/solutions/workflow-learnings/probe-the-instrument-the-way-ci-runs-it.md index c44d43218d..00c415798a 100644 --- a/docs/solutions/workflow-learnings/probe-the-instrument-the-way-ci-runs-it.md +++ b/docs/solutions/workflow-learnings/probe-the-instrument-the-way-ci-runs-it.md @@ -71,11 +71,56 @@ When instruments in one program disagree about their own domain, every different unreadable until someone notices. Align the scopes, or the next person spends a day attributing a discrepancy to the thing being measured instead of to the measuring. +## Read the reported COUNT, not the exit code + +There is a second probing technique that sidesteps §1 entirely. It is **more informative for shape +coverage** — not strictly more informative, since it cannot answer §1's question at all: a count tells +you what the detector saw, never whether the ratchet would have failed on it. + +Instead of asking "did the tool exit non-zero", parse the tool's own **per-file count** out of its +output: + +```bash +# Capture output and status SEPARATELY: the pipeline below ends in `tr`, so a piped form reports +# `tr`'s success and a crashed detector reads as a clean zero — the failure this document is about. +out=$(node scripts/check-move-target-literals.mjs 2>&1); rc=$? +[ "$rc" -le 1 ] || { printf 'detector failed (rc=%s)\n%s\n' "$rc" "$out" >&2; exit 1; } + +count=$(printf '%s\n' "$out" | grep -a "my-probe-tmp" | grep -aoE "^ +[0-9]+" | tr -d ' ') +[ -n "$count" ] || { printf 'no matching line — probe not scanned?\n%s\n' "$out" >&2; exit 1; } +printf '%s\n' "$count" +``` + +`rc -le 1` because these ratchets use 1 for "violations found" and 2+ for "could not run"; a missing +match is reported rather than silently returning empty, since an absent line and a zero count are +different findings. + +Two properties matter: + +- **Immune to the report-only trap.** A report-only run still prints the count. The number moves from + 0 to 1 whether or not `--strict` was passed, so a missing flag cannot fake a pass. +- **It measures WHICH SHAPES, not just whether something fired.** An exit code is one bit for the whole + run. Auditing a detector means asking "of these five spellings, which are seen?" — and five separate + binary runs cannot distinguish *partial* detection from a probe file that failed to compile. The + count answers per form, in one run: the move-target audit read `direct 1 / backtick 1 / ternary 0 / + const 0`, which named the gap immediately. + +The two techniques answer different questions and both are worth keeping. Use `pnpm check:*` to ask +**"can this ratchet fail?"** — the §1 question, and the one to ask first. Use per-file counts to ask +**"what can it see?"** — the shape-coverage question, where an exit code is too coarse. + +One caveat that applies to both: confirm the probe file is actually being scanned, by watching the +tool's *scanned-file* total move. A probe that never compiled and a probe the tool never discovered +both report zero hits, and neither is a finding. + ## What to actually do - Before trusting a ratchet's number, run it once with a deliberate violation and confirm it goes red. A ratchet nobody has seen fail is a ratchet nobody has verified. -- When probing what a tool can see, use `pnpm check:*`, not a bare `node scripts/...`. +- Match the probe to the question: `pnpm check:*` for **"can this ratchet fail?"** (§1 — always run + it through the CI entry point, never a bare `node scripts/...`), and the per-file **count** probe for + **"what shapes can it see?"**. The count cannot answer the first question and the exit code cannot + answer the second. - Prefer one discovery mechanism across a family of tools. If a tool must differ, say so in its header and say why, so a cross-tool differential is readable. - `git ls-files --cached --others --exclude-standard` is the tracked-plus-untracked form (dedupe the