docs(solutions): add the count-based probe, which the report-only trap cannot fool (#3257)

## What

Adds one technique to #3255. Docs only.

#3255 records that probing a ratchet **by exit code** can read green
because the tool is report-only without `--strict` — a real trap that
nearly got a healthy gate reported as dead. There is a second technique
that sidesteps it entirely and is strictly more informative: **parse the
tool's own per-file count.**

```bash
node scripts/check-move-target-literals.mjs 2>&1 | grep -a "my-probe-tmp" \
  | grep -aoE "^ +[0-9]+" | tr -d ' '
```

**Immune to the report-only trap** — a report-only run still *prints*
the count, so the number moves 0 → 1 whether or not `--strict` was
passed.

**It measures which shapes, not just whether something fired.** An exit
code is one bit for the whole run. Auditing a detector means asking *"of
these five spellings, which are seen?"*, and five separate binary runs
cannot distinguish **partial** detection from a probe file that failed
to compile. The move-target audit read `direct 1 / backtick 1 / ternary
0 / const 0` in a single run, which named the gap immediately.

## Both belong

| question | technique |
|---|---|
| **can this ratchet fail at all?** | `pnpm check:*` — ask this first
(#3255 §1) |
| **what can it see?** | per-file counts — an exit code is too coarse |

I also added a caveat that applies to both: confirm the probe is
actually being scanned by watching the tool's **scanned-file total**
move. A probe that never compiled and a probe the tool never discovered
both report zero hits, and neither is a finding — that one cost me a
wasted measurement before I noticed the total had stayed at 1961.

## Why this is worth a follow-up rather than a comment

#3255's rule as written — *"use `pnpm check:*`, not a bare `node
scripts/...`"* — would have made the shape-coverage audits impossible,
since `--strict` collapses five distinct per-form answers into one bit.
The rule is right for its question and wrong for the other one, and the
distinction is easy to lose once only the rule survives in someone's
memory.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Added guidance for evaluating ratchets using per-file output counts.
* Documented report-only and shape-coverage limitations, count-based
versus failure-based checks, and verifying that probe files were
scanned.
  * Included a command example for probing ratchet behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
gsxdsm
2026-07-31 16:03:58 -07:00
committed by GitHub
parent c8a6af13a0
commit d14294b6cb

View File

@@ -71,11 +71,56 @@ When instruments in one program disagree about their own domain, every different
unreadable until someone notices. Align the scopes, or the next person spends a day attributing a
discrepancy to the thing being measured instead of to the measuring.
## Read the reported COUNT, not the exit code
There is a second probing technique that sidesteps §1 entirely. It is **more informative for shape
coverage** — not strictly more informative, since it cannot answer §1's question at all: a count tells
you what the detector saw, never whether the ratchet would have failed on it.
Instead of asking "did the tool exit non-zero", parse the tool's own **per-file count** out of its
output:
```bash
# Capture output and status SEPARATELY: the pipeline below ends in `tr`, so a piped form reports
# `tr`'s success and a crashed detector reads as a clean zero — the failure this document is about.
out=$(node scripts/check-move-target-literals.mjs 2>&1); rc=$?
[ "$rc" -le 1 ] || { printf 'detector failed (rc=%s)\n%s\n' "$rc" "$out" >&2; exit 1; }
count=$(printf '%s\n' "$out" | grep -a "my-probe-tmp" | grep -aoE "^ +[0-9]+" | tr -d ' ')
[ -n "$count" ] || { printf 'no matching line — probe not scanned?\n%s\n' "$out" >&2; exit 1; }
printf '%s\n' "$count"
```
`rc -le 1` because these ratchets use 1 for "violations found" and 2+ for "could not run"; a missing
match is reported rather than silently returning empty, since an absent line and a zero count are
different findings.
Two properties matter:
- **Immune to the report-only trap.** A report-only run still prints the count. The number moves from
0 to 1 whether or not `--strict` was passed, so a missing flag cannot fake a pass.
- **It measures WHICH SHAPES, not just whether something fired.** An exit code is one bit for the whole
run. Auditing a detector means asking "of these five spellings, which are seen?" — and five separate
binary runs cannot distinguish *partial* detection from a probe file that failed to compile. The
count answers per form, in one run: the move-target audit read `direct 1 / backtick 1 / ternary 0 /
const 0`, which named the gap immediately.
The two techniques answer different questions and both are worth keeping. Use `pnpm check:*` to ask
**"can this ratchet fail?"** — the §1 question, and the one to ask first. Use per-file counts to ask
**"what can it see?"** — the shape-coverage question, where an exit code is too coarse.
One caveat that applies to both: confirm the probe file is actually being scanned, by watching the
tool's *scanned-file* total move. A probe that never compiled and a probe the tool never discovered
both report zero hits, and neither is a finding.
## What to actually do
- Before trusting a ratchet's number, run it once with a deliberate violation and confirm it goes red.
A ratchet nobody has seen fail is a ratchet nobody has verified.
- When probing what a tool can see, use `pnpm check:*`, not a bare `node scripts/...`.
- Match the probe to the question: `pnpm check:*` for **"can this ratchet fail?"** (§1 — always run
it through the CI entry point, never a bare `node scripts/...`), and the per-file **count** probe for
**"what shapes can it see?"**. The count cannot answer the first question and the exit code cannot
answer the second.
- Prefer one discovery mechanism across a family of tools. If a tool must differ, say so in its header
and say why, so a cross-tool differential is readable.
- `git ls-files --cached --others --exclude-standard` is the tracked-plus-untracked form (dedupe the