From d14294b6cb2ceef033514956b47521a8f08bbd70 Mon Sep 17 00:00:00 2001 From: gsxdsm Date: Fri, 31 Jul 2026 16:03:58 -0700 Subject: [PATCH] docs(solutions): add the count-based probe, which the report-only trap cannot fool (#3257) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## What Adds one technique to #3255. Docs only. #3255 records that probing a ratchet **by exit code** can read green because the tool is report-only without `--strict` — a real trap that nearly got a healthy gate reported as dead. There is a second technique that sidesteps it entirely and is strictly more informative: **parse the tool's own per-file count.** ```bash node scripts/check-move-target-literals.mjs 2>&1 | grep -a "my-probe-tmp" \ | grep -aoE "^ +[0-9]+" | tr -d ' ' ``` **Immune to the report-only trap** — a report-only run still *prints* the count, so the number moves 0 → 1 whether or not `--strict` was passed. **It measures which shapes, not just whether something fired.** An exit code is one bit for the whole run. Auditing a detector means asking *"of these five spellings, which are seen?"*, and five separate binary runs cannot distinguish **partial** detection from a probe file that failed to compile. The move-target audit read `direct 1 / backtick 1 / ternary 0 / const 0` in a single run, which named the gap immediately. ## Both belong | question | technique | |---|---| | **can this ratchet fail at all?** | `pnpm check:*` — ask this first (#3255 §1) | | **what can it see?** | per-file counts — an exit code is too coarse | I also added a caveat that applies to both: confirm the probe is actually being scanned by watching the tool's **scanned-file total** move. A probe that never compiled and a probe the tool never discovered both report zero hits, and neither is a finding — that one cost me a wasted measurement before I noticed the total had stayed at 1961. ## Why this is worth a follow-up rather than a comment #3255's rule as written — *"use `pnpm check:*`, not a bare `node scripts/...`"* — would have made the shape-coverage audits impossible, since `--strict` collapses five distinct per-form answers into one bit. The rule is right for its question and wrong for the other one, and the distinction is easy to lose once only the rule survives in someone's memory. ## Summary by CodeRabbit * **Documentation** * Added guidance for evaluating ratchets using per-file output counts. * Documented report-only and shape-coverage limitations, count-based versus failure-based checks, and verifying that probe files were scanned. * Included a command example for probing ratchet behavior. --------- Co-authored-by: Claude Opus 5 (1M context) --- ...probe-the-instrument-the-way-ci-runs-it.md | 47 ++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/docs/solutions/workflow-learnings/probe-the-instrument-the-way-ci-runs-it.md b/docs/solutions/workflow-learnings/probe-the-instrument-the-way-ci-runs-it.md index c44d43218d..00c415798a 100644 --- a/docs/solutions/workflow-learnings/probe-the-instrument-the-way-ci-runs-it.md +++ b/docs/solutions/workflow-learnings/probe-the-instrument-the-way-ci-runs-it.md @@ -71,11 +71,56 @@ When instruments in one program disagree about their own domain, every different unreadable until someone notices. Align the scopes, or the next person spends a day attributing a discrepancy to the thing being measured instead of to the measuring. +## Read the reported COUNT, not the exit code + +There is a second probing technique that sidesteps §1 entirely. It is **more informative for shape +coverage** — not strictly more informative, since it cannot answer §1's question at all: a count tells +you what the detector saw, never whether the ratchet would have failed on it. + +Instead of asking "did the tool exit non-zero", parse the tool's own **per-file count** out of its +output: + +```bash +# Capture output and status SEPARATELY: the pipeline below ends in `tr`, so a piped form reports +# `tr`'s success and a crashed detector reads as a clean zero — the failure this document is about. +out=$(node scripts/check-move-target-literals.mjs 2>&1); rc=$? +[ "$rc" -le 1 ] || { printf 'detector failed (rc=%s)\n%s\n' "$rc" "$out" >&2; exit 1; } + +count=$(printf '%s\n' "$out" | grep -a "my-probe-tmp" | grep -aoE "^ +[0-9]+" | tr -d ' ') +[ -n "$count" ] || { printf 'no matching line — probe not scanned?\n%s\n' "$out" >&2; exit 1; } +printf '%s\n' "$count" +``` + +`rc -le 1` because these ratchets use 1 for "violations found" and 2+ for "could not run"; a missing +match is reported rather than silently returning empty, since an absent line and a zero count are +different findings. + +Two properties matter: + +- **Immune to the report-only trap.** A report-only run still prints the count. The number moves from + 0 to 1 whether or not `--strict` was passed, so a missing flag cannot fake a pass. +- **It measures WHICH SHAPES, not just whether something fired.** An exit code is one bit for the whole + run. Auditing a detector means asking "of these five spellings, which are seen?" — and five separate + binary runs cannot distinguish *partial* detection from a probe file that failed to compile. The + count answers per form, in one run: the move-target audit read `direct 1 / backtick 1 / ternary 0 / + const 0`, which named the gap immediately. + +The two techniques answer different questions and both are worth keeping. Use `pnpm check:*` to ask +**"can this ratchet fail?"** — the §1 question, and the one to ask first. Use per-file counts to ask +**"what can it see?"** — the shape-coverage question, where an exit code is too coarse. + +One caveat that applies to both: confirm the probe file is actually being scanned, by watching the +tool's *scanned-file* total move. A probe that never compiled and a probe the tool never discovered +both report zero hits, and neither is a finding. + ## What to actually do - Before trusting a ratchet's number, run it once with a deliberate violation and confirm it goes red. A ratchet nobody has seen fail is a ratchet nobody has verified. -- When probing what a tool can see, use `pnpm check:*`, not a bare `node scripts/...`. +- Match the probe to the question: `pnpm check:*` for **"can this ratchet fail?"** (§1 — always run + it through the CI entry point, never a bare `node scripts/...`), and the per-file **count** probe for + **"what shapes can it see?"**. The count cannot answer the first question and the exit code cannot + answer the second. - Prefer one discovery mechanism across a family of tools. If a tool must differ, say so in its header and say why, so a cross-tool differential is readable. - `git ls-files --cached --others --exclude-standard` is the tracked-plus-untracked form (dedupe the