## What
A new `docs/solutions` note recording a failure that hit **three
different tools in one session**, each time reading as a pass. Docs
only.
## The three costumes
| what happened | looked like | was |
|---|---|---|
| `git stash --keep-index` swept the new test file out of the tree | "45
passed" | the pre-existing count; the new test never ran |
| a blinding script hit an unmapped role and `sys.exit(2)` **with no
message**; `&&` skipped the check, `;` let the run proceed | "375/375
green under blinding" | nothing blinded — run was against unmodified
source |
| a gate piped to `tail -1`, printing a blank line | "gate ran, no
complaints" | exit code 1; the FNXC stamp check had failed, and **CI
caught it in #3238** |
## Why it deserves its own note
**A passing run and a run that never happened produce the same evidence:
no failure text.** Every other bug announces itself; this one is defined
by the absence of an announcement. The instinct that catches ordinary
bugs — *"nothing looks wrong"* — is precisely the instinct that
certifies this one.
It gets worse under automation, where output is piped and skimmed. `|
tail -1`, `| grep "Tests"`, `>/dev/null 2>&1` all discard the part that
would have said `No test files found` or `command not found`.
## The five rules, each paid for above
1. **Assert the exit code before any pipe.** A pipeline's status is the
*last* stage's — `cmd | tail -1` reports `tail`'s success, never
`cmd`'s.
2. **Confirm the run did the work.** "Test Files 1 passed" when you
expected 16 is a finding, not a pass.
3. **A tool that can no-op must say what it did** — print the
substitution and location, fail loudly where it cannot act.
4. **Verify the mutation, not the tool's promise** — `git diff --stat`,
not the exit code.
5. **Break the guard on purpose once** and watch it fail. A guard never
observed failing has not been shown to work — the standard this repo
already applies to product ratchets, turned on your own verification.
## The uncomfortable part, kept in
The third instance was a rule **I added to AGENTS.md myself in #3174**,
broken for the second time. I ran the gate. I read `tail -1`. I moved
on.
Writing a rule down does not make you follow it. The only reason it was
caught is that **CI read the output when I did not** — an argument for
the gate existing, not for me having been careful.
Cross-linked from the resolver-audit note, whose every wrong reading
came from a run that never happened rather than from the blinding
itself. That connection is the point: I spent this session auditing a
program whose subject is defects hiding behind green results, and
reproduced the same class three times in my own tooling.
```
lint clean; fnxc-future-dates: none added (exit code checked before piping this time)
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added workflow guidance explaining why silent or seemingly successful
output does not confirm that a test, script, or validation gate ran.
* Documented verification practices including checking exit codes, work
counts, no-op detection, post-run changes, and intentional failure
checks.
* Added a case study highlighting how filtered output can conceal
verification failures.
* Added cross-references connecting resolver interpretation, test
execution, and conversion coverage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
851369a480