Session setup, track bookkeeping, intentional skill exclusions, token-cache
metrics, zero-count recovery summaries, and expected-missing PROMPT seed reads
were flooding the default log pane. Gate them behind FUSION_DEBUG so only
state transitions and operator-actionable warnings remain visible.
Quarantine the timing-sensitive Kimi K3 SDK catalog test without changing timeout budgets.
- Reuse the native model registry once per test file.
- Add the observed CI timeout to the dashboard quarantine ledger and config.
- Document validation and timeout-budget preservation requirements.
Files changed:
docs/testing.md | 8 ++++++++
...ister-model-routes-kimi-k3-supplemental.test.ts | 23 ++++++++++++++++++++--
packages/dashboard/vitest.config.ts | 8 ++++++++
scripts/lib/test-quarantine.json | 5 +++++
4 files changed, 42 insertions(+), 2 deletions(-)
Fusion-Task-Id: FN-8647
Fusion-Task-Lineage: 31e79677-d923-4003-a8e8-082159334e65
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
## What
My blind-spot table in #3251 audited **one axis**. Adds the one that
missed three defects. Docs only.
That table records what each of the five lifecycle ratchets can and
cannot **see**. I probed that carefully — several spellings per tool —
and then wrote *"nothing found; sound"* for two of them.
Within a day, three of those same tools turned out to share a completely
different defect: **they wrote to the tree they were checking**,
auto-tightening their own baseline during a plain check run.
| gate | wrote during a check | fixed by |
|---|---|---|
| `check-fnxc-future-dates` | yes | #3287 |
| `lifecycle-column-census` | yes, under `--strict` | #3289 |
| `check-sql-column-literals` | yes | #3292 |
**No number of detection probes could have surfaced that.** The table
asserted one property carefully and said nothing about the other *while
reading as comprehensive* — which is precisely the failure it documents
in the tools it audits.
## The rule it adds
1. **What can it see?** — probe each spelling of the thing it claims to
catch.
2. **Can it fail at all?** — invoke it as `package.json` does; a
report-only run exits 0 forever (#3255).
3. **Does it write?** — `git status --porcelain` before and after, on a
clean tree.
With the trap on the third spelled out: these gates write only when a
tightening is **available**, so a clean tree after a run proves the
*trigger* is absent, not that the tool is read-only. Inflate a baseline
entry first, then run it. I hit exactly this while reviewing #3292 — ran
all three gates on main, saw a clean tree, and had to stop myself
concluding the SQL gate was fine.
## Why the pattern, not the people
Three tools converged on write-during-check independently. That argues
the design is **attractive**, not that three authors were careless: the
tightening is correct, the write saves a step, and the message even
tells you to commit it. It only becomes a defect at the moment a second
person runs the same gate — which is invisible from inside any one of
them.
What it cost, measured: #3283 and #3285 are the same `+0/-1`, five
minutes apart, by two authors, **neither of whom wrote that line**.
```
lint clean; fnxc-future-dates clean
```
Extends the doc from #3255/#3273 with the failure that cost the most in
a single session: **one stale install produced five wrong reports on one
issue** (#3264).
## What happened
A `node_modules` that had drifted from the lockfile — `jsdom@29.0.1`
installed, `29.1.1` pinned — generated failures that existed on no CI
machine and no other checkout. They were not subtle: deterministic,
reproducible on demand, with plausible stack traces and real-looking
assertion diffs.
Each round of triage got **more precise about the wrong data**:
| round | claim | why it was wrong |
| --- | --- | --- |
| 1 | "4 deterministic failures" | measured in a 4-file batch, called it
isolation |
| 2 | "3 deterministic, 2 order-dependent" | isolated correctly, but a
race is not deterministic |
| 3 | "TaskCard is broken" | stale jsdom; the CSS assertion was correct
|
| 4 | "no contamination" | true of four app files; published unqualified
|
| 5 | "quarantine these two" | never read the failure text — both were
timeouts |
The through-line is not carelessness about the code. **The environment
was never treated as part of the claim**, so no amount of care about the
analysis could recover it.
## The checks, in the order they cost the most
```bash
pnpm install --frozen-lockfile # node_modules is not evidence until it matches the lockfile
<run the file ALONE, 3+ times> # isolation and repetition answer different questions
<read the failure TEXT> # a timeout and an assertion failure need opposite responses
uptime # a loaded box manufactures timeouts that mean nothing
```
## Why the load check earned its place
Two tests "failing" in a full-suite run were `Test timed out in 15000ms`
on a box at **load average 9.7 with 84 users**. Under AGENTS.md's
quarantine-on-sight rule that reads as a flake to quarantine — and the
ledger's **14-day deletion ratchet would have made the lost coverage
permanent**.
The rule presumes the failure is a property of the test, not of the
machine. A wall-clock budget crossed under local contention is evidence
about the hardware. I was one comment away from deleting healthy
coverage on that basis.
## The tell
A finding is environment-derived when it is **local, recent, and
unshared**: nobody else has reported it, CI is green, and it appeared
without a commit that could explain it. Any two of those should stop a
report before it is written. All three applied here, and the report went
out anyway — five times.
## Verification
Docs only; no code paths change. `fnxc-future-dates`,
`lifecycle-columns`, `quarantine-ledger` exit 0. No changeset — internal
docs are excluded.
Context: the one finding in #3264 that survived all five rounds is #3286
(merged), and it survived because it was verified by **reverting the
product change** rather than by trusting a red — 3/3/2 failures without
the fix, 27/27 across four runs with it.
Extends the doc merged in #3255 with two more instances of the same
pattern, both found this session, **neither involving a ratchet**. Four
instances now, from four unrelated directions:
| what was read as "pass" | what the green actually meant |
| --- | --- |
| `node scripts/check-*.mjs` exits 0 | report-only mode — the failure
path needs `--strict` |
| a census reports 0 for a new file | the file is untracked, so it was
never scanned |
| a backgrounded `cmd > log; grep …` reports exit 0 | that is `grep`'s
status; the suite inside had 8 failures |
| a rebased branch's tests pass | the rebase never started, so it ran on
the **old** base |
The two new ones are worth writing down because they are not about
tooling anyone built here — they are about how results are read.
**Exit codes belong to the last command in the pipeline.** A
backgrounded `run_tests > log 2>&1; echo done; grep X log` exits with
`grep`'s status, so the harness reported "completed, exit code 0" for a
dashboard suite that had 8 failures. I nearly recorded that suite as
green. Read the summary out of the log; never infer a suite's result
from a wrapper's exit code.
**A failed rebase leaves you on the old base, and the tests still pass
there.** `git rebase` refused with `cannot rebase: You have unstaged
changes`, so the branch never moved. `git diff origin/main` then listed
20+ files including other workers' commits — which reads exactly like my
branch had reverted their work — and a full test run on that tree came
back green. Both signals were true about a tree nobody cared about.
```
git merge-base --is-ancestor origin/main HEAD
```
said STALE while the tests said pass. That is the only check that
separates the two, and it belongs before any claim of "verified on
current main".
The shared tell, stated once: **a result too clean, or too alarming, for
what changed.** Every probe shape passing including ones that obviously
should not; a two-file branch appearing to revert twenty. When the
answer does not fit the size of the question, find out what was actually
measured before believing it.
## Verification
Docs only; no code paths change. `lifecycle-columns`,
`move-target-literals`, `inert-sync-lanes`, `quarantine-ledger` all exit
0. No changeset — AGENTS.md excludes internal docs.
**Pre-existing red, not from this branch:** `check:fnxc-future-dates`
currently fails on main from a `2026-08-01-00:50` stamp in
`packages/core/src/task-store/lifecycle-ops.ts` (commit `e52da740a5`) —
a timezone-ahead clock writing tomorrow's date, at 23:45 UTC. Already
claimed by **#3269 and #3270**, so I have not touched it; flagging only
so this branch's CI result is not misattributed. It is the same
recurring class this doc's sibling rule addresses: take the stamp from
`date -u`, not the local clock.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added guidance for identifying misleadingly successful CI and test
results.
* Documented checks for report-only runs, untracked files, masked
failures, and tests running on an outdated code base.
* Included recommendations for reviewing logs and verifying branch
ancestry.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What
Adds one technique to #3255. Docs only.
#3255 records that probing a ratchet **by exit code** can read green
because the tool is report-only without `--strict` — a real trap that
nearly got a healthy gate reported as dead. There is a second technique
that sidesteps it entirely and is strictly more informative: **parse the
tool's own per-file count.**
```bash
node scripts/check-move-target-literals.mjs 2>&1 | grep -a "my-probe-tmp" \
| grep -aoE "^ +[0-9]+" | tr -d ' '
```
**Immune to the report-only trap** — a report-only run still *prints*
the count, so the number moves 0 → 1 whether or not `--strict` was
passed.
**It measures which shapes, not just whether something fired.** An exit
code is one bit for the whole run. Auditing a detector means asking *"of
these five spellings, which are seen?"*, and five separate binary runs
cannot distinguish **partial** detection from a probe file that failed
to compile. The move-target audit read `direct 1 / backtick 1 / ternary
0 / const 0` in a single run, which named the gap immediately.
## Both belong
| question | technique |
|---|---|
| **can this ratchet fail at all?** | `pnpm check:*` — ask this first
(#3255 §1) |
| **what can it see?** | per-file counts — an exit code is too coarse |
I also added a caveat that applies to both: confirm the probe is
actually being scanned by watching the tool's **scanned-file total**
move. A probe that never compiled and a probe the tool never discovered
both report zero hits, and neither is a finding — that one cost me a
wasted measurement before I noticed the total had stayed at 1961.
## Why this is worth a follow-up rather than a comment
#3255's rule as written — *"use `pnpm check:*`, not a bare `node
scripts/...`"* — would have made the shape-coverage audits impossible,
since `--strict` collapses five distinct per-form answers into one bit.
The rule is right for its question and wrong for the other one, and the
distinction is easy to lose once only the rule survives in someone's
memory.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added guidance for evaluating ratchets using per-file output counts.
* Documented report-only and shape-coverage limitations, count-based
versus failure-based checks, and verifying that probe files were
scanned.
* Included a command example for probing ratchet behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Records two instrument-level defects found this session. Both were in
the tools the program uses as ground truth, and both looked exactly like
a pass.
## 1. A ratchet that could not fail from the command I typed
`check-move-target-literals` is report-only unless given `--strict`,
which `package.json` supplies. Probed bare, it returned **exit 0 for
every probe** — including a blatant `moveTask(id, "in-review")` pasted
into `scheduler.ts`.
That is the exact signature of a dead ratchet, and I nearly reported
another worker's guard as inert on the strength of it. The guard was
fine; my invocation could not fail. What makes it dangerous is the
output: a report-only run prints its normal summary line and exits 0, so
the terminal is indistinguishable from a genuine pass.
## 2. A ratchet that could not see the file I had just written
`lifecycle-column-census` and `check-move-target-literals` discovered
files with `git ls-files` — **tracked only** — while the other five walk
the filesystem.
| new file with a plain legacy guard | result |
| --- | --- |
| same guard in an already-tracked file | caught |
| new file, untracked | **missed, exit 0** |
| identical file, `git add`ed | caught, exit 1 |
The detectors are fine. The blindness is discovery, and it lands at the
one moment the number is consulted: add a helper, check your own work,
read zero, commit — and it surfaces later in someone else's CI run,
attributed to a push instead of to the edit. The tool was answering
about the last commit while being asked about the working tree.
## 3. Why it is worth a doc rather than two one-line fixes
Individually these are cheap. Together they cost a day.
Because `check-inert-sync-lane-conversions` walks the filesystem and the
census did not, the **same probe file** was caught by one and missed by
the other. I read that differential as a claim about expression walking
and investigated it as one — the real cause was that two instruments in
the same program disagreed about which files exist.
When the measuring tools disagree about their own domain, every
differential between them is unreadable until someone notices. That is
the transferable lesson, and it is not visible from either fix alone.
## Status of the fixes
- Census discovery scope: **#3254** (open).
- Type-assertion blind spot in the sync-lane ratchet: **#3252** (open).
- `check-move-target-literals` discovery scope: reported to **#3253**,
whose author is already in that file — not touching it.
## Verification
Docs only; no code paths change. All eight ratchets exit 0. No changeset
— AGENTS.md excludes internal docs.
## What
This note already prescribes: *"Before trusting a ratchet: mutate the
shape it claims to catch and confirm it exits non-zero."* This is that
checklist item **executed against all five lifecycle gates** on one
tree, one staged probe file per form. Docs only.
**Two of the five were wrong.**
| gate | catches | does NOT catch |
|---|---|---|
| `lifecycle-column-census` | `===` / `!==` | ~~membership, switch~~
**fixed (#3247)** |
| `check-move-target-literals` | direct + backtick destinations |
~~ternary~~ **fixed (#3250)**; still misses a destination bound to a
local |
| `check-sql-column-literals` | `"column"` comparisons — **including
plain template literals**, not only drizzle `sql` tags | nothing; the
one miss probed was an identifier the schema never uses |
| `check-inert-sync-lane-conversions` | lane reads via the
`resolvePlannerLanes` helper | a **direct**
`store.resolveTaskWorkflowIrSync(...)` read feeding
`resolveLifecycleColumns` — inert by the same mechanism, untracked |
| `check-fnxc-future-dates` | future stamps | nothing — it caught this
table's author, twice |
## The two lessons the table encodes
**A ratchet's blind spot is invisible in exactly the way its subject
is.** Both fixed gaps sat next to a printed zero *and a sentence
promising nothing could land silently*. The count was true. The sentence
was true only for the forms the parser happened to visit. That is the
same shape as the conversions this program spent weeks finding — code
that looks converted because the instrument cannot see the difference.
**Probe correctness is its own trap.** The first census probe measured
nothing: the scanner enumerates git-tracked files, the probe was
untracked, and the scanned-file count staying flat reads *exactly* like
"no gap". A `DELIBERATE-LITERAL` probe likewise read as a broken escape
hatch until the marker moved to its own line — mid-expression it
attaches to the wrong node, which is the documented gotcha, and it still
caught the person who had just written it down.
## Reported, not fixed
The inert-sync gap is left open deliberately: it is one narrow shape,
the only in-tree instance (`replan-target.ts:95`) is documented, new
conversions would use the tracked helper, and that gate has uncommitted
work from another worker. Recording it beats editing a file someone else
is mid-change on.
```
lint clean; fnxc-future-dates: none added; all five gates --strict green on this tree
```
## What
Follow-up to #3243. That note said *"take a second measurement of a
different kind"* — true, and useless at 2am without the technique. This
adds the one that actually settled every case, plus a fourth instance
that occurred after #3243 was written. Docs only.
## The technique
Enumerate recent failing CI runs and compute a **per-file failure
rate**. Seven runs separated three populations that are
indistinguishable from a single local run:
| rate on CI | meaning | action |
|---|---|---|
| **7/7** | consistent, real | fix, or diagnose and hand off with
evidence |
| **1/7** | intermittent | flake or race; two in one subsystem is a
product-race smell |
| **0/7** (fails only locally) | environment | fix your sandbox, change
**nothing** in the repo |
Measured on this repo's main while writing it: `planning-browser-e2e`
**7/7**, `postgres/schema-applier` **1/7**, `report-store.pg` **1/7**.
## The fourth instance
#3243 documented three reversals. A fourth happened after it merged: a
component test with **2 failing cases locally, 0/7 on CI**. That makes
**three separate local-only failures in a single session** — a
model-routes test hanging offline, a component test with four failing
cases, and a set of assertions I was ready to call a regression.
Each felt like a finding. All three were my sandbox. That is frequent
enough to be a habit rather than bad luck, which is why it is worth a
row in a table rather than a mention.
## The cost asymmetry, which should drive the default
Acting on a **0/7** by quarantining **deletes coverage that is green
everywhere else**. Acting on a **7/7** by investigating costs an hour.
The errors are not symmetric, so when unsure which row you are in, the
cheap move is always more samples from the *other* environment — not
more confidence about the one you have.
This is the concrete form of the point the standing quarantine rule
already encodes with *"without a corresponding real bug"*: **"I saw it
fail" is not that clause**, and the failure-rate table is how you tell
the difference before acting.
```
lint clean; fnxc-future-dates: none added (exit code checked before piping)
```
## What
A `docs/solutions` note recording three diagnoses I reversed **in one
session**, all wrong the same way. Docs only.
## The three
| observed | my story | what it was |
|---|---|---|
| `planning-browser-e2e` fails at width **769**, passes at **768** |
layout regression at the tablet breakpoint, from FN-8606 | a **race** —
5 passes in 6 runs; on every pass the control sits inside the viewport
at 769 (`right: 753 ≤ 769`) |
| a model-routes test fails **3 of 3** locally | red on main; quarantine
candidate | **green on CI**; a sandbox interaction. The fixture is
configured offline, so a sandbox should not have changed the outcome —
the tell was there from run one |
| one approach could not cover a resolver | the site is **unpinnable** |
a *different shape* covered it — a helper that **resolves** rather than
one that **receives** |
Each was plausible, mechanistic, and consistent with the evidence I had.
That is what made each dangerous: **a diagnosis that explains your one
data point feels finished.**
Each survived exactly until a second measurement **of a different kind**
— another environment, more samples, an instrumented probe. Re-running
the same command is not a second measurement.
## The reusable part
| observation | tempting story | check first |
|---|---|---|
| fails at boundary X, passes at X−1 | structural bug at the boundary |
run it 5 more times — boundaries are where races surface |
| **consistent** locally, green on CI | main is broken | the
environment; consistency is not universality |
| **intermittent** locally, consistent on CI | flaky test | a race the
slower runner loses every time |
| one approach failed | the site cannot be done | whether a different
*shape* of the approach works |
## Why it matters beyond debugging hygiene
Two of the three would have caused real damage if acted on:
- Quarantining the model-routes test — the action the standing rule
seems to license on "observed failing" — would have **deleted coverage
that is green everywhere else**. The rule's *"without a corresponding
real bug"* clause is load-bearing, and a local observation does not
satisfy it.
- "Unpinnable" hardened a single failed approach into a property of the
site. Left standing, it becomes a permanent excuse not to look — the
same failure I corrected in an inherited note earlier today, which had
recorded four resolvers as unmeasurable for environment reasons that did
not hold here.
Hence the last rule: **record cautions as environment-scoped, not as
properties of the code.** Say where you measured.
```
lint clean; fnxc-future-dates: none added (exit code checked before piping)
```
