Measured 14 backend connections against max_connections=100, so the api-lane
hook timeouts are not connection exhaustion; the PostgreSQL Failed query lines
are a torn-down reconciler polling after the fact. Full 15-lane run at 23,584
tests reproduced nothing, and the DDL admission gate never degraded, so that
mechanism is unsupported too. Records what a future attempt must capture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three consecutive full-lane runs on the same tree each failed a DIFFERENT file,
every one passing in isolation, with hook timeouts arriving alongside PostgreSQL
'Failed query' warnings from two api lanes sharing one database. That is the same
class FN-9131 investigated for core's loaded PostgreSQL directory.
Deliberately not quarantined: quarantine is file-level and the failing file moves,
so it would evict healthy coverage without touching the cause. Recorded with the
evidence so the next person does not re-derive it, and so the rescue is aimed at
the lane runner's connection/concurrency budget rather than at whichever test lost
the race that run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Running the dashboard's REAL test command (`run-quality-tests.mjs`, which shards
into 15 lanes) surfaced two failures that a plain `vitest run` never shows —
worth noting on its own, since measuring around a package's own command is how
a suite gets called green on a number the project does not produce.
`scripts/__tests__/run-quality-tests.test.ts` spawns the package's own
`pnpm --filter @fusion/dashboard test` to prove the package-command wiring. Inside
a full lane run that child inherited a Corepack environment that stopped to ask
about downloading pnpm, so it never launched, the lane log came back empty, and
the assertion read 0 launched projects. It passed in isolation only because that
shell had already resolved pnpm. Both spawn sites now set
COREPACK_ENABLE_DOWNLOAD_PROMPT=0; nothing about the assertions changed.
Also records PlanningModeModal.planning-flow as a suite-only flake rather than
forcing it green: it fails only in lane `app:backfill-3` under four concurrent
6GB shards, passes 83/83 in isolation, and picked a DIFFERENT case on each of two
runs — a render-settle timing problem, not a product defect. Recorded as a first
sighting; a repeat of the same case is an on-sight quarantine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`handoff-to-review-atomicity.pg.test.ts` aborted its `beforeAll` at the 15s
budget on the first `pnpm test:gate` of a session; not reproduced in 8 later
runs across three shapes (gate x2, pg-gate x3, isolated x3).
Same mode as entries 6 and 7, but narrower: it happened under the capped
four-fork lane with two selected files, so fork oversubscription does not
explain it. Recorded the cold-cluster correlation as a hypothesis rather than a
finding — reproducing it means stopping the embedded cluster, and this host runs
a live Fusion instance.
Discloses that the failing run's full output was lost to a tail pipe, and that
inline quarantine was unavailable regardless (quarantinedCoreTests must stay
empty); eviction of a transactional-invariant gate file is owner-escalated.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Full engine suite at a97aa84a20: 3 failures out of 12,414. All three diagnosed:
- self-healing-pending-wedge-notification's marker-selection case fails ONLY in
a full-suite run (expects 1 elapsed marker, sees 2) and passes deterministically
alone. This is its SECOND sighting, so per AGENTS.md it is an on-sight
quarantine with no further discretion: ledger entry + matching vitest exclude,
same commit, 2026-09-06 deletion deadline.
- spec-drift-reconciler's exponential-backoff case shows the same shape on a
FIRST sighting, so it is recorded in the observed register instead of evicting
that file's other passing coverage. Both are timer-driven reconciler tests that
only fail alongside other suites, pointing at cross-file fake-timer state.
- merge-orphan-durable-write-inventory drift was pure lineHint movement (19
changed, zero newly unclassified entries) after product edits shifted lines.
Regenerated.
Also fixes check-quarantine-ledger.mjs, which could not see the exclude I added:
its comment stripper treated the `/**` inside glob literals like "node_modules/**"
and "src/**/*.slow.test.ts" as a block-comment opener and deleted through to the
next "*/", swallowing whole array literals and every entry after them. It now
scans string-aware, so the lockstep check actually holds. Nothing was appeased:
no timeout widened, no retry added, no assertion relaxed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The engine suite's failures are not independent bugs; they are a handful of
systemic drifts where a guard shipped and its fixtures were never updated.
- Required pre-merge gates (FN-158): the door refuses a card whose enabled
optional groups produced no result, and the built-in workflow enables Plan and
Code Review by default. Merge-mechanics fixtures now declare an explicit empty
list, stating the intent they always had. group-merge-coordinator's
"post-Code-Review member" instead gets real PASSING workflowStepResults,
because recording the pass is what that fixture actually means.
- Branch-write provenance: the shared reliability fixture creates a task with a
branch, which now requires an explicit origin. It stands in for an
engine-created branch, so it says so.
- updateTaskAtomic: a production write seam missing from several fake stores,
copied from the faithful fake in merger-ai.test.ts.
- Durable-write inventory: eight unclassified TaskStore surfaces classified,
including the two batched reads this branch added.
- workflow-graph-merge-region-collapse asserted completion-summary AFTER
code-review; the IR wires it before, and production logs agree.
merger-ai.test.ts alone goes 37 -> 0. Engine failures 288 -> ~200.
Also records a first-sighting suite-only flake in the observed register per the
standing rule, rather than quarantining a file with substantial coverage.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reorganize the suite-only flake register around its current decision state.
- Separate active observations, gate evictions, and archived closed records while preserving entry anchors.
- Mark entry 1 as an escalated second sighting and restore FN-9126/FN-9131, FN-9128, and FN-9127 ownership.
- Add regression checks for active counts, exact states and owners, and testing-guide anchors.
Files changed:
.../suite-only-flakes-observed-register.md | 189 +++++++++++++--------
scripts/__tests__/observed-flake-register.test.mjs | 65 +++++++
2 files changed, 181 insertions(+), 73 deletions(-)
Fusion-Task-Id: FN-9145
Fusion-Task-Lineage: 39b3292a-db48-4be4-878c-7d2b026abae3
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
**Problem:** Each dashboard/engine project consumer polled
`task_deleted` outbox on a fixed 5s setInterval, so ~44 per-project
consumers thundered together on the same cadence — an idle DB query
storm and CPU hot-spot even when projects were paused/idle.
**Fix:** The outbox consumer reschedules itself from each poll outcome:
an idle poll (zero events) grows the next delay by
`TASK_DELETED_OUTBOX_BACKOFF_STEP_MS` toward `MAX_POLL_MS`, with ±20%
jitter so the consumers de-synchronize; a poll that delivered events
resets to the fast base. A paused/idle project drains its outbox and
backoff alone drops the DB load.
**Includes:** regression test (bounded jitter + idle growth),
performance changeset, solution doc, deploy handoff script.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance**
- Reduced unnecessary idle polling by gradually increasing the polling
interval, up to 60 seconds, with bounded timing variation.
- Restored the faster 5-second polling cadence when new events, waits,
or transient errors occur.
- Preserved event ordering, delivery guarantees, acknowledgements, and
independent behavior across concurrent consumers.
- **Documentation**
- Added guidance on polling behavior, deployment verification, and
monitoring targets.
- **Tests**
- Added coverage for backoff growth, jitter limits, event bursts,
concurrent consumers, error handling, retries, and clean shutdown.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Problem:** Scheduler was re-reading each task's
`task_workflow_selection` once per park-resolution (sweep, hold-release,
moved, unpause/wake), causing a nonstop PostgreSQL query storm (~232
idx_scan/s) on idle polling — a major engine CPU hot-spot.
**Fix:** Memoize the workflow selection per scheduler tick/event —
thread a shared, per-event selection cache through
`resolveWorkflowIrForTask` and all park-resolution handlers, then throw
it away. Each task resolves its parked columns with at most one read of
`task_workflow_selection` per tick. A selection write is always observed
on the next event's fresh cache (never a global/infinite LRU).
**Includes:** regression test asserting the once-per-tick read
invariant, performance changeset + per-tick-cache solution doc,
deploy+verify handoff script, and the parallel quarantine-ledger merge
(origin FN-9125 + RUFU-072 OOM entries both retained).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance Improvements**
* Reduced repeated workflow-selection reads during scheduler ticks and
related event processing.
* Improved scheduler and health API responsiveness through per-operation
caching and read deduplication.
* Preserved existing behavior, including retry handling for failed reads
and synchronous data-store support.
* **Documentation**
* Added architectural guidance covering workflow-selection performance,
caching behavior, and verification criteria.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Fusion <noreply@runfusion.ai>
Preserve the quarantined plugin-runner coverage while recording the inconclusive reproduction campaign.
- Record two loaded-suite runs at each of 2, 6, and 8 workers.
- Keep all 82 plugin-runner tests, the ledger entry, and the default-lane exclusion through 2026-08-30.
- Clarify that opt-in strict checks do not block normal PR lanes.
Files changed:
.../test-failures/suite-only-flakes-observed-register.md | 13 ++++++++++++-
docs/testing.md | 3 +++
2 files changed, 15 insertions(+), 1 deletion(-)
Fusion-Task-Id: FN-9135
Fusion-Task-Lineage: 239425e5-7393-4b8a-a114-0a14f0065847
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Document the second sighting of the project-identity PostgreSQL timeout and hand off structural diagnosis.
- Record the sterile 27-worker reproduction against the integration SHA
- Preserve pass, failure, and skip counters for both PostgreSQL-directory runs
- Mark quarantine as policy-forbidden and assign root-cause follow-up to FN-9131
Files changed:
.../solutions/test-failures/suite-only-flakes-observed-register.md | 7 +++++++
1 file changed, 7 insertions(+)
Fusion-Task-Id: FN-9126
Fusion-Task-Lineage: 9e31b00f-268b-46fa-af7a-58973c93d8fc
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Keep visible Planning Mode controls bound to the current turn during asynchronous refreshes.
- preserve active question and plan-review workspaces during same-session hydration
- submit from live session state and retain dirty answers across response identity changes
- add desktop and mobile regression coverage for deferred hydration and stale Stop polling
- document the ownership race and add a patch changeset
Files changed:
.changeset/fn-9117-planning-turn-ownership.md | 7 +
.../suite-only-flakes-observed-register.md | 24 ++++
.../dashboard/app/components/PlanningModeModal.tsx | 141 ++++++++++++---------
.../PlanningModeModal.planning-flow.test.tsx | 132 +++++++++++++++++++
.../PlanningModeModal.ui-interactions.test.tsx | 20 +++
5 files changed, 265 insertions(+), 59 deletions(-)
Fusion-Task-Id: FN-9117
Fusion-Task-Lineage: 18d63f63-b532-49eb-a6d1-241e31c5b2c7
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
One clean solo-lane sighting of the parametrized duplicate-response
reconciliation test (mobile row; the desktop row failed once earlier
under a contaminated concurrent run, recorded as context). Third
distinct flaky test in this suite plus one prior FN-8936 stabilization
— the AGENTS.md repeated-quarantine rule calls that a subsystem
product-race smell, so the product-path audit is filed as FN-9116
rather than stabilizing a fourth time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Confirm the W33 spike as cold-start variance and preserve a repeatable diagnosis protocol.
- Record five sequential phase-timed samples and the 20.5-second median threshold.
- Link the controlled remeasurement protocol from the testing guide.
- Explain the timing snapshot handoff and why incomplete CI artifacts were not published.
- Preserve the no-appeasement requirement beside boot-smoke phase timing.
Files changed:
.../boot-smoke-w33-walltime-anomaly.md | 100 +++++++++++++++++++++
docs/testing.md | 2 +-
scripts/boot-smoke.mjs | 7 ++
3 files changed, 108 insertions(+), 1 deletion(-)
Fusion-Task-Id: FN-9105
Fusion-Task-Lineage: 6956b037-db0f-4560-bb04-136080a975b5
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Third incident of the same class (#1857/FN-7391, FN-9101, GitHub #3462):
Fusion's Anthropic auth-card/storage ids (anthropic-subscription,
anthropic-api-key) leaking into pi, which only registers the execution
provider 'anthropic'. FusionAuthStorage.login is the single seam that hands
a provider id to ModelRuntime.login; it now normalizes via
toExecutionModelProviderId so a future caller bug degrades to a correct
upstream anthropic login instead of a hard 'Unknown provider' failure, with
a regression test pinning the seam. The invariant, incident history, and
guard inventory are captured in
docs/solutions/integration-issues/anthropic-storage-ids-are-never-pi-provider-ids.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Triage announced specification completion before its finally block marked the
plan work item terminal, so the Plan Review seeder saw its own still-running
predecessor as an "active continuation", bailed, and the discarded result
silently stranded the card until FN-8592 self-healing re-seeded it ~10 minutes
later (529 occurrences in 18 days).
- seedStrandedPlanReviewContinuation gains retirePredecessorId: idle check
excludes the named predecessor, then retires it and installs the successor in
ONE transaction under the task lock; a bailed seed mutates nothing.
- triage threads planningWorkItemId through PlanningHandoffReport; the runtime
reaction passes it as retirePredecessorId.
- reactToSpecificationComplete consumes the seed result: typed quiet parks
(incl. new "no-pre-release-plan-review"), bounded retries with a fresh
task/IR snapshot per attempt (mid-retry pause/needs-replan honored), loud
warning naming self-healing on exhaustion.
- Tests: PG both-orderings/no-mutation-on-bail/cross-task cases, direct engine
seeder handoff cases, reaction retry/park/pause/replan cases.
- docs/solutions: new planning-handoff-race writeup; graph-entry-contract doc
reclassifies the FN-8592 sweep as backstop-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
