FN-9125: document flake findings and quarantine plugin runner

Classify the suite-only failures by actual PostgreSQL dependency and preserve unresolved evidence for follow-up.

- Record non-reproduction results and assign PostgreSQL investigations to focused follow-up tasks.
- Quarantine the independent in-memory plugin runner test under the deletion ratchet.
- Document evidence requirements for future PostgreSQL flake diagnosis.

Files changed:
 .../suite-only-flakes-observed-register.md         | 32 ++++++++++++++++++++--
 docs/testing.md                                    |  4 +++
 packages/engine/vitest.config.ts                   | 10 +++++++
 scripts/lib/test-quarantine.json                   |  8 +++++-
 4 files changed, 51 insertions(+), 3 deletions(-)

Fusion-Task-Id: FN-9125

Fusion-Task-Lineage: 1dc80163-a0dc-4241-bab1-75a2cafb9abe

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>

docs/solutions/test-failures/suite-only-flakes-observed-register.md

@@ -35,6 +35,13 @@ This register preserves first-sighting evidence under the narrow exception in [A
 | full core suite (3rd) | 4825 passed |
 | file alone ×2 | 6 passed, 6 passed |
 
+**Evidence gathering pending 2026-08-16 (FN-9125):** Current-sha diagnosis did not reproduce this historical first sighting: three six-worker full-core lanes and a twelve-worker PostgreSQL-directory run retained full output without this subject failing. The harness uses a shared golden template plus per-module copies, but no direct evidence tied this identity's null read to shared state. This is not superseded or resolved: the required complete loaded failure capture is absent. Core PostgreSQL quarantine is policy-forbidden, so FN-9126 owns CI/host-specific activity instrumentation, full failure capture, and the escalation decision.
+
+| verification | result |
+|---|---|
+| full core ×3, 6 workers | subject passed; unrelated settings-revision-attribution failure |
+| PostgreSQL directory, 12 workers | subject passed; unrelated satellite-store ordering failure |
+
 ## 2. Schema applier retains registered dependents
 
 - **File:** `packages/core/src/__tests__/postgres/schema-applier.test.ts`
@@ -47,6 +54,13 @@ This register preserves first-sighting evidence under the narrow exception in [A
 | file alone ×2 on the same tree | 75 passed, 75 passed |
 | file alone on `origin/main` | passed |
 
+**Evidence gathering pending 2026-08-16 (FN-9125):** Current-sha loaded reproduction did not fail this assertion. This file still owns an inline unique `CREATE DATABASE` plus full baseline path rather than the shared template harness, but that is a distinct cost profile, not evidence that it caused the historical dependent-registration failure. This is not superseded or resolved: the required complete loaded failure capture is absent. FN-9128 exclusively owns entry 2's CI allocation/profile investigation and full failure capture; core PostgreSQL quarantine is policy-forbidden.
+
+| verification | result |
+|---|---|
+| full core ×3, 6 workers | subject passed; unrelated settings-revision-attribution failure |
+| PostgreSQL directory, 12 workers | subject passed; unrelated satellite-store ordering failure |
+
 ## 3. Plugin runner complete-lane lifecycle hook
 
 - **File:** `packages/engine/src/__tests__/plugin-runner.test.ts`
@@ -62,6 +76,13 @@ This register preserves first-sighting evidence under the narrow exception in [A
 
 Seven tests failed in `plugin-runner.test.ts`, but only this one identity survived capture: `--reporter=dot | tail -3` truncated the `FAIL` lines and retained only the summary.
 
+**Quarantined 2026-08-16 (FN-9125):** Source inspection proved this unit file uses a local mocked TaskStore and has no PostgreSQL or harness import, so it does not belong to the database cluster. Three current full-engine lanes did not reproduce it, but the historical loaded failure lacks enough identities for a structural repair. The deletion-ratchet ledger and engine-default exclude were added together; assertions and timeouts are unchanged.
+
+| verification | result |
+|---|---|
+| full engine ×3, 6 threads | subject passed; 35–36 unrelated baseline-red files remained |
+| targeted plugin-runner | covered by subsequent quarantine-ledger verification |
+
 ## 4. Planning Mode direct task handoff
 
 - **File:** `packages/dashboard/app/components/__tests__/PlanningModeModal.planning-flow.test.tsx`
@@ -93,9 +114,9 @@ The failure exercises the pre-existing mobile tab transition, while the task-cre
 
 **Suite re-admitted 2026-08-10 (FN-8936):** This first-sighting mobile observation did not receive a second failure. The shared file-level quarantine was removed only after the direct-handoff root cause was structurally fixed and the unexcluded loaded suite, including this mobile coverage, passed.
 
-## Common shape and unverified suspicion
+## Common shape and investigated result
 
-Entries 1–3 are PostgreSQL-backed or PostgreSQL-suite-adjacent, pass in isolation, and appeared only under full-suite parallelism. This points at shared database state between those test files rather than any one test. It is **unverified and uninvestigated**, not a diagnosis; do not infer a root-cause fix from this record. Entry 6 instead records a merge-gate eviction after a loaded-lane setup-hook timeout; `FNXC:PgTestTemplateDb 2026-07-19-17:20` and `FNXC:PgTestWorkerCap 2026-07-18-18:00` are already-landed mitigations for that mode, not new diagnoses to re-open. The Planning Mode entries are separate frontend timing observations.
+FN-9125 established that entry 3 is not PostgreSQL-suite-adjacent: `plugin-runner.test.ts` uses an in-memory mocked TaskStore and has no PostgreSQL/harness import. Its historical loaded-engine failure is quarantined under the normal deletion ratchet. Entries 1, 2, and 7 remain evidence-gathering-pending PostgreSQL observations: current full-output runs at six workers plus a twelve-worker PostgreSQL-directory run did not reproduce any subject identity, so FN-9125 cannot claim them superseded or resolved. The golden-template/advisory-lock lifecycle and schema-applier's inline baseline path are concrete architecture facts, not a demonstrated cause of these assertions. Core policy forbids inline PG quarantine: FN-9126 owns entry 1, FN-9128 exclusively owns entry 2, and FN-9127 owns entry 7 for CI/host-specific `pg_stat_activity`, lifecycle timing, and a complete loaded failure capture before any source or fan-out change. Entry 6 instead records a merge-gate eviction after a loaded-lane setup-hook timeout; `FNXC:PgTestTemplateDb 2026-07-19-17:20` and `FNXC:PgTestWorkerCap 2026-07-18-18:00` are already-landed mitigations for that mode, not new diagnoses to re-open. The Planning Mode entries are separate frontend timing observations.
 
 ## Policy and escalation
 
@@ -137,6 +158,13 @@ FN-8928 evicted the file from the blocking gate under the AGENTS.md gate rule; d
 
 The timeout occurred after all test assertions and is unrelated to FN-8979's canonical mission-blocker contract. This file retains substantial coverage, so this first observation is recorded rather than quarantined. A second sighting requires the normal file-level quarantine decision.
 
+**Evidence gathering pending 2026-08-16 (FN-9125):** The shared-harness teardown is serial (store, layer, admin client, `DROP DATABASE WITH (FORCE)`, temporary directory), so a loaded close/drop block remains a plausible historical mechanism. Three targeted dot-reporter runs and loaded core reproduction did not produce a timeout or a measurable slow phase. This is not superseded or resolved: the required complete loaded failure capture is absent. FN-9127 owns CI/host-specific phase instrumentation, full failure capture, and the escalation decision; core PostgreSQL quarantine is policy-forbidden.
+
+| verification | result |
+|---|---|
+| targeted dot reporter ×3 | 61 tests passed; afterAll passed |
+| full core ×3, 6 workers | subject passed; unrelated settings-revision-attribution failure |
+
 ## 8. Planning Mode duplicate-response generation reconciliation
 
 - **File:** `packages/dashboard/app/components/__tests__/PlanningModeModal.planning-flow.test.tsx`

docs/testing.md

@@ -461,6 +461,10 @@ Legitimate legacy exceptions must be recorded in `scripts/lib/test-timeout-appea
 
 **2026-06-19 CLI affected-lane rescue (FN-6734):** a broad `@runfusion/fusion` lane reported default 5s test-body timeouts and `fusion-test-workers-*`/fixture `ENOTEMPTY` cleanup noise while isolated files exposed closeable real-store handles and a runtime-dist mock that was sensitive to package-lane module graph ordering. The rescue closed each real `TaskStore`/`AgentStore` before removing its temp fixture, kept task-list truncation coverage under the default timeout by reducing filler size rather than assertions, and preloaded the built `@fusion/core` barrel with `vi.importActual` before `vi.doMock` so complete dist artifacts exercise the CLI surface while partial stale dist skips cleanly. Prove this class with targeted file runs, `pnpm --filter @runfusion/fusion test`, the timeout-appeasement guard, bounded temp-prefix cleanup output, and the normal workspace gate/build; leave the CLI quarantine array empty when no file is actually quarantined.
 
+<!-- FNXC:PostgresFlakeDiagnosis 2026-08-16-17:30: FN-9125 requires a suspected PostgreSQL cluster to be split by actual imports and retained full runner output before changing the shared harness. A mocked engine unit with no harness import is an independent deletion-ratchet decision; current non-reproduction of a core PG first sighting is escalated because policy keeps the core PG quarantine array empty. -->
+
+**2026-08-16 PostgreSQL first-sighting diagnosis (FN-9125):** First classify each file by its real dependency path, not nearby failure timing. Retain complete `tee` output for repeated package lanes and record fan-out, server capacity, and target identities. If a current run cannot link a PostgreSQL test assertion to golden-template, DDL, pool, or teardown evidence, do not add timeouts/retries or infer a harness change: obtain CI/host activity and phase timings. A core PostgreSQL file is escalated rather than quarantined because `quarantinedCoreTests` remains policy-pinned empty; a non-PG engine file follows normal ledger-plus-exclude lockstep.
+
 <!-- FNXC:EngineTestReliability 2026-06-27-10:05: FN-7119 rescued the 2026-06-26 engine scheduler/reliability quarantine burst by completing local TaskStore fakes for the scheduler heartbeat `updateSettings({ engineLastActiveAt })` write before adjusting any call-count assertions. When a scheduler batch reports zero mock calls or missing audit events after a heartbeat-era scheduler change, first mirror the production store surface in shared fakes and re-run the exact files together under `engine-default` / `engine-reliability`; do not weaken call-count invariants or quarantine ledger/config rows after the fake drift is fixed. -->
 
 <!-- FNXC:TestQuarantine 2026-06-19-14:15: FN-6740 audited the same-day quarantine ledger as a coordinated deletion-ratchet batch. The ledger had 14 entries (3 dashboard, 6 core, 5 CLI) and every entry was mirrored in its package Vitest exclude; keep follow-up rescue/delete work scoped by subsystem so ledger/config edits remain lockstep and do not collide. -->

packages/engine/vitest.config.ts

@@ -336,6 +336,16 @@ export default defineConfig({
             // / `test:all` invoked from the root `test:full` script.
             "src/**/*.slow.test.ts",
             /*
+            FNXC:PluginRunnerFlake 2026-08-16-17:30:
+            FN-9125 confirmed this in-memory mocked unit file has no PostgreSQL
+            dependency, but historical loaded-engine evidence captured seven
+            failures without sufficient identities for a structural fix. Quarantine
+            the whole file under the deletion ratchet instead of weakening its
+            lifecycle assertions; scripts/lib/test-quarantine.json is the paired
+            dated ledger record.
+            */
+            "src/__tests__/plugin-runner.test.ts",
+            /*
             FNXC:FullSuiteBookkeeping 2026-08-09-03:49:
             All 11 engine-default entries from the 2026-08-05 full-suite quarantine wave (run 30982276306) were deleted under the deletion ratchet after operator directive. These tested pre-refactor APIs (getBuiltinWorkflow removed post-U10b), stale mock shapes, census/allowlist drift, and mock-hoist errors that no longer have a production path to exercise.
             */

scripts/lib/test-quarantine.json

@@ -1,4 +1,10 @@
 {
   "$comment": "Flaky-test quarantine ledger (deletion ratchet — see AGENTS.md 'Flaky tests: quarantine on sight' and docs/testing.md 'Quarantine ledger and the deletion ratchet'). A test observed failing without a corresponding real bug is quarantined ON SIGHT: add an entry here AND a matching one-line `exclude` entry in that package's vitest config, in the same commit. Every entry needs a non-empty `reason` (link the failing run) and a `quarantinedAt` date — the entry expires 14 days later, at which point the test file is DELETED unless someone rescues it with evidence it catches real regressions plus a root-cause fix (never appeasement). There is deliberately no loader module and no automation around this file: it is a dated record, the vitest config is the enforcement.",
-  "entries": []
+  "entries": [
+    {
+      "file": "packages/engine/src/__tests__/plugin-runner.test.ts",
+      "reason": "FN-9125: first-sighting register entry 3; failure evidence https://github.com/Runfusion/Fusion/pull/2799#issuecomment-5134921769 captures seven loaded-engine failures on the PR #2799 merged-with-main tree, but only `PluginRunner > task lifecycle hooks > should invoke onTaskCompleted when the complete lane is RENAMED` survived because the original dot output was truncated. FN-9125 confirmed the file is in-memory/mocked and not PostgreSQL-backed; no current structural failure was reproducible. Quarantined under the deletion ratchet rather than weakening lifecycle assertions.",
+      "quarantinedAt": "2026-08-16"
+    }
+  ]
 }