diff --git a/docs/solutions/test-failures/suite-only-flakes-observed-register.md b/docs/solutions/test-failures/suite-only-flakes-observed-register.md index 07133eabe5..71edc7214a 100644 --- a/docs/solutions/test-failures/suite-only-flakes-observed-register.md +++ b/docs/solutions/test-failures/suite-only-flakes-observed-register.md @@ -35,6 +35,13 @@ This register preserves first-sighting evidence under the narrow exception in [A | full core suite (3rd) | 4825 passed | | file alone ×2 | 6 passed, 6 passed | +**Evidence gathering pending 2026-08-16 (FN-9125):** Current-sha diagnosis did not reproduce this historical first sighting: three six-worker full-core lanes and a twelve-worker PostgreSQL-directory run retained full output without this subject failing. The harness uses a shared golden template plus per-module copies, but no direct evidence tied this identity's null read to shared state. This is not superseded or resolved: the required complete loaded failure capture is absent. Core PostgreSQL quarantine is policy-forbidden, so FN-9126 owns CI/host-specific activity instrumentation, full failure capture, and the escalation decision. + +| verification | result | +|---|---| +| full core ×3, 6 workers | subject passed; unrelated settings-revision-attribution failure | +| PostgreSQL directory, 12 workers | subject passed; unrelated satellite-store ordering failure | + ## 2. Schema applier retains registered dependents - **File:** `packages/core/src/__tests__/postgres/schema-applier.test.ts` @@ -47,6 +54,13 @@ This register preserves first-sighting evidence under the narrow exception in [A | file alone ×2 on the same tree | 75 passed, 75 passed | | file alone on `origin/main` | passed | +**Evidence gathering pending 2026-08-16 (FN-9125):** Current-sha loaded reproduction did not fail this assertion. This file still owns an inline unique `CREATE DATABASE` plus full baseline path rather than the shared template harness, but that is a distinct cost profile, not evidence that it caused the historical dependent-registration failure. This is not superseded or resolved: the required complete loaded failure capture is absent. FN-9128 exclusively owns entry 2's CI allocation/profile investigation and full failure capture; core PostgreSQL quarantine is policy-forbidden. + +| verification | result | +|---|---| +| full core ×3, 6 workers | subject passed; unrelated settings-revision-attribution failure | +| PostgreSQL directory, 12 workers | subject passed; unrelated satellite-store ordering failure | + ## 3. Plugin runner complete-lane lifecycle hook - **File:** `packages/engine/src/__tests__/plugin-runner.test.ts` @@ -62,6 +76,13 @@ This register preserves first-sighting evidence under the narrow exception in [A Seven tests failed in `plugin-runner.test.ts`, but only this one identity survived capture: `--reporter=dot | tail -3` truncated the `FAIL` lines and retained only the summary. +**Quarantined 2026-08-16 (FN-9125):** Source inspection proved this unit file uses a local mocked TaskStore and has no PostgreSQL or harness import, so it does not belong to the database cluster. Three current full-engine lanes did not reproduce it, but the historical loaded failure lacks enough identities for a structural repair. The deletion-ratchet ledger and engine-default exclude were added together; assertions and timeouts are unchanged. + +| verification | result | +|---|---| +| full engine ×3, 6 threads | subject passed; 35–36 unrelated baseline-red files remained | +| targeted plugin-runner | covered by subsequent quarantine-ledger verification | + ## 4. Planning Mode direct task handoff - **File:** `packages/dashboard/app/components/__tests__/PlanningModeModal.planning-flow.test.tsx` @@ -93,9 +114,9 @@ The failure exercises the pre-existing mobile tab transition, while the task-cre **Suite re-admitted 2026-08-10 (FN-8936):** This first-sighting mobile observation did not receive a second failure. The shared file-level quarantine was removed only after the direct-handoff root cause was structurally fixed and the unexcluded loaded suite, including this mobile coverage, passed. -## Common shape and unverified suspicion +## Common shape and investigated result -Entries 1–3 are PostgreSQL-backed or PostgreSQL-suite-adjacent, pass in isolation, and appeared only under full-suite parallelism. This points at shared database state between those test files rather than any one test. It is **unverified and uninvestigated**, not a diagnosis; do not infer a root-cause fix from this record. Entry 6 instead records a merge-gate eviction after a loaded-lane setup-hook timeout; `FNXC:PgTestTemplateDb 2026-07-19-17:20` and `FNXC:PgTestWorkerCap 2026-07-18-18:00` are already-landed mitigations for that mode, not new diagnoses to re-open. The Planning Mode entries are separate frontend timing observations. +FN-9125 established that entry 3 is not PostgreSQL-suite-adjacent: `plugin-runner.test.ts` uses an in-memory mocked TaskStore and has no PostgreSQL/harness import. Its historical loaded-engine failure is quarantined under the normal deletion ratchet. Entries 1, 2, and 7 remain evidence-gathering-pending PostgreSQL observations: current full-output runs at six workers plus a twelve-worker PostgreSQL-directory run did not reproduce any subject identity, so FN-9125 cannot claim them superseded or resolved. The golden-template/advisory-lock lifecycle and schema-applier's inline baseline path are concrete architecture facts, not a demonstrated cause of these assertions. Core policy forbids inline PG quarantine: FN-9126 owns entry 1, FN-9128 exclusively owns entry 2, and FN-9127 owns entry 7 for CI/host-specific `pg_stat_activity`, lifecycle timing, and a complete loaded failure capture before any source or fan-out change. Entry 6 instead records a merge-gate eviction after a loaded-lane setup-hook timeout; `FNXC:PgTestTemplateDb 2026-07-19-17:20` and `FNXC:PgTestWorkerCap 2026-07-18-18:00` are already-landed mitigations for that mode, not new diagnoses to re-open. The Planning Mode entries are separate frontend timing observations. ## Policy and escalation @@ -137,6 +158,13 @@ FN-8928 evicted the file from the blocking gate under the AGENTS.md gate rule; d The timeout occurred after all test assertions and is unrelated to FN-8979's canonical mission-blocker contract. This file retains substantial coverage, so this first observation is recorded rather than quarantined. A second sighting requires the normal file-level quarantine decision. +**Evidence gathering pending 2026-08-16 (FN-9125):** The shared-harness teardown is serial (store, layer, admin client, `DROP DATABASE WITH (FORCE)`, temporary directory), so a loaded close/drop block remains a plausible historical mechanism. Three targeted dot-reporter runs and loaded core reproduction did not produce a timeout or a measurable slow phase. This is not superseded or resolved: the required complete loaded failure capture is absent. FN-9127 owns CI/host-specific phase instrumentation, full failure capture, and the escalation decision; core PostgreSQL quarantine is policy-forbidden. + +| verification | result | +|---|---| +| targeted dot reporter ×3 | 61 tests passed; afterAll passed | +| full core ×3, 6 workers | subject passed; unrelated settings-revision-attribution failure | + ## 8. Planning Mode duplicate-response generation reconciliation - **File:** `packages/dashboard/app/components/__tests__/PlanningModeModal.planning-flow.test.tsx` diff --git a/docs/testing.md b/docs/testing.md index 883aed03f0..d108b02134 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -461,6 +461,10 @@ Legitimate legacy exceptions must be recorded in `scripts/lib/test-timeout-appea **2026-06-19 CLI affected-lane rescue (FN-6734):** a broad `@runfusion/fusion` lane reported default 5s test-body timeouts and `fusion-test-workers-*`/fixture `ENOTEMPTY` cleanup noise while isolated files exposed closeable real-store handles and a runtime-dist mock that was sensitive to package-lane module graph ordering. The rescue closed each real `TaskStore`/`AgentStore` before removing its temp fixture, kept task-list truncation coverage under the default timeout by reducing filler size rather than assertions, and preloaded the built `@fusion/core` barrel with `vi.importActual` before `vi.doMock` so complete dist artifacts exercise the CLI surface while partial stale dist skips cleanly. Prove this class with targeted file runs, `pnpm --filter @runfusion/fusion test`, the timeout-appeasement guard, bounded temp-prefix cleanup output, and the normal workspace gate/build; leave the CLI quarantine array empty when no file is actually quarantined. + + +**2026-08-16 PostgreSQL first-sighting diagnosis (FN-9125):** First classify each file by its real dependency path, not nearby failure timing. Retain complete `tee` output for repeated package lanes and record fan-out, server capacity, and target identities. If a current run cannot link a PostgreSQL test assertion to golden-template, DDL, pool, or teardown evidence, do not add timeouts/retries or infer a harness change: obtain CI/host activity and phase timings. A core PostgreSQL file is escalated rather than quarantined because `quarantinedCoreTests` remains policy-pinned empty; a non-PG engine file follows normal ledger-plus-exclude lockstep. + diff --git a/packages/engine/vitest.config.ts b/packages/engine/vitest.config.ts index 91e3405e1d..cc038df584 100644 --- a/packages/engine/vitest.config.ts +++ b/packages/engine/vitest.config.ts @@ -336,6 +336,16 @@ export default defineConfig({ // / `test:all` invoked from the root `test:full` script. "src/**/*.slow.test.ts", /* + FNXC:PluginRunnerFlake 2026-08-16-17:30: + FN-9125 confirmed this in-memory mocked unit file has no PostgreSQL + dependency, but historical loaded-engine evidence captured seven + failures without sufficient identities for a structural fix. Quarantine + the whole file under the deletion ratchet instead of weakening its + lifecycle assertions; scripts/lib/test-quarantine.json is the paired + dated ledger record. + */ + "src/__tests__/plugin-runner.test.ts", + /* FNXC:FullSuiteBookkeeping 2026-08-09-03:49: All 11 engine-default entries from the 2026-08-05 full-suite quarantine wave (run 30982276306) were deleted under the deletion ratchet after operator directive. These tested pre-refactor APIs (getBuiltinWorkflow removed post-U10b), stale mock shapes, census/allowlist drift, and mock-hoist errors that no longer have a production path to exercise. */ diff --git a/scripts/lib/test-quarantine.json b/scripts/lib/test-quarantine.json index 06ccff151c..5b6e25133d 100644 --- a/scripts/lib/test-quarantine.json +++ b/scripts/lib/test-quarantine.json @@ -1,4 +1,10 @@ { "$comment": "Flaky-test quarantine ledger (deletion ratchet — see AGENTS.md 'Flaky tests: quarantine on sight' and docs/testing.md 'Quarantine ledger and the deletion ratchet'). A test observed failing without a corresponding real bug is quarantined ON SIGHT: add an entry here AND a matching one-line `exclude` entry in that package's vitest config, in the same commit. Every entry needs a non-empty `reason` (link the failing run) and a `quarantinedAt` date — the entry expires 14 days later, at which point the test file is DELETED unless someone rescues it with evidence it catches real regressions plus a root-cause fix (never appeasement). There is deliberately no loader module and no automation around this file: it is a dated record, the vitest config is the enforcement.", - "entries": [] + "entries": [ + { + "file": "packages/engine/src/__tests__/plugin-runner.test.ts", + "reason": "FN-9125: first-sighting register entry 3; failure evidence https://github.com/Runfusion/Fusion/pull/2799#issuecomment-5134921769 captures seven loaded-engine failures on the PR #2799 merged-with-main tree, but only `PluginRunner > task lifecycle hooks > should invoke onTaskCompleted when the complete lane is RENAMED` survived because the original dot output was truncated. FN-9125 confirmed the file is in-memory/mocked and not PostgreSQL-backed; no current structural failure was reproducible. Quarantined under the deletion ratchet rather than weakening lifecycle assertions.", + "quarantinedAt": "2026-08-16" + } + ] }