root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FN-8928: evict flaky workflow IR PG gate canary

Browse Source 
Remove the flaky sync-workflow-IR PostgreSQL canary from the blocking merge gate while preserving non-blocking coverage.

- Remove the default workflow-IR PostgreSQL test from the gate canary script.
- Update gate-policy coverage expectations and flake-eviction documentation.
- Record the observed setup-hook timeout and retained regression coverage.

Files changed:
 .../suite-only-flakes-observed-register.md         | 25 +++++++++++++--
 docs/testing.md                                    |  6 ++--
 packages/core/package.json                         |  2 +-
 .../sync-workflow-ir-is-always-default.pg.test.ts  |  6 ++++
 .../__tests__/engine-vitest-gate-policy.test.mjs   | 37 +++++++++++-----------
 5 files changed, 51 insertions(+), 25 deletions(-)

Fusion-Task-Id: FN-8928

Fusion-Task-Lineage: b725ba1a-fb33-4d49-89b4-277a64246cdd

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
2026-08-10 02:32:24 -07:00
parent 5b9e6643d6
commit 6cf95433bf
5 changed files with 51 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
docs/solutions/test-failures/suite-only-flakes-observed-register.md
View File

@@ -9,6 +9,7 @@ applies_when:
- "A test fails under full-suite parallelism but passes when run alone"
- "A first flake sighting is in a file whose remaining coverage is substantial"
- "Capturing evidence before a file-level quarantine decision"
- "A merge-gate canary is evicted from the blocking gate after a flake sighting"
tags:
- flake
- postgres
@@ -18,7 +19,7 @@ tags:
# Observed suite-only flakes register
This register preserves first-sighting evidence under the narrow exception in [AGENTS.md](../../../AGENTS.md#standing-rule-flaky-tests-are-quarantined-on-sight-deletion-ratchet). It is not a quarantine: the normal default remains a ledger entry plus matching Vitest `exclude` in the same commit.
This register preserves first-sighting evidence under the narrow exception in [AGENTS.md](../../../AGENTS.md#standing-rule-flaky-tests-are-quarantined-on-sight-deletion-ratchet), and merge-gate eviction records. An eviction record documents a gate flake removed from the blocking canary list while coverage continues in the non-blocking lane. It is not a quarantine: the normal default remains a ledger entry plus matching Vitest `exclude` in the same commit.
## 1. Project identity returns no stored identity
@@ -94,12 +95,30 @@ The failure exercises the pre-existing mobile tab transition, while the task-cre
## Common shape and unverified suspicion
The first three entries are PostgreSQL-backed or PostgreSQL-suite-adjacent, pass in isolation, and appear only under full-suite parallelism. This points at shared database state between those test files rather than any one test. It is **unverified and uninvestigated**, not a diagnosis; do not infer a root-cause fix from this record. The Planning Mode entries are separate frontend timing observations.
Entries 1–3 are PostgreSQL-backed or PostgreSQL-suite-adjacent, pass in isolation, and appeared only under full-suite parallelism. This points at shared database state between those test files rather than any one test. It is **unverified and uninvestigated**, not a diagnosis; do not infer a root-cause fix from this record. Entry 6 instead records a merge-gate eviction after a loaded-lane setup-hook timeout; `FNXC:PgTestTemplateDb 2026-07-19-17:20` and `FNXC:PgTestWorkerCap 2026-07-18-18:00` are already-landed mitigations for that mode, not new diagnoses to re-open. The Planning Mode entries are separate frontend timing observations.
## Policy and escalation
Quarantine is file-level, while these files retain 6 / 75 / 80 passing tests. Under the first-sighting exception in AGENTS.md, recording preserves that valuable coverage while retaining the evidence needed for action. A **second sighting** of any registered test is an on-sight quarantine: add it to `scripts/lib/test-quarantine.json` and the matching Vitest `exclude` in one lockstep commit; this register entry is then evidence for the ledger `reason`.
Quarantine is file-level, while the first-sighting exception preserves coverage in files retaining 6 / 75 / 80 passing tests. Under that exception, recording preserves valuable coverage. A **second sighting** of a registered test is an on-sight quarantine: add it to `scripts/lib/test-quarantine.json` and the matching Vitest `exclude` in one lockstep commit; this register entry is then evidence for the ledger `reason`.
Merge-gate eviction records follow a separate branch: the gate can no longer be reddened by that file, while the non-blocking suite retains coverage. A further failure there is an ordinary on-sight quarantine. For PostgreSQL files, the gate-policy assertion forbidding a core-config quarantine exclude makes that an owner decision escalated as its own task rather than an inline edit.
Capture **full runner output** before recording or quarantining a failure—for example, tee it to a file. Never pipe a dot reporter through `tail`: the summary survives while the `FAIL` identity lines needed for a quarantine entry are exactly what gets truncated.
Source: [Runfusion/Fusion issue #2862](https://github.com/Runfusion/Fusion/issues/2862).
## 6. Sync workflow IR default canary setup hook
- **File:** `packages/core/src/__tests__/postgres/sync-workflow-ir-is-always-default.pg.test.ts`
- **Exact test:** `resolveTaskWorkflowIrSync ignores a task's real workflow (PostgreSQL)` suite `beforeAll` setup hook.
- **Observed tree/SHA:** FN-8912 evidence; local confirmation tree `51437558ac352dad3481e0dbe9622fa51af4c599`.
- **Observed frequency:** 1 observed merge-gate sighting in FN-8912; not reproduced locally. This is an **evicted merge-gate canary**, not a first-sighting register exception.
| run | result |
|---|---|
| FN-8912 loaded `pnpm test:gate` | **setup hook timed out** at the inherited 15s budget; direct scoped rerun passed |
| shape A: capped `test:pg-gate` ×5 | 3 files / 13 tests passed each run |
| shape B: isolated target ×3 | 1 file / 3 tests passed each run |
| shape C: uncapped default-config PostgreSQL directory ×5 | 153 files / 1263 passed plus 1 skipped each run |
FN-8928 evicted the file from the blocking gate under the AGENTS.md gate rule; default-core discovery preserves its regression coverage. Shape C was clean, so no quarantine escalation was required. A later non-blocking-core failure is an ordinary on-sight quarantine decision. `FNXC:PgTestTemplateDb 2026-07-19-17:20` (run-shared golden template) and `FNXC:PgTestWorkerCap 2026-07-18-18:00` (four-fork PG-gate cap) are already-landed mitigations for this same 15s setup-hook timeout mode.

6
docs/testing.md
View File

@@ -6,7 +6,7 @@ This guide consolidates the detailed testing guidance moved from `AGENTS.md`.
## The merge gate
CI blocks PRs on exactly four checks (`.github/workflows/pr-checks.yml`): **Lint, Typecheck, Build, Gate**. The Gate job runs the boot smoke (`scripts/boot-smoke.mjs`: CLI `--help` + a real `fn serve` answering `GET /api/health`) and `pnpm test:gate`: 11 static policy validators, 22 curated `engine-core` files, three PostgreSQL canaries, four core unit files, then the CI-shape test. Everything else — the 4-way shards, the engine slow tier, the dashboard inventory guard — runs NON-BLOCKING in `.github/workflows/full-suite.yml` on push to main.
CI blocks PRs on exactly four checks (`.github/workflows/pr-checks.yml`): **Lint, Typecheck, Build, Gate**. The Gate job runs the boot smoke (`scripts/boot-smoke.mjs`: CLI `--help` + a real `fn serve` answering `GET /api/health`) and `pnpm test:gate`: 11 static policy validators, 22 curated `engine-core` files, two PostgreSQL canaries, four core unit files, then the CI-shape test. Everything else — the 4-way shards, the engine slow tier, the dashboard inventory guard — runs NON-BLOCKING in `.github/workflows/full-suite.yml` on push to main.
Gate membership is the explicit allow-list in `packages/engine/vitest.config.ts` (`engine-core` project). Admission requires evidence of value (the test catches real regressions); tests never graduate in by default. A flaky gate test is evicted by deleting its allow-list line — the eviction PR does not need the flaky test to pass. The whole `engine-core` project must stay under ~60s wall-clock.
@@ -16,7 +16,7 @@ Gate membership is the explicit allow-list in `packages/engine/vitest.config.ts`
<!-- FNXC:MergeGatePerformance 2026-08-04-16:09: FN-8783 keeps engine-core's forks, worker budget, parallelism, bundle rebuild, and all 22 files while caching only Vitest transform artifacts between warm runs. This is never a test-result cache: every assertion and mock boundary remains evaluated for each invocation. -->
**FN-8783 warm result:** The paired W32 protocol recorded in task document `FN-8783/docs` measured the complete-gate median at **15.4s baseline** and **10.2s candidate** across five serialized AB/BA pairs on the same macOS arm64 host (Node 26.3.0, pnpm 10.33.0, identical lockfile). The final engine-core transform-cache profile used one priming run (6.3s), then five warm runs (**5.1, 5.2, 5.1, 5.0, 5.2s; median 5.1s**) versus the pre-cache 6.2s focused engine-core result. The residual full-gate critical path is the unchanged concurrent engine/PG/unit/CI-shape work; task evidence records commands, SHAs, preparation, raw timing order, and coverage counts.
**PostgreSQL and unit gate policy:** `packages/core`'s `test:pg-gate` intentionally runs `handoff-to-review-atomicity.pg.test.ts`, `task-lifecycle-e2e.pg.test.ts`, and `sync-workflow-ir-is-always-default.pg.test.ts`, preserving atomic-handoff, lifecycle, and default-IR real-backend canaries. `test:unit-gate` runs `task-merge.test.ts`, `legacy-adoption.test.ts`, `no-hardcoded-lifecycle-columns.test.ts`, and `sync-workflow-ir-callsite-allowlist.test.ts`. Every other former PG gate member remains enabled and discovered by the non-blocking command `pnpm --filter @fusion/core test` (default config: `src/**/*.test.ts`, no PG quarantine exclusions). `scripts/__tests__/engine-vitest-gate-policy.test.mjs` pins the exact three PG and four unit files, all waits, CI-shape ordering, and every engine/static member.
**PostgreSQL and unit gate policy:** `packages/core`'s `test:pg-gate` intentionally runs `handoff-to-review-atomicity.pg.test.ts` and `task-lifecycle-e2e.pg.test.ts`, preserving atomic-handoff and lifecycle real-backend canaries. `sync-workflow-ir-is-always-default.pg.test.ts` was evicted under the merge-gate flake rule; its coverage remains in the non-blocking core suite (see the [observed suite-only flakes register](solutions/test-failures/suite-only-flakes-observed-register.md#6-sync-workflow-ir-default-canary-setup-hook)). `test:unit-gate` runs `task-merge.test.ts`, `legacy-adoption.test.ts`, `no-hardcoded-lifecycle-columns.test.ts`, and `sync-workflow-ir-callsite-allowlist.test.ts`. Every other former PG gate member remains enabled and discovered by the non-blocking command `pnpm --filter @fusion/core test` (default config: `src/**/*.test.ts`, no PG quarantine exclusions). `scripts/__tests__/engine-vitest-gate-policy.test.mjs` pins the exact two PG and four unit files, all waits, CI-shape ordering, and every engine/static member.
<!-- FNXC:EngineTests 2026-07-08-03:00: FN-7667 decouples the engine-core gate's module graph from full-barrel growth so new feature modules don't silently inflate every gate fork's transform/import cost. -->
**Gate-safe `@fusion/core` barrel:** the `engine-core` project resolves `@fusion/core` to `packages/core/src/index.gate.ts` (a project-scoped `resolve.alias`, not the root map), not the full `packages/core/src/index.ts` barrel. `index.gate.ts` is a byte-for-byte copy of the full barrel minus the `export ... from` statements for modules added to the barrel after the last re-audit baseline — i.e. it re-exports everything the full barrel does except genuinely new, gate-irrelevant feature modules (diffed against the prior baseline commit's barrel, not hand-picked from what gate *test* files import — production modules under test pull in far more of the barrel transitively than their own imports suggest). `engine-default`/`engine-reliability`/`engine-slow` are unaffected and keep resolving the full barrel. `@fusion/engine` is untouched (no gate file imports it). When adding a new barrel module that no gate test needs, mirror the exclusion in `index.gate.ts` rather than letting gate wall-time grow — see the FNXC comment at the top of `index.gate.ts` and `packages/engine/vitest.config.ts`'s `engine-core` project for the audit procedure.
@@ -364,6 +364,8 @@ Flaky tests are quarantined ON SIGHT and deleted on a 2-week clock. This is writ
Quarantine is the default when a sighting is reproducible enough to justify evicting a file's coverage. The only exception is the narrow first-sighting record authority in AGENTS.md: a high-value file may be recorded in the [observed suite-only flakes register](solutions/test-failures/suite-only-flakes-observed-register.md) instead (`docs/solutions/test-failures/suite-only-flakes-observed-register.md`). A second sighting of a registered flake moves it to the ledger plus matching Vitest `exclude` in one lockstep commit.
**Merge-gate eviction is a separate branch.** A flake inside the blocking merge gate is evicted from its allow-list or canary script, not skipped and not timeout-widened; the eviction does not need the flaky test to pass. Coverage may remain in the non-blocking suite. Quarantining a PostgreSQL file additionally conflicts with the gate-policy assertion that `quarantinedCoreTests` stays empty, so that owner decision is escalated rather than performed inline. FN-8928's [default-IR canary record](solutions/test-failures/suite-only-flakes-observed-register.md#6-sync-workflow-ir-default-canary-setup-hook) is the worked example: FN-8912 observed a loaded-lane setup-hook timeout, so the file was evicted rather than quarantined.
**To quarantine a test** (a test that failed without a corresponding real bug in the change), in one commit:
1. Add an entry to `scripts/lib/test-quarantine.json`:

2
packages/core/package.json
View File

@@ -53,7 +53,7 @@
"typecheck": "tsc --noEmit",
"test": "vitest run --silent=passed-only --reporter=dot",
"test:embedded-postgres": "vitest run src/__tests__/postgres/embedded-lifecycle.test.ts --silent=passed-only --reporter=dot",
"test:pg-gate": "vitest run --config vitest.pg.config.ts src/__tests__/postgres/handoff-to-review-atomicity.pg.test.ts src/__tests__/postgres/task-lifecycle-e2e.pg.test.ts src/__tests__/postgres/sync-workflow-ir-is-always-default.pg.test.ts --silent=passed-only --reporter=dot",
"test:pg-gate": "vitest run --config vitest.pg.config.ts src/__tests__/postgres/handoff-to-review-atomicity.pg.test.ts src/__tests__/postgres/task-lifecycle-e2e.pg.test.ts --silent=passed-only --reporter=dot",
"test:unit-gate": "vitest run src/__tests__/task-merge.test.ts src/__tests__/legacy-adoption.test.ts src/__tests__/no-hardcoded-lifecycle-columns.test.ts src/__tests__/sync-workflow-ir-callsite-allowlist.test.ts --silent=passed-only --reporter=dot"
},
"devDependencies": {

6
packages/core/src/__tests__/postgres/sync-workflow-ir-is-always-default.pg.test.ts
View File

@@ -24,6 +24,12 @@ assertion to read instead of a plausible-looking guard.
If a future change makes the sync path authoritative, this test SHOULD fail — that is the signal to
revisit those ten sites, not to delete the assertion.
FNXC:TestQuarantine 2026-08-10-09:16:
FN-8928 removed this file from the PostgreSQL merge gate after FN-8912 observed its setup hook
exceed the inherited 15s budget in the loaded lane. The non-blocking core suite retains this
regression proof. `PgTestTemplateDb 2026-07-19-17:20` and `PgTestWorkerCap 2026-07-18-18:00`
already mitigate this mode; this disposition does not alter the harness, assertions, or budgets.
*/
import { it, expect, beforeAll, beforeEach, afterEach, afterAll } from "vitest";
import {

37
scripts/__tests__/engine-vitest-gate-policy.test.mjs
View File

@@ -74,6 +74,12 @@ test("engine-core remains an explicit allow-listed merge gate", () => {
FN-8783 measured the W32 gate after six policy files joined the former
16-file lane. Exact membership is the coverage contract: an efficiency change
may reduce scheduling overhead, never silently drop an assertion group.
FNXC:TestInfrastructure 2026-08-10-09:16:
`project-engine.test.ts` is now intentionally excluded by its FN-8811
quarantine in the engine config, while `check-prerebase-inert.mjs` joined the
blocking static composition. Keep this ledger aligned with those authoritative
declarations so unrelated policy drift cannot mask PG-gate membership checks.
*/
const expectedMembers = [
'"src/__tests__/legacy-column-literal-census.test.ts"',
@@ -83,7 +89,6 @@ test("engine-core remains an explicit allow-listed merge gate", () => {
'"src/__tests__/merger-diff-scope.test.ts"',
'"src/__tests__/merger-landed-files-capture.test.ts"',
'"src/__tests__/branch-attribution.test.ts"',
'"src/__tests__/project-engine.test.ts"',
'"src/__tests__/merge-single-flight-invariant.test.ts"',
'"src/__tests__/workflow-step-verdict-parsing.test.ts"',
'"src/__tests__/u9-merge-region-node-config-authority.test.ts"',
@@ -99,7 +104,7 @@ test("engine-core remains an explicit allow-listed merge gate", () => {
'"src/__tests__/workflow-node-handlers.test.ts"',
'"src/__tests__/workflow-policy-ownership-map.test.ts"',
];
assert.deepEqual(includeEntries, expectedMembers, "engine-core must retain its complete ordered 22-file coverage map");
assert.deepEqual(includeEntries, expectedMembers, "engine-core must retain its complete ordered 21-file coverage map");
});
test("root and package gate scripts still propagate real Vitest failures", () => {
@@ -121,6 +126,7 @@ test("root and package gate scripts still propagate real Vitest failures", () =>
staticCheck("no-cwd-relative-dashboard-test-reads"),
staticCheck(["no-", "kill-", "40" + "40"].join("")),
staticCheck("no-getdatabase"),
staticCheck("prerebase-inert"),
staticCheck("capacity-pool-id"),
staticCheck("no-node-only-core-imports-in-dashboard"),
staticCheck("pi-versions-pinned"),
@@ -166,30 +172,23 @@ test("pg gate canaries remain a subset of the enabled non-blocking PG suite", ()
const gateMembers = core.scripts?.["test:pg-gate"]?.match(/src\/__tests__\/postgres\/[^ ]+\.pg\.test\.ts/g) ?? [];
/*
FNXC:TestInfrastructure 2026-07-31-20:30:
The third canary is RECORDED here, not approved here — and the distinction is the point.
FN-8928 evicted `sync-workflow-ir-is-always-default.pg.test.ts` after FN-8912 observed its
setup hook exceed the inherited 15s budget in the loaded PG gate lane. The AGENTS.md gate rule
requires eviction rather than a skip; its default-core discovery remains enabled, preserving the
regression proof outside the blocking canary list. Local evidence was clean in five loaded-gate,
three isolated, and five uncapped default-config PostgreSQL runs, which does not undo the observed
gate flake or the required disposition.
#2759 (`ae4ff9c111`) added `sync-workflow-ir-is-always-default.pg.test.ts` and its gate entry in one
commit without updating this list, so the assertion has been red on `main` since. A red policy test
protects nothing: while it fails, the NEXT gate admission is invisible too, which is the opposite of
what a narrow-canary ledger is for. Restoring it re-arms that protection.
The admission carries the evidence of value AGENTS.md requires, which is why recording it is not a
rubber stamp. It pins that `resolveTaskWorkflowIrSync` returns the DEFAULT IR for every task in
production — so a guard written as `resolveLifecycleColumns(store.resolveTaskWorkflowIrSync(id))?.hold`
reads as converted, counts as census progress, and is silently wrong for every custom workflow
because the non-optional return type hides the substitution. Ten call sites depend on that fact
today. A whole class of inert conversions is cheaper to catch at the gate than in review.
If the gate's owner disagrees with a third canary, the fix is to remove it from
`packages/core`'s `test:pg-gate` script and shorten this list again — not to leave the ledger red.
A red policy test protects nothing: while it fails, the NEXT gate admission or eviction is invisible,
which is the opposite of what a narrow-canary ledger is for. The ledger must never be left red.
*/
const expectedCanaries = [
"src/__tests__/postgres/handoff-to-review-atomicity.pg.test.ts",
"src/__tests__/postgres/task-lifecycle-e2e.pg.test.ts",
"src/__tests__/postgres/sync-workflow-ir-is-always-default.pg.test.ts",
];
const formerGateMembers = [
...expectedCanaries,
"src/__tests__/postgres/sync-workflow-ir-is-always-default.pg.test.ts",
"src/__tests__/postgres/store-list.pg.test.ts",
"src/__tests__/postgres/soft-delete-resurrection-FN-5233.pg.test.ts",
"src/__tests__/postgres/agent-logs-and-monitor.pg.test.ts",
@@ -223,6 +222,6 @@ test("pg gate canaries remain a subset of the enabled non-blocking PG suite", ()
assert.ok(discoveredPgFiles.has(file), `former PG gate member must remain discovered: ${file}`);
}
const removedFromGate = formerGateMembers.filter((file) => !gateMembers.includes(file));
assert.equal(removedFromGate.length, 21, "all non-canary former gate members must remain in the non-blocking lane");
assert.equal(removedFromGate.length, 22, "all non-canary former gate members must remain in the non-blocking lane");
assert.ok(removedFromGate.every((file) => discoveredPgFiles.has(file)), "removed PG members must remain discoverable");
});