diff --git a/docs/test-velocity-baseline.md b/docs/test-velocity-baseline.md index fd2a81c0a1..b1b5375d14 100644 --- a/docs/test-velocity-baseline.md +++ b/docs/test-velocity-baseline.md @@ -5,7 +5,7 @@ ## Latest baseline - Cycle: **2026-W30** -- Captured at: **2026-07-22T21:58:46.970Z** +- Captured at: **2026-07-22T22:40:53.357Z** - Timing snapshot: `scripts/test-timings.json` captured at **2026-06-27T05:41:42.568Z** - Quarantine ledger: `scripts/lib/test-quarantine.json` @@ -13,9 +13,9 @@ | Metric | Current | Delta vs previous | |---|---:|---:| -| Merge gate wall-time (`pnpm test:gate`) | 29.3s | -15.9s | -| Boot smoke wall-time (`pnpm smoke:boot`) | 24.0s | +5.8s | -| Changed-only test wall-time (`pnpm test`) | 35.7s | -1.0s | +| Merge gate wall-time (`pnpm test:gate`) | 9.1s | -20.1s | +| Boot smoke wall-time (`pnpm smoke:boot`) | 18.5s | -5.6s | +| Changed-only test wall-time (`pnpm test`) | 14.9s | -20.7s | | Quarantine / flake count | 2 | 0 | | Deletion-due quarantines | 0 | n/a | @@ -72,16 +72,16 @@ | Row | Captured at | Gate | Boot smoke | `pnpm test` | Quarantine count | |---|---|---:|---:|---:|---:| -| Previous | 2026-07-22T20:52:05.508Z | 45.2s | 18.3s | 36.7s | 2 | -| Latest | 2026-07-22T21:58:46.970Z | 29.3s | 24.0s | 35.7s | 2 | -| Delta | — | -15.9s | +5.8s | -1.0s | 0 | +| Previous | 2026-07-22T21:58:46.970Z | 29.3s | 24.0s | 35.7s | 2 | +| Latest | 2026-07-22T22:40:53.357Z | 9.1s | 18.5s | 14.9s | 2 | +| Delta | — | -20.1s | -5.6s | -20.7s | 0 | _Future weekly rows append to `scripts/test-velocity-history.json`; compare the latest row against the previous row before posting to #leads._ ## Post to #leads ```text -FN-6612 weekly test velocity: gate 29.3s (-15.9s), boot smoke 24.0s (+5.8s), pnpm test 35.7s (-1.0s), quarantine ledger 2 (0). Slowest file: packages/dashboard/src/__tests__/insights-routes.test.ts at 26.5s. Deletion-due quarantines: 0. +FN-6612 weekly test velocity: gate 9.1s (-20.1s), boot smoke 18.5s (-5.6s), pnpm test 14.9s (-20.7s), quarantine ledger 2 (0). Slowest file: packages/dashboard/src/__tests__/insights-routes.test.ts at 26.5s. Deletion-due quarantines: 0. ``` ## How to refresh diff --git a/docs/testing.md b/docs/testing.md index cff7396a58..1d92a67006 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -6,10 +6,13 @@ This guide consolidates the detailed testing guidance moved from `AGENTS.md`. ## The merge gate -CI blocks PRs on exactly four checks (`.github/workflows/pr-checks.yml`): **Lint, Typecheck, Build, Gate**. The Gate job runs the boot smoke (`scripts/boot-smoke.mjs`: CLI `--help` + a real `fn serve` answering `GET /api/health`) and `pnpm test:gate` (the curated `engine-core` vitest project + the CI-shape test). Everything else — the 4-way shards, the engine slow tier, the dashboard inventory guard — runs NON-BLOCKING in `.github/workflows/full-suite.yml` on push to main. +CI blocks PRs on exactly four checks (`.github/workflows/pr-checks.yml`): **Lint, Typecheck, Build, Gate**. The Gate job runs the boot smoke (`scripts/boot-smoke.mjs`: CLI `--help` + a real `fn serve` answering `GET /api/health`) and `pnpm test:gate` (static process guards, curated `engine-core`, two PostgreSQL canaries, and the CI-shape test). Everything else — the 4-way shards, the engine slow tier, the dashboard inventory guard — runs NON-BLOCKING in `.github/workflows/full-suite.yml` on push to main. Gate membership is the explicit allow-list in `packages/engine/vitest.config.ts` (`engine-core` project). Admission requires evidence of value (the test catches real regressions); tests never graduate in by default. A flaky gate test is evicted by deleting its allow-list line — the eviction PR does not need the flaky test to pass. The whole `engine-core` project must stay under ~60s wall-clock. + +**PostgreSQL gate policy:** `packages/core`'s `test:pg-gate` intentionally runs only `task-lifecycle-e2e.pg.test.ts` and `handoff-to-review-atomicity.pg.test.ts`, preserving real-backend lifecycle and atomic-handoff canaries. It runs concurrently with `engine-core` after the static guards; the root script waits for **both** lanes and propagates either failure before running CI-shape. Every other former PG gate member remains enabled and discovered by the non-blocking command `pnpm --filter @fusion/core test` (default config: `src/**/*.test.ts`, no PG quarantine exclusions). `scripts/__tests__/engine-vitest-gate-policy.test.mjs` pins the two canaries and fails if any removed member is deleted, undiscoverable, or hidden by the default lane's script/config. + **Gate-safe `@fusion/core` barrel:** the `engine-core` project resolves `@fusion/core` to `packages/core/src/index.gate.ts` (a project-scoped `resolve.alias`, not the root map), not the full `packages/core/src/index.ts` barrel. `index.gate.ts` is a byte-for-byte copy of the full barrel minus the `export ... from` statements for modules added to the barrel after the last re-audit baseline — i.e. it re-exports everything the full barrel does except genuinely new, gate-irrelevant feature modules (diffed against the prior baseline commit's barrel, not hand-picked from what gate *test* files import — production modules under test pull in far more of the barrel transitively than their own imports suggest). `engine-default`/`engine-reliability`/`engine-slow` are unaffected and keep resolving the full barrel. `@fusion/engine` is untouched (no gate file imports it). When adding a new barrel module that no gate test needs, mirror the exclusion in `index.gate.ts` rather than letting gate wall-time grow — see the FNXC comment at the top of `index.gate.ts` and `packages/engine/vitest.config.ts`'s `engine-core` project for the audit procedure. diff --git a/package.json b/package.json index 6f35b6e505..6209697934 100644 --- a/package.json +++ b/package.json @@ -21,7 +21,7 @@ "check:changesets": "node scripts/check-changeset-format.mjs", "check:quarantine-ledger": "node scripts/check-quarantine-ledger.mjs", "check:mock-completeness": "node scripts/check-mock-completeness.mjs", - "test:gate": "node scripts/check-no-nohup.mjs && node scripts/check-no-kill-4040.mjs && node scripts/check-no-getdatabase.mjs && node scripts/check-no-node-only-core-imports-in-dashboard.mjs && node scripts/check-pi-versions-pinned.mjs && node scripts/check-no-test-timeout-appeasement.mjs && node scripts/check-changeset-format.mjs && node scripts/check-mock-completeness.mjs && pnpm --filter @fusion/engine test:core && pnpm --filter @fusion/core test:pg-gate && pnpm --filter @runfusion/fusion test:ci-shape", + "test:gate": "node scripts/check-no-nohup.mjs && node scripts/check-no-kill-4040.mjs && node scripts/check-no-getdatabase.mjs && node scripts/check-no-node-only-core-imports-in-dashboard.mjs && node scripts/check-pi-versions-pinned.mjs && node scripts/check-no-test-timeout-appeasement.mjs && node scripts/check-changeset-format.mjs && node scripts/check-mock-completeness.mjs && sh -c 'pnpm --filter @fusion/engine test:core & engine_pid=$!; pnpm --filter @fusion/core test:pg-gate & pg_pid=$!; status=0; wait $engine_pid || status=1; wait $pg_pid || status=1; exit $status' && pnpm --filter @runfusion/fusion test:ci-shape", "smoke:boot": "node scripts/boot-smoke.mjs", "local": "node scripts/start-local.mjs", "dev": "node scripts/dev-with-memory.mjs", diff --git a/packages/core/package.json b/packages/core/package.json index 6f077d68f4..ce42fec2be 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -43,7 +43,7 @@ "typecheck": "tsc --noEmit", "test": "vitest run --silent=passed-only --reporter=dot", "test:embedded-postgres": "vitest run src/__tests__/postgres/embedded-lifecycle.test.ts --silent=passed-only --reporter=dot", - "test:pg-gate": "vitest run --config vitest.pg.config.ts src/__tests__/postgres/handoff-to-review-atomicity.pg.test.ts src/__tests__/postgres/store-list.pg.test.ts src/__tests__/postgres/task-lifecycle-e2e.pg.test.ts src/__tests__/postgres/soft-delete-resurrection-FN-5233.pg.test.ts src/__tests__/postgres/agent-logs-and-monitor.pg.test.ts src/__tests__/postgres/todo-store.pg.test.ts src/__tests__/postgres/workflow-definitions.pg.test.ts src/__tests__/postgres/message-store.pg.test.ts src/__tests__/postgres/insight-store.pg.test.ts src/__tests__/postgres/insight-run-execution.pg.test.ts src/__tests__/postgres/research-store.pg.test.ts src/__tests__/postgres/mission-store.pg.test.ts src/__tests__/postgres/goal-store.pg.test.ts src/__tests__/postgres/artifacts-documents-evals.pg.test.ts src/__tests__/postgres/command-center-analytics.pg.test.ts src/__tests__/postgres/command-center-remaining-analytics.pg.test.ts src/__tests__/postgres/research-execution.pg.test.ts src/__tests__/postgres/async-store-events.pg.test.ts src/__tests__/postgres/signal-ingestion.pg.test.ts src/__tests__/postgres/mission-autopilot.pg.test.ts src/__tests__/postgres/workflow-create.pg.test.ts src/__tests__/postgres/monitor-trait-storm-guard.pg.test.ts src/__tests__/postgres/agent-wake-getagent.pg.test.ts --silent=passed-only --reporter=dot" + "test:pg-gate": "vitest run --config vitest.pg.config.ts src/__tests__/postgres/handoff-to-review-atomicity.pg.test.ts src/__tests__/postgres/task-lifecycle-e2e.pg.test.ts --silent=passed-only --reporter=dot" }, "devDependencies": { "@earendil-works/pi-coding-agent": "0.81.1", diff --git a/scripts/__tests__/engine-vitest-gate-policy.test.mjs b/scripts/__tests__/engine-vitest-gate-policy.test.mjs index 9a544da03c..e175a803d5 100644 --- a/scripts/__tests__/engine-vitest-gate-policy.test.mjs +++ b/scripts/__tests__/engine-vitest-gate-policy.test.mjs @@ -1,6 +1,6 @@ import test from "node:test"; import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; +import { readdirSync, readFileSync } from "node:fs"; import path from "node:path"; import { fileURLToPath } from "node:url"; @@ -58,10 +58,17 @@ test("engine-core remains an explicit allow-listed merge gate", () => { const includeEntries = [...engineCoreBlock.matchAll(/"src\/__tests__\/[^"\n]+\.test\.ts"/g)].map((match) => match[0]); assert.equal(new Set(includeEntries).size, includeEntries.length, "engine-core allow-list must not contain duplicates"); - assert.ok(includeEntries.length >= 18, "engine-core allow-list must not be gutted to avoid the runtime abort"); + /* + FNXC:MergeGatePerformance 2026-07-22-15:36: + FN-8497 exercises this policy after profiling the complete gate. The current + curated engine lane has 16 explicit files after the documented SQLite and + obsolete graph-runner retirements; guard its real floor instead of the stale + 18-file count, while still requiring the replacement graph executor seam. + */ + assert.ok(includeEntries.length >= 16, "engine-core allow-list must not be gutted to avoid the runtime abort"); assert.ok( - includeEntries.includes('"src/__tests__/workflow-graph-task-runner.test.ts"'), - "engine-core must keep workflow graph gate coverage", + includeEntries.includes('"src/__tests__/workflow-graph-executor-parity.test.ts"'), + "engine-core must keep workflow graph executor gate coverage", ); assert.ok( includeEntries.includes('"src/__tests__/heartbeat-monitor.test.ts"'), @@ -78,6 +85,70 @@ test("root and package gate scripts still propagate real Vitest failures", () => "vitest run --silent=passed-only --reporter=dot --project=engine-core", ); assert.match(root.scripts?.["test:gate"] ?? "", /pnpm --filter @fusion\/engine test:core/); + assert.match(root.scripts?.["test:gate"] ?? "", /wait \$engine_pid \|\| status=1/); + assert.match(root.scripts?.["test:gate"] ?? "", /wait \$pg_pid \|\| status=1/); assert.doesNotMatch(root.scripts?.["test:gate"] ?? "", /NODE_NO_WARNINGS/); assert.doesNotMatch(root.scripts?.["test"] ?? "", /NODE_NO_WARNINGS/); }); + +/* +FNXC:MergeGatePerformance 2026-07-22-15:35: +FN-8497 keeps only lifecycle and transactional-handoff PostgreSQL canaries in +`test:pg-gate`: 23 independent PG files each create/copy a real database, so +putting the whole integration inventory on every PR made the sequential merge +gate take 26–45 seconds. The other PG files must remain ordinary enabled core +tests; this structural guard prevents a future package-script/config change +from silently converting the speed fix into lost coverage. +*/ +test("pg gate canaries remain a subset of the enabled non-blocking PG suite", () => { + const core = readJson("packages/core/package.json"); + const coreConfig = read("packages/core/vitest.config.ts"); + const pgDirectory = path.join(repoRoot, "packages/core/src/__tests__/postgres"); + const discoveredPgFiles = new Set( + readdirSync(pgDirectory) + .filter((file) => file.endsWith(".pg.test.ts")) + .map((file) => `src/__tests__/postgres/${file}`), + ); + const gateMembers = core.scripts?.["test:pg-gate"]?.match(/src\/__tests__\/postgres\/[^ ]+\.pg\.test\.ts/g) ?? []; + const expectedCanaries = [ + "src/__tests__/postgres/handoff-to-review-atomicity.pg.test.ts", + "src/__tests__/postgres/task-lifecycle-e2e.pg.test.ts", + ]; + const formerGateMembers = [ + ...expectedCanaries, + "src/__tests__/postgres/store-list.pg.test.ts", + "src/__tests__/postgres/soft-delete-resurrection-FN-5233.pg.test.ts", + "src/__tests__/postgres/agent-logs-and-monitor.pg.test.ts", + "src/__tests__/postgres/todo-store.pg.test.ts", + "src/__tests__/postgres/workflow-definitions.pg.test.ts", + "src/__tests__/postgres/message-store.pg.test.ts", + "src/__tests__/postgres/insight-store.pg.test.ts", + "src/__tests__/postgres/insight-run-execution.pg.test.ts", + "src/__tests__/postgres/research-store.pg.test.ts", + "src/__tests__/postgres/mission-store.pg.test.ts", + "src/__tests__/postgres/goal-store.pg.test.ts", + "src/__tests__/postgres/artifacts-documents-evals.pg.test.ts", + "src/__tests__/postgres/command-center-analytics.pg.test.ts", + "src/__tests__/postgres/command-center-remaining-analytics.pg.test.ts", + "src/__tests__/postgres/research-execution.pg.test.ts", + "src/__tests__/postgres/async-store-events.pg.test.ts", + "src/__tests__/postgres/signal-ingestion.pg.test.ts", + "src/__tests__/postgres/mission-autopilot.pg.test.ts", + "src/__tests__/postgres/workflow-create.pg.test.ts", + "src/__tests__/postgres/monitor-trait-storm-guard.pg.test.ts", + "src/__tests__/postgres/agent-wake-getagent.pg.test.ts", + ]; + + assert.deepEqual(gateMembers, expectedCanaries, "the PG gate must stay a narrow, explicit canary list"); + assert.match(core.scripts?.test ?? "", /^vitest run\b/, "the non-blocking core lane must execute Vitest"); + assert.doesNotMatch(core.scripts?.test ?? "", /\s(?:--exclude|--include)\b/, "the non-blocking core lane must not narrow discovery"); + assert.match(coreConfig, /include:\s*\["src\/\*\*\/\*.test\.ts"\]/, "the default core config must discover PG tests"); + assert.match(coreConfig, /const quarantinedCoreTests: string\[\] = \[\]/, "no PG test may be hidden by quarantine exclusion"); + + for (const file of formerGateMembers) { + assert.ok(discoveredPgFiles.has(file), `former PG gate member must remain discovered: ${file}`); + } + const removedFromGate = formerGateMembers.filter((file) => !gateMembers.includes(file)); + assert.equal(removedFromGate.length, 21, "all non-canary former gate members must remain in the non-blocking lane"); + assert.ok(removedFromGate.every((file) => discoveredPgFiles.has(file)), "removed PG members must remain discoverable"); +}); diff --git a/scripts/test-velocity-history.json b/scripts/test-velocity-history.json index c07b438606..9fe516a22a 100644 --- a/scripts/test-velocity-history.json +++ b/scripts/test-velocity-history.json @@ -1930,6 +1930,121 @@ "Timing snapshot is 25 days old; verify slowest-file attribution before treating the table as the current culprit.", "9 of the listed slowest files no longer exist at the recorded paths: `packages/dashboard/src/__tests__/insights-routes.test.ts`, `packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts`, `packages/dashboard/src/__tests__/workflow-routes.test.ts`, `packages/core/src/__tests__/db.test.ts`, `packages/core/src/__tests__/mission-store.test.ts`, …. Refresh scripts/test-timings.json before choosing a slow-test rewrite." ] + }, + { + "capturedAt": "2026-07-22T22:40:53.357Z", + "gateMs": 9144, + "bootSmokeMs": 18454, + "testMs": 14943, + "quarantineCount": 2, + "slowestTop20": [ + { + "file": "packages/dashboard/src/__tests__/insights-routes.test.ts", + "ms": 26500, + "package": "@fusion/dashboard" + }, + { + "file": "packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts", + "ms": 24700, + "package": "@fusion/engine" + }, + { + "file": "packages/dashboard/src/__tests__/workflow-routes.test.ts", + "ms": 22000, + "package": "@fusion/dashboard" + }, + { + "file": "packages/core/src/__tests__/db.test.ts", + "ms": 21200, + "package": "@fusion/core" + }, + { + "file": "packages/dashboard/app/components/__tests__/GitManagerModal.test.tsx", + "ms": 16900, + "package": "@fusion/dashboard" + }, + { + "file": "packages/core/src/__tests__/mission-store.test.ts", + "ms": 16000, + "package": "@fusion/core" + }, + { + "file": "packages/cli/src/__tests__/extension.test.ts", + "ms": 15700, + "package": "@runfusion/fusion" + }, + { + "file": "packages/dashboard/app/components/__tests__/AgentPromptsManager.test.tsx", + "ms": 14800, + "package": "@fusion/dashboard" + }, + { + "file": "packages/dashboard/app/components/__tests__/App.test.tsx", + "ms": 14600, + "package": "@fusion/dashboard" + }, + { + "file": "packages/dashboard/app/components/__tests__/TaskDetailModal.inline-editing-and-integrations.test.tsx", + "ms": 14100, + "package": "@fusion/dashboard" + }, + { + "file": "packages/dashboard/app/components/__tests__/TaskDetailModal.rendering.test.tsx", + "ms": 13700, + "package": "@fusion/dashboard" + }, + { + "file": "packages/dashboard/src/__tests__/routes-auth.test.ts", + "ms": 13600, + "package": "@fusion/dashboard" + }, + { + "file": "packages/core/src/__tests__/agent-store.test.ts", + "ms": 13400, + "package": "@fusion/core" + }, + { + "file": "packages/engine/src/__tests__/workspace-merger-idempotency.test.ts", + "ms": 12700, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/self-healing-workspace.test.ts", + "ms": 11800, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/pr-response-run.test.ts", + "ms": 11600, + "package": "@fusion/engine" + }, + { + "file": "packages/dashboard/app/components/__tests__/ListView.test.tsx", + "ms": 11300, + "package": "@fusion/dashboard" + }, + { + "file": "plugins/fusion-plugin-compound-engineering/src/__tests__/sync.test.ts", + "ms": 11000, + "package": "@fusion-plugin-examples/compound-engineering" + }, + { + "file": "packages/dashboard/app/components/__tests__/AgentDetailView.settings.test.tsx", + "ms": 10700, + "package": "@fusion/dashboard" + }, + { + "file": "packages/dashboard/app/components/__tests__/SecretsView.test.tsx", + "ms": 10700, + "package": "@fusion/dashboard" + } + ], + "measurementFailures": [], + "timingSnapshotCapturedAt": "2026-06-27T05:41:42.568Z", + "timingNotes": [ + "Timing snapshot is 25 days old; verify slowest-file attribution before treating the table as the current culprit.", + "9 of the listed slowest files no longer exist at the recorded paths: `packages/dashboard/src/__tests__/insights-routes.test.ts`, `packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts`, `packages/dashboard/src/__tests__/workflow-routes.test.ts`, `packages/core/src/__tests__/db.test.ts`, `packages/core/src/__tests__/mission-store.test.ts`, …. Refresh scripts/test-timings.json before choosing a slow-test rewrite." + ] } ] }