diff --git a/AGENTS.md b/AGENTS.md index 1997bf2446..4f9701ba5e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -86,6 +86,7 @@ The merge gate is thin and trusted: CI blocks PRs on exactly Lint, Typecheck, Bu pnpm test # gate suite + changed-only affected tests (bounded; never full-suite) pnpm test:gate # the merge gate: curated engine-core suite + CI-shape test pnpm smoke:boot # boot smoke: CLI --help + real serve /api/health +pnpm test:velocity # weekly report-only test velocity baseline; use -- --measure --write-report to refresh pnpm test:full # full workspace suite — explicit opt-in only pnpm lint pnpm build @@ -195,7 +196,8 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme ## Reference docs (deeper detail) - `./docs/architecture.md` — lifecycle invariants, self-healing rules, reliability interaction backstops, run-audit internals. -- `./docs/testing.md` — full testing lanes, worker fanout guidance, test taxonomy, and file organization. +- `./docs/testing.md` — full testing lanes, worker fanout guidance, test taxonomy, weekly velocity baseline, and file organization. +- `./docs/test-velocity-baseline.md` — weekly #leads-ready test feedback-loop velocity report generated by `scripts/test-velocity-baseline.mjs`. - `./docs/dashboard-guide.md` — dashboard behavior and **Styling Guide** details. User-facing docs for Merge Advance Notice and Smart Pull live here. - `./docs/PLUGIN_AUTHORING.md` — plugin authoring guide, lifecycle hooks, routes, tools, and dashboard-extension surfaces. - `./docs/agents.md` — pi extension scope, coordination tools, checkout leasing, runtime config. diff --git a/docs/test-velocity-baseline.md b/docs/test-velocity-baseline.md new file mode 100644 index 0000000000..687d2a6dcd --- /dev/null +++ b/docs/test-velocity-baseline.md @@ -0,0 +1,92 @@ +# Test velocity baseline + +> Weekly FN-6612 signal-per-second baseline. Measure and report feedback-loop velocity; do **not** add slow tests or wire this report into blocking PR checks. The merge gate remains the existing thin Lint, Typecheck, Build, and Gate path. + +## Latest baseline + +- Cycle: **2026-W25** +- Captured at: **2026-06-18T03:04:28.794Z** +- Timing snapshot: `scripts/test-timings.json` captured at **2026-06-03T23:45:49.672Z** +- Quarantine ledger: `scripts/lib/test-quarantine.json` + +## Metrics + +| Metric | Current | Delta vs previous | +|---|---:|---:| +| Merge gate wall-time (`pnpm test:gate`) | 6.2s | -2.3s | +| Boot smoke wall-time (`pnpm smoke:boot`) | 18.2s | n/a | +| Changed-only test wall-time (`pnpm test`) | 7.7s | -30.7s | +| Quarantine / flake count | 2 | 0 | +| Deletion-due quarantines | 0 | n/a | + +## Measurement failures + +- None recorded. + +## Slowest 20 test files + +| Rank | File | Package | Duration | +|---:|---|---|---:| +| 1 | `packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts` | @fusion/engine | 13.9s | +| 2 | `packages/core/src/__tests__/agent-store.test.ts` | @fusion/core | 11.6s | +| 3 | `packages/dashboard/src/__tests__/routes-agents.test.ts` | @fusion/dashboard | 11.2s | +| 4 | `packages/core/src/__tests__/mission-store.test.ts` | @fusion/core | 10.7s | +| 5 | `packages/core/src/__tests__/db.test.ts` | @fusion/core | 10.1s | +| 6 | `packages/dashboard/src/__tests__/routes-git.test.ts` | @fusion/dashboard | 9.4s | +| 7 | `packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts` | @fusion/engine | 9.0s | +| 8 | `packages/engine/src/__tests__/merger-ai.test.ts` | @fusion/engine | 8.7s | +| 9 | `packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts` | @fusion/engine | 8.4s | +| 10 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts` | @fusion/engine | 8.4s | +| 11 | `packages/core/src/__tests__/task-documents.test.ts` | @fusion/core | 8.3s | +| 12 | `packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts` | @fusion/engine | 7.8s | +| 13 | `packages/cli/src/__tests__/extension.test.ts` | @runfusion/fusion | 7.0s | +| 14 | `packages/core/src/__tests__/run-audit.test.ts` | @fusion/core | 6.9s | +| 15 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts` | @fusion/engine | 6.1s | +| 16 | `packages/dashboard/src/__tests__/routes-planning.test.ts` | @fusion/dashboard | 5.6s | +| 17 | `packages/core/src/__tests__/store-merge-queue.test.ts` | @fusion/core | 5.2s | +| 18 | `packages/dashboard/app/components/__tests__/FileEditor.test.tsx` | @fusion/dashboard | 5.1s | +| 19 | `packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts` | @fusion/engine | 4.9s | +| 20 | `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` | @fusion/engine | 4.9s | + +## Quarantine age buckets + +| Age bucket | Count | +|---|---:| +| 0-6 days | 2 | +| 7-13 days | 0 | +| deletion due (>=14 days) | 0 | +| unknown/future | 0 | + +### Deletion-due entries + +| File | Quarantined at | Age (days) | +|---|---:|---:| +| — | — | — | + +## Before / after trend + +| Row | Captured at | Gate | Boot smoke | `pnpm test` | Quarantine count | +|---|---|---:|---:|---:|---:| +| Previous | 2026-06-18T02:53:34.158Z | 8.5s | unavailable | 38.4s | 2 | +| Latest | 2026-06-18T03:04:28.794Z | 6.2s | 18.2s | 7.7s | 2 | +| Delta | — | -2.3s | n/a | -30.7s | 0 | + +_Future weekly rows append to `scripts/test-velocity-history.json`; compare the latest row against the previous row before posting to #leads._ + +## Post to #leads + +```text +FN-6612 weekly test velocity: gate 6.2s (-2.3s), boot smoke 18.2s (n/a), pnpm test 7.7s (-30.7s), quarantine ledger 2 (0). Slowest file: packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts at 13.9s. Deletion-due quarantines: 0. +``` + +## How to refresh + +```bash +pnpm test:velocity -- --measure --write-report +``` + +Report-only regeneration is cheap and does not run any suite: + +```bash +pnpm test:velocity +``` diff --git a/docs/testing.md b/docs/testing.md index 76ddcb840e..276d5ca1b1 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -225,6 +225,24 @@ shard artifacts into `.timings/` first (the default lookup directory), or pass scheduled job can gate on freshness via `node scripts/ci-test-shard.mjs --check-timings-staleness`, which exits non-zero when the snapshot is missing or older than the 30-day budget. +## Weekly test velocity baseline + +FN-6612 tracks feedback-loop velocity as signal-per-second, not as a new blocking gate. Refresh the weekly baseline from a clean worktree with: + +```bash +pnpm test:velocity -- --measure --write-report +``` + +The script runs `pnpm test:gate`, `pnpm smoke:boot`, and `pnpm test` with bounded async process supervision, then appends the measured row to `scripts/test-velocity-history.json` and rewrites the postable artifact at `docs/test-velocity-baseline.md`. It reads the slowest 20 files from the committed `scripts/test-timings.json` snapshot and the flake/quarantine count plus 14-day deletion-clock buckets directly from `scripts/lib/test-quarantine.json`; do not run the full suite just to populate the slowest-file table. + +Use cheap report-only regeneration when measurements already exist: + +```bash +pnpm test:velocity +``` + +Each week, copy the `Post to #leads` block from `docs/test-velocity-baseline.md`. If a measured command fails because the local environment is not ready, keep the failure recorded in the report instead of fabricating a time, then fix or rerun separately as appropriate. Do not wire `pnpm test:velocity`, `test:full`, or any slow-suite expansion into PR checks; the merge gate stays the thin Lint, Typecheck, Build, and Gate path. + ## Targeted commands ```bash diff --git a/package.json b/package.json index 88c3bae98c..a6df96a0c0 100644 --- a/package.json +++ b/package.json @@ -34,6 +34,7 @@ "test:scripts": "node --test scripts/__tests__/*.test.mjs", "fn:cache-stats": "node scripts/cache-stats.mjs", "test:full": "node scripts/test-changed.mjs --full --no-cache && pnpm --filter @fusion/engine test:slow", + "test:velocity": "node scripts/test-velocity-baseline.mjs", "test:feedback-baseline": "node scripts/test-feedback-baseline.mjs", "test:ci:shard": "node scripts/ci-test-shard.mjs", "test:serial": "FUSION_TEST_CONCURRENCY=1 FUSION_TEST_WORKSPACE_CONCURRENCY=1 pnpm test:full", diff --git a/scripts/__tests__/test-velocity-baseline.test.mjs b/scripts/__tests__/test-velocity-baseline.test.mjs new file mode 100644 index 0000000000..f7590b1672 --- /dev/null +++ b/scripts/__tests__/test-velocity-baseline.test.mjs @@ -0,0 +1,121 @@ +import { describe, it } from "node:test"; +import assert from "node:assert/strict"; + +import { + readQuarantineCount, + renderReport, + topSlowestFiles, +} from "../test-velocity-baseline.mjs"; + +function makeTimings(count = 25) { + const files = {}; + for (let index = 0; index < count; index += 1) { + files[`packages/a/src/__tests__/case-${String(index).padStart(2, "0")}.test.ts`] = 100 + index; + } + files["packages/a/src/__tests__/tie-z.test.ts"] = 500; + files["packages/a/src/__tests__/tie-a.test.ts"] = 500; + return { + packages: { + "@pkg/b": { + files: { + "packages/b/src/__tests__/winner.test.ts": 1000, + }, + }, + "@pkg/a": { files }, + }, + }; +} + +describe("topSlowestFiles", () => { + it("returns exactly 20 rows in descending duration order with package attribution and stable ties", () => { + const rows = topSlowestFiles(makeTimings(), 20); + + assert.equal(rows.length, 20); + assert.deepEqual(rows[0], { + file: "packages/b/src/__tests__/winner.test.ts", + ms: 1000, + package: "@pkg/b", + }); + assert.deepEqual(rows.slice(1, 3).map((row) => row.file), [ + "packages/a/src/__tests__/tie-a.test.ts", + "packages/a/src/__tests__/tie-z.test.ts", + ]); + assert.ok(rows.every((row, index) => index === 0 || rows[index - 1].ms >= row.ms)); + assert.equal(rows[1].package, "@pkg/a"); + }); +}); + +describe("readQuarantineCount", () => { + it("counts entries by age bucket and flags deletion-due quarantines after 14 days", () => { + const result = readQuarantineCount( + { + entries: [ + { file: "fresh.test.ts", quarantinedAt: "2026-06-15" }, + { file: "warning.test.ts", quarantinedAt: "2026-06-08" }, + { file: "due.test.ts", quarantinedAt: "2026-06-01" }, + { file: "unknown.test.ts", quarantinedAt: "not-a-date" }, + ], + }, + { now: new Date("2026-06-17T12:00:00.000Z") }, + ); + + assert.equal(result.total, 4); + assert.deepEqual(result.byAgeBucket, { + "0-6d": 1, + "7-13d": 1, + deletionDue: 1, + unknown: 1, + }); + assert.deepEqual(result.deletionDueEntries, [ + { file: "due.test.ts", quarantinedAt: "2026-06-01", ageDays: 16 }, + ]); + }); +}); + +describe("renderReport", () => { + it("includes metrics, slowest rows, quarantine count, and previous-run deltas", () => { + const report = renderReport({ + gateMs: 12_000, + bootSmokeMs: 2_000, + testMs: 45_000, + capturedAt: "2026-06-17T12:00:00.000Z", + previous: { + capturedAt: "2026-06-10T12:00:00.000Z", + gateMs: 10_000, + bootSmokeMs: 3_000, + testMs: 50_000, + quarantineCount: 3, + }, + slowest: [ + { file: "packages/a/src/__tests__/slow.test.ts", package: "@pkg/a", ms: 3210 }, + ], + quarantine: { + total: 2, + byAgeBucket: { "0-6d": 1, "7-13d": 1, deletionDue: 0, unknown: 0 }, + deletionDueEntries: [], + deletionDueCount: 0, + }, + }); + + assert.match(report, /\| Merge gate wall-time \(`pnpm test:gate`\) \| 12\.0s \| \+2\.0s \|/); + assert.match(report, /\| Boot smoke wall-time \(`pnpm smoke:boot`\) \| 2\.0s \| -1\.0s \|/); + assert.match(report, /\| Changed-only test wall-time \(`pnpm test`\) \| 45\.0s \| -5\.0s \|/); + assert.match(report, /\| Quarantine \/ flake count \| 2 \| -1 \|/); + assert.match(report, /`packages\/a\/src\/__tests__\/slow\.test\.ts` \| @pkg\/a \| 3\.2s/); + assert.match(report, /FN-6612 weekly test velocity: gate 12\.0s \(\+2\.0s\)/); + }); + + it("renders seed-baseline trend placeholders when there is no previous entry", () => { + const report = renderReport({ + gateMs: 1_000, + bootSmokeMs: null, + testMs: 2_000, + capturedAt: "2026-06-17T12:00:00.000Z", + slowest: [], + quarantine: { total: 0, byAgeBucket: {}, deletionDueEntries: [], deletionDueCount: 0 }, + }); + + assert.match(report, /\| Previous \| _\(seed baseline\)_ \| — \| — \| — \| — \|/); + assert.match(report, /\| Delta \| — \| n\/a \| n\/a \| n\/a \| n\/a \|/); + }); +}); diff --git a/scripts/test-velocity-baseline.mjs b/scripts/test-velocity-baseline.mjs new file mode 100755 index 0000000000..0319a30ad9 --- /dev/null +++ b/scripts/test-velocity-baseline.mjs @@ -0,0 +1,339 @@ +#!/usr/bin/env node + +import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { spawn } from "node:child_process"; +import { performance } from "node:perf_hooks"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const currentFilePath = fileURLToPath(import.meta.url); +const repoRoot = path.resolve(path.dirname(currentFilePath), ".."); + +export const DEFAULT_TIMINGS_PATH = "scripts/test-timings.json"; +export const DEFAULT_QUARANTINE_PATH = "scripts/lib/test-quarantine.json"; +export const DEFAULT_HISTORY_PATH = "scripts/test-velocity-history.json"; +export const DEFAULT_REPORT_PATH = "docs/test-velocity-baseline.md"; +export const DEFAULT_MEASURE_TIMEOUT_MS = 10 * 60 * 1000; +export const DELETION_CLOCK_DAYS = 14; + +const MEASURE_COMMANDS = [ + { key: "gateMs", label: "Merge gate (`pnpm test:gate`)", command: "pnpm", args: ["test:gate"] }, + { key: "bootSmokeMs", label: "Boot smoke (`pnpm smoke:boot`)", command: "pnpm", args: ["smoke:boot"] }, + { key: "testMs", label: "Changed-only tests (`pnpm test`)", command: "pnpm", args: ["test"] }, +]; + +function readJson(relativePath, fallback = null, rootDir = repoRoot) { + const absolutePath = path.join(rootDir, relativePath); + if (!existsSync(absolutePath)) return fallback; + return JSON.parse(readFileSync(absolutePath, "utf8")); +} + +function writeJson(relativePath, value, rootDir = repoRoot) { + const absolutePath = path.join(rootDir, relativePath); + mkdirSync(path.dirname(absolutePath), { recursive: true }); + writeFileSync(absolutePath, `${JSON.stringify(value, null, 2)}\n`, "utf8"); +} + +function writeText(relativePath, value, rootDir = repoRoot) { + const absolutePath = path.join(rootDir, relativePath); + mkdirSync(path.dirname(absolutePath), { recursive: true }); + writeFileSync(absolutePath, value, "utf8"); +} + +function normalizeMs(value) { + if (value == null || value === "") return null; + const parsed = Number(value); + if (!Number.isFinite(parsed) || parsed < 0) { + throw new Error(`Expected a non-negative millisecond value, got ${value}`); + } + return Math.round(parsed); +} + +function toDate(value) { + if (!value) return null; + const parsed = new Date(value); + return Number.isNaN(parsed.getTime()) ? null : parsed; +} + +function ageDays(quarantinedAt, now) { + const quarantinedAtDate = toDate(quarantinedAt); + if (!quarantinedAtDate) return null; + return Math.floor((now.getTime() - quarantinedAtDate.getTime()) / 86_400_000); +} + +function isoWeek(date) { + const working = new Date(Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate())); + const day = working.getUTCDay() || 7; + working.setUTCDate(working.getUTCDate() + 4 - day); + const yearStart = new Date(Date.UTC(working.getUTCFullYear(), 0, 1)); + const week = Math.ceil(((working - yearStart) / 86_400_000 + 1) / 7); + return `${working.getUTCFullYear()}-W${String(week).padStart(2, "0")}`; +} + +export function formatDuration(ms) { + if (ms == null) return "unavailable"; + const rounded = Math.round(ms); + const sign = rounded < 0 ? "-" : ""; + const absoluteMs = Math.abs(rounded); + if (absoluteMs < 1000) return `${sign}${absoluteMs}ms`; + const seconds = absoluteMs / 1000; + if (seconds < 60) return `${sign}${seconds.toFixed(1)}s`; + const minutes = Math.floor(seconds / 60); + const remainingSeconds = Math.round(seconds - minutes * 60); + return `${sign}${minutes}m ${String(remainingSeconds).padStart(2, "0")}s`; +} + +export function readQuarantineCount(json, { now = new Date() } = {}) { + const entries = Array.isArray(json?.entries) ? json.entries : []; + const buckets = { + "0-6d": 0, + "7-13d": 0, + deletionDue: 0, + unknown: 0, + }; + const deletionDueEntries = []; + + for (const entry of entries) { + const age = ageDays(entry?.quarantinedAt, now); + const normalized = { + file: entry?.file ?? "unknown", + quarantinedAt: entry?.quarantinedAt ?? null, + ageDays: age, + }; + if (age == null || age < 0) { + buckets.unknown += 1; + } else if (age >= DELETION_CLOCK_DAYS) { + buckets.deletionDue += 1; + deletionDueEntries.push(normalized); + } else if (age >= 7) { + buckets["7-13d"] += 1; + } else { + buckets["0-6d"] += 1; + } + } + + return { + total: entries.length, + byAgeBucket: buckets, + deletionDueEntries, + deletionDueCount: deletionDueEntries.length, + }; +} + +export function topSlowestFiles(timingsJson, n = 20) { + const rows = []; + for (const [packageName, packageTiming] of Object.entries(timingsJson?.packages ?? {})) { + for (const [file, ms] of Object.entries(packageTiming?.files ?? {})) { + const parsed = Number(ms); + rows.push({ file, ms: Number.isFinite(parsed) ? parsed : 0, package: packageName }); + } + } + + rows.sort((a, b) => b.ms - a.ms || a.package.localeCompare(b.package) || a.file.localeCompare(b.file)); + return rows.slice(0, n); +} + +function delta(latest, previous, field) { + if (!previous || latest?.[field] == null || previous?.[field] == null) return "n/a"; + const diff = latest[field] - previous[field]; + const sign = diff > 0 ? "+" : ""; + return `${sign}${formatDuration(diff)}`; +} + +function renderMetricRow(name, latest, previous, field) { + return `| ${name} | ${formatDuration(latest?.[field])} | ${delta(latest, previous, field)} |`; +} + +function trendCell(current, prior) { + if (current == null || prior == null) return "n/a"; + const diff = current - prior; + return `${diff > 0 ? "+" : ""}${diff}`; +} + +export function renderReport({ gateMs, bootSmokeMs, testMs, slowest = [], quarantine, capturedAt, previous = null, measurementFailures = [], timingSnapshotCapturedAt = null } = {}) { + const latest = { + gateMs: normalizeMs(gateMs), + bootSmokeMs: normalizeMs(bootSmokeMs), + testMs: normalizeMs(testMs), + quarantineCount: quarantine?.total ?? quarantine?.quarantineCount ?? 0, + capturedAt: capturedAt ?? new Date().toISOString(), + }; + const cycle = isoWeek(new Date(latest.capturedAt)); + const slowRows = slowest + .map((row, index) => `| ${index + 1} | \`${row.file}\` | ${row.package} | ${formatDuration(row.ms)} |`) + .join("\n"); + const dueRows = (quarantine?.deletionDueEntries ?? []) + .map((entry) => `| \`${entry.file}\` | ${entry.quarantinedAt ?? "unknown"} | ${entry.ageDays ?? "unknown"} |`) + .join("\n"); + const failures = measurementFailures.length > 0 + ? measurementFailures.map((failure) => `- ${failure.label}: ${failure.status}`).join("\n") + : "- None recorded."; + const previousRows = previous + ? `| Previous | ${previous.capturedAt ?? "unknown"} | ${formatDuration(previous.gateMs)} | ${formatDuration(previous.bootSmokeMs)} | ${formatDuration(previous.testMs)} | ${previous.quarantineCount ?? "n/a"} |\n| Latest | ${latest.capturedAt} | ${formatDuration(latest.gateMs)} | ${formatDuration(latest.bootSmokeMs)} | ${formatDuration(latest.testMs)} | ${latest.quarantineCount} |\n| Delta | — | ${delta(latest, previous, "gateMs")} | ${delta(latest, previous, "bootSmokeMs")} | ${delta(latest, previous, "testMs")} | ${trendCell(latest.quarantineCount, previous.quarantineCount)} |` + : `| Previous | _(seed baseline)_ | — | — | — | — |\n| Latest | ${latest.capturedAt} | ${formatDuration(latest.gateMs)} | ${formatDuration(latest.bootSmokeMs)} | ${formatDuration(latest.testMs)} | ${latest.quarantineCount} |\n| Delta | — | n/a | n/a | n/a | n/a |`; + + return `# Test velocity baseline\n\n> Weekly FN-6612 signal-per-second baseline. Measure and report feedback-loop velocity; do **not** add slow tests or wire this report into blocking PR checks. The merge gate remains the existing thin Lint, Typecheck, Build, and Gate path.\n\n## Latest baseline\n\n- Cycle: **${cycle}**\n- Captured at: **${latest.capturedAt}**\n- Timing snapshot: \`${DEFAULT_TIMINGS_PATH}\`${timingSnapshotCapturedAt ? ` captured at **${timingSnapshotCapturedAt}**` : ""}\n- Quarantine ledger: \`${DEFAULT_QUARANTINE_PATH}\`\n\n## Metrics\n\n| Metric | Current | Delta vs previous |\n|---|---:|---:|\n${renderMetricRow("Merge gate wall-time (`pnpm test:gate`)", latest, previous, "gateMs")}\n${renderMetricRow("Boot smoke wall-time (`pnpm smoke:boot`)", latest, previous, "bootSmokeMs")}\n${renderMetricRow("Changed-only test wall-time (`pnpm test`)", latest, previous, "testMs")}\n| Quarantine / flake count | ${latest.quarantineCount} | ${trendCell(latest.quarantineCount, previous?.quarantineCount)} |\n| Deletion-due quarantines | ${quarantine?.deletionDueCount ?? 0} | n/a |\n\n## Measurement failures\n\n${failures}\n\n## Slowest 20 test files\n\n| Rank | File | Package | Duration |\n|---:|---|---|---:|\n${slowRows || "| — | — | — | — |"}\n\n## Quarantine age buckets\n\n| Age bucket | Count |\n|---|---:|\n| 0-6 days | ${quarantine?.byAgeBucket?.["0-6d"] ?? 0} |\n| 7-13 days | ${quarantine?.byAgeBucket?.["7-13d"] ?? 0} |\n| deletion due (>=14 days) | ${quarantine?.byAgeBucket?.deletionDue ?? 0} |\n| unknown/future | ${quarantine?.byAgeBucket?.unknown ?? 0} |\n\n### Deletion-due entries\n\n| File | Quarantined at | Age (days) |\n|---|---:|---:|\n${dueRows || "| — | — | — |"}\n\n## Before / after trend\n\n| Row | Captured at | Gate | Boot smoke | \`pnpm test\` | Quarantine count |\n|---|---|---:|---:|---:|---:|\n${previousRows}\n\n_Future weekly rows append to \`${DEFAULT_HISTORY_PATH}\`; compare the latest row against the previous row before posting to #leads._\n\n## Post to #leads\n\n\`\`\`text\nFN-6612 weekly test velocity: gate ${formatDuration(latest.gateMs)} (${delta(latest, previous, "gateMs")}), boot smoke ${formatDuration(latest.bootSmokeMs)} (${delta(latest, previous, "bootSmokeMs")}), pnpm test ${formatDuration(latest.testMs)} (${delta(latest, previous, "testMs")}), quarantine ledger ${latest.quarantineCount} (${trendCell(latest.quarantineCount, previous?.quarantineCount)}). Slowest file: ${slowest[0]?.file ?? "none"} at ${formatDuration(slowest[0]?.ms)}. Deletion-due quarantines: ${quarantine?.deletionDueCount ?? 0}.\n\`\`\`\n\n## How to refresh\n\n\`\`\`bash\npnpm test:velocity -- --measure --write-report\n\`\`\`\n\nReport-only regeneration is cheap and does not run any suite:\n\n\`\`\`bash\npnpm test:velocity\n\`\`\`\n`; +} + +function historyEntries(history) { + if (Array.isArray(history)) return history; + if (Array.isArray(history?.entries)) return history.entries; + return []; +} + +function createEntry({ capturedAt = new Date().toISOString(), gateMs = null, bootSmokeMs = null, testMs = null, quarantine, slowest, measurementFailures = [], timingSnapshotCapturedAt = null }) { + return { + capturedAt, + gateMs: normalizeMs(gateMs), + bootSmokeMs: normalizeMs(bootSmokeMs), + testMs: normalizeMs(testMs), + quarantineCount: quarantine?.total ?? 0, + slowestTop20: slowest, + measurementFailures, + timingSnapshotCapturedAt, + }; +} + +function parseArgs(argv) { + const args = { measure: false, writeReport: false, reportOnly: true, timeoutMs: DEFAULT_MEASURE_TIMEOUT_MS, help: false }; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--") continue; + else if (arg === "--measure") args.measure = true; + else if (arg === "--write-report") args.writeReport = true; + else if (arg === "--report-only") args.reportOnly = true; + else if (arg === "--timeout-ms") args.timeoutMs = Number(argv[++index]); + else if (arg === "--help" || arg === "-h") args.help = true; + else throw new Error(`Unknown argument: ${arg}`); + } + if (!Number.isFinite(args.timeoutMs) || args.timeoutMs <= 0) { + throw new Error(`Expected --timeout-ms to be a positive number, got ${args.timeoutMs}`); + } + return args; +} + +async function timeCommand({ command, args, label, timeoutMs, cwd, stdout, stderr }) { + const started = performance.now(); + stderr.write(`[test-velocity] measuring ${label}\n`); + return new Promise((resolve) => { + const child = spawn(command, args, { cwd, shell: false, stdio: ["ignore", "pipe", "pipe"] }); + let settled = false; + const timer = setTimeout(() => { + if (settled) return; + settled = true; + child.kill("SIGTERM"); + const elapsedMs = Math.round(performance.now() - started); + resolve({ ms: null, failure: { label, status: `timeout after ${formatDuration(timeoutMs)} (${elapsedMs}ms elapsed)` } }); + }, timeoutMs); + + child.stdout.on("data", (chunk) => stdout.write(chunk)); + child.stderr.on("data", (chunk) => stderr.write(chunk)); + child.on("error", (error) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve({ ms: null, failure: { label, status: `spawn error: ${error.message}` } }); + }); + child.on("close", (code, signal) => { + if (settled) return; + settled = true; + clearTimeout(timer); + const elapsedMs = Math.round(performance.now() - started); + if (code === 0 && !signal) { + resolve({ ms: elapsedMs, failure: null }); + } else { + resolve({ ms: null, failure: { label, status: signal ? `signal ${signal} after ${formatDuration(elapsedMs)}` : `exit ${code} after ${formatDuration(elapsedMs)}` } }); + } + }); + }); +} + +async function measureCommands({ timeoutMs, cwd, stdout, stderr }) { + const results = {}; + const failures = []; + for (const measurement of MEASURE_COMMANDS) { + const result = await timeCommand({ ...measurement, timeoutMs, cwd, stdout, stderr }); + results[measurement.key] = result.ms; + if (result.failure) failures.push(result.failure); + } + return { ...results, measurementFailures: failures }; +} + +function renderFromEntry(entry, previous, quarantine) { + return renderReport({ + gateMs: entry?.gateMs, + bootSmokeMs: entry?.bootSmokeMs, + testMs: entry?.testMs, + slowest: entry?.slowestTop20 ?? [], + quarantine: quarantine ?? { total: entry?.quarantineCount ?? 0, byAgeBucket: {}, deletionDueEntries: [], deletionDueCount: 0 }, + capturedAt: entry?.capturedAt, + previous, + measurementFailures: entry?.measurementFailures ?? [], + timingSnapshotCapturedAt: entry?.timingSnapshotCapturedAt ?? null, + }); +} + +export async function main(argv = process.argv.slice(2), { rootDir = repoRoot, stdout = process.stdout, stderr = process.stderr, now = new Date() } = {}) { + let args; + try { + args = parseArgs(argv); + } catch (error) { + stderr.write(`${error.message}\n`); + return 1; + } + + if (args.help) { + stdout.write("Usage: node scripts/test-velocity-baseline.mjs [--measure] [--write-report] [--report-only] [--timeout-ms ]\n"); + return 0; + } + + const history = readJson(DEFAULT_HISTORY_PATH, { entries: [] }, rootDir); + const entries = historyEntries(history); + const timings = readJson(DEFAULT_TIMINGS_PATH, { packages: {} }, rootDir); + const quarantineJson = readJson(DEFAULT_QUARANTINE_PATH, { entries: [] }, rootDir); + const quarantine = readQuarantineCount(quarantineJson, { now }); + const slowest = topSlowestFiles(timings, 20); + + if (args.measure) { + const measured = await measureCommands({ timeoutMs: args.timeoutMs, cwd: rootDir, stdout, stderr }); + const entry = createEntry({ + capturedAt: now.toISOString(), + gateMs: measured.gateMs, + bootSmokeMs: measured.bootSmokeMs, + testMs: measured.testMs, + quarantine, + slowest, + measurementFailures: measured.measurementFailures, + timingSnapshotCapturedAt: timings?.capturedAt ?? null, + }); + entries.push(entry); + writeJson(DEFAULT_HISTORY_PATH, { entries }, rootDir); + } + + const latest = entries.at(-1) ?? createEntry({ + capturedAt: now.toISOString(), + quarantine, + slowest, + timingSnapshotCapturedAt: timings?.capturedAt ?? null, + }); + const previous = entries.length > 1 ? entries.at(-2) : null; + const report = renderFromEntry(latest, previous, quarantine); + + if (args.writeReport || args.reportOnly) { + writeText(DEFAULT_REPORT_PATH, report, rootDir); + stdout.write(`Updated ${DEFAULT_REPORT_PATH}\n`); + } else { + stdout.write(report); + } + + return 0; +} + +if (import.meta.url === pathToFileURL(process.argv[1]).href) { + /* + FNXC:TestVelocityBaseline 2026-06-17-00:00: + FN-6612 requires a weekly signal-per-second baseline for gate time, boot smoke time, changed-only test time, slowest files, and quarantine age. This script measures and reports those values only; it must stay out of blocking PR checks so the merge gate remains thin. + */ + const exitCode = await main(); + process.exitCode = exitCode; +} diff --git a/scripts/test-velocity-history.json b/scripts/test-velocity-history.json new file mode 100644 index 0000000000..a358d63ab8 --- /dev/null +++ b/scripts/test-velocity-history.json @@ -0,0 +1,231 @@ +{ + "entries": [ + { + "capturedAt": "2026-06-18T02:53:34.158Z", + "gateMs": 8478, + "bootSmokeMs": null, + "testMs": 38405, + "quarantineCount": 2, + "slowestTop20": [ + { + "file": "packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts", + "ms": 13900, + "package": "@fusion/engine" + }, + { + "file": "packages/core/src/__tests__/agent-store.test.ts", + "ms": 11600, + "package": "@fusion/core" + }, + { + "file": "packages/dashboard/src/__tests__/routes-agents.test.ts", + "ms": 11200, + "package": "@fusion/dashboard" + }, + { + "file": "packages/core/src/__tests__/mission-store.test.ts", + "ms": 10700, + "package": "@fusion/core" + }, + { + "file": "packages/core/src/__tests__/db.test.ts", + "ms": 10100, + "package": "@fusion/core" + }, + { + "file": "packages/dashboard/src/__tests__/routes-git.test.ts", + "ms": 9400, + "package": "@fusion/dashboard" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts", + "ms": 9000, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/merger-ai.test.ts", + "ms": 8700, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts", + "ms": 8400, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts", + "ms": 8400, + "package": "@fusion/engine" + }, + { + "file": "packages/core/src/__tests__/task-documents.test.ts", + "ms": 8300, + "package": "@fusion/core" + }, + { + "file": "packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts", + "ms": 7800, + "package": "@fusion/engine" + }, + { + "file": "packages/cli/src/__tests__/extension.test.ts", + "ms": 7000, + "package": "@runfusion/fusion" + }, + { + "file": "packages/core/src/__tests__/run-audit.test.ts", + "ms": 6900, + "package": "@fusion/core" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts", + "ms": 6100, + "package": "@fusion/engine" + }, + { + "file": "packages/dashboard/src/__tests__/routes-planning.test.ts", + "ms": 5600, + "package": "@fusion/dashboard" + }, + { + "file": "packages/core/src/__tests__/store-merge-queue.test.ts", + "ms": 5200, + "package": "@fusion/core" + }, + { + "file": "packages/dashboard/app/components/__tests__/FileEditor.test.tsx", + "ms": 5100, + "package": "@fusion/dashboard" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts", + "ms": 4900, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts", + "ms": 4900, + "package": "@fusion/engine" + } + ], + "measurementFailures": [ + { + "label": "Boot smoke (`pnpm smoke:boot`)", + "status": "exit 1 after 411ms" + } + ], + "timingSnapshotCapturedAt": "2026-06-03T23:45:49.672Z" + }, + { + "capturedAt": "2026-06-18T03:04:28.794Z", + "gateMs": 6177, + "bootSmokeMs": 18220, + "testMs": 7740, + "quarantineCount": 2, + "slowestTop20": [ + { + "file": "packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts", + "ms": 13900, + "package": "@fusion/engine" + }, + { + "file": "packages/core/src/__tests__/agent-store.test.ts", + "ms": 11600, + "package": "@fusion/core" + }, + { + "file": "packages/dashboard/src/__tests__/routes-agents.test.ts", + "ms": 11200, + "package": "@fusion/dashboard" + }, + { + "file": "packages/core/src/__tests__/mission-store.test.ts", + "ms": 10700, + "package": "@fusion/core" + }, + { + "file": "packages/core/src/__tests__/db.test.ts", + "ms": 10100, + "package": "@fusion/core" + }, + { + "file": "packages/dashboard/src/__tests__/routes-git.test.ts", + "ms": 9400, + "package": "@fusion/dashboard" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts", + "ms": 9000, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/merger-ai.test.ts", + "ms": 8700, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts", + "ms": 8400, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts", + "ms": 8400, + "package": "@fusion/engine" + }, + { + "file": "packages/core/src/__tests__/task-documents.test.ts", + "ms": 8300, + "package": "@fusion/core" + }, + { + "file": "packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts", + "ms": 7800, + "package": "@fusion/engine" + }, + { + "file": "packages/cli/src/__tests__/extension.test.ts", + "ms": 7000, + "package": "@runfusion/fusion" + }, + { + "file": "packages/core/src/__tests__/run-audit.test.ts", + "ms": 6900, + "package": "@fusion/core" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts", + "ms": 6100, + "package": "@fusion/engine" + }, + { + "file": "packages/dashboard/src/__tests__/routes-planning.test.ts", + "ms": 5600, + "package": "@fusion/dashboard" + }, + { + "file": "packages/core/src/__tests__/store-merge-queue.test.ts", + "ms": 5200, + "package": "@fusion/core" + }, + { + "file": "packages/dashboard/app/components/__tests__/FileEditor.test.tsx", + "ms": 5100, + "package": "@fusion/dashboard" + }, + { + "file": "packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts", + "ms": 4900, + "package": "@fusion/engine" + }, + { + "file": "packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts", + "ms": 4900, + "package": "@fusion/engine" + } + ], + "measurementFailures": [], + "timingSnapshotCapturedAt": "2026-06-03T23:45:49.672Z" + } + ] +}