FN-6617: add weekly test velocity baseline

Adds a report-only weekly baseline for test feedback-loop velocity.

- Add a test:velocity script that measures gate, boot smoke, and changed-only test durations and regenerates a #leads-ready report.
- Track committed baseline history, slowest-file summaries, and quarantine deletion-clock counts.
- Document the weekly refresh workflow in testing guidance and agent instructions.

Files changed:
 AGENTS.md                                         |   4 +-
 docs/test-velocity-baseline.md                    |  92 ++++++
 docs/testing.md                                   |  18 ++
 package.json                                      |   1 +
 scripts/__tests__/test-velocity-baseline.test.mjs | 121 ++++++++
 scripts/test-velocity-baseline.mjs                | 339 ++++++++++++++++++++++
 scripts/test-velocity-history.json                | 231 +++++++++++++++
 7 files changed, 805 insertions(+), 1 deletion(-)

Fusion-Task-Id: FN-6617

Fusion-Task-Lineage: 50a07298-74ec-4481-b288-4fadb7d3d29b
This commit is contained in:
gsxdsm
2026-06-17 20:09:05 -07:00
parent a013bc0309
commit fbd9ad2f3b
7 changed files with 805 additions and 1 deletions

View File

@@ -86,6 +86,7 @@ The merge gate is thin and trusted: CI blocks PRs on exactly Lint, Typecheck, Bu
pnpm test # gate suite + changed-only affected tests (bounded; never full-suite)
pnpm test:gate # the merge gate: curated engine-core suite + CI-shape test
pnpm smoke:boot # boot smoke: CLI --help + real serve /api/health
pnpm test:velocity # weekly report-only test velocity baseline; use -- --measure --write-report to refresh
pnpm test:full # full workspace suite — explicit opt-in only
pnpm lint
pnpm build
@@ -195,7 +196,8 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
## Reference docs (deeper detail)
- `./docs/architecture.md` — lifecycle invariants, self-healing rules, reliability interaction backstops, run-audit internals.
- `./docs/testing.md` — full testing lanes, worker fanout guidance, test taxonomy, and file organization.
- `./docs/testing.md` — full testing lanes, worker fanout guidance, test taxonomy, weekly velocity baseline, and file organization.
- `./docs/test-velocity-baseline.md` — weekly #leads-ready test feedback-loop velocity report generated by `scripts/test-velocity-baseline.mjs`.
- `./docs/dashboard-guide.md` — dashboard behavior and **Styling Guide** details. User-facing docs for Merge Advance Notice and Smart Pull live here.
- `./docs/PLUGIN_AUTHORING.md` — plugin authoring guide, lifecycle hooks, routes, tools, and dashboard-extension surfaces.
- `./docs/agents.md` — pi extension scope, coordination tools, checkout leasing, runtime config.

View File

@@ -0,0 +1,92 @@
# Test velocity baseline
> Weekly FN-6612 signal-per-second baseline. Measure and report feedback-loop velocity; do **not** add slow tests or wire this report into blocking PR checks. The merge gate remains the existing thin Lint, Typecheck, Build, and Gate path.
## Latest baseline
- Cycle: **2026-W25**
- Captured at: **2026-06-18T03:04:28.794Z**
- Timing snapshot: `scripts/test-timings.json` captured at **2026-06-03T23:45:49.672Z**
- Quarantine ledger: `scripts/lib/test-quarantine.json`
## Metrics
| Metric | Current | Delta vs previous |
|---|---:|---:|
| Merge gate wall-time (`pnpm test:gate`) | 6.2s | -2.3s |
| Boot smoke wall-time (`pnpm smoke:boot`) | 18.2s | n/a |
| Changed-only test wall-time (`pnpm test`) | 7.7s | -30.7s |
| Quarantine / flake count | 2 | 0 |
| Deletion-due quarantines | 0 | n/a |
## Measurement failures
- None recorded.
## Slowest 20 test files
| Rank | File | Package | Duration |
|---:|---|---|---:|
| 1 | `packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts` | @fusion/engine | 13.9s |
| 2 | `packages/core/src/__tests__/agent-store.test.ts` | @fusion/core | 11.6s |
| 3 | `packages/dashboard/src/__tests__/routes-agents.test.ts` | @fusion/dashboard | 11.2s |
| 4 | `packages/core/src/__tests__/mission-store.test.ts` | @fusion/core | 10.7s |
| 5 | `packages/core/src/__tests__/db.test.ts` | @fusion/core | 10.1s |
| 6 | `packages/dashboard/src/__tests__/routes-git.test.ts` | @fusion/dashboard | 9.4s |
| 7 | `packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts` | @fusion/engine | 9.0s |
| 8 | `packages/engine/src/__tests__/merger-ai.test.ts` | @fusion/engine | 8.7s |
| 9 | `packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts` | @fusion/engine | 8.4s |
| 10 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts` | @fusion/engine | 8.4s |
| 11 | `packages/core/src/__tests__/task-documents.test.ts` | @fusion/core | 8.3s |
| 12 | `packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts` | @fusion/engine | 7.8s |
| 13 | `packages/cli/src/__tests__/extension.test.ts` | @runfusion/fusion | 7.0s |
| 14 | `packages/core/src/__tests__/run-audit.test.ts` | @fusion/core | 6.9s |
| 15 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts` | @fusion/engine | 6.1s |
| 16 | `packages/dashboard/src/__tests__/routes-planning.test.ts` | @fusion/dashboard | 5.6s |
| 17 | `packages/core/src/__tests__/store-merge-queue.test.ts` | @fusion/core | 5.2s |
| 18 | `packages/dashboard/app/components/__tests__/FileEditor.test.tsx` | @fusion/dashboard | 5.1s |
| 19 | `packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts` | @fusion/engine | 4.9s |
| 20 | `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` | @fusion/engine | 4.9s |
## Quarantine age buckets
| Age bucket | Count |
|---|---:|
| 0-6 days | 2 |
| 7-13 days | 0 |
| deletion due (>=14 days) | 0 |
| unknown/future | 0 |
### Deletion-due entries
| File | Quarantined at | Age (days) |
|---|---:|---:|
| — | — | — |
## Before / after trend
| Row | Captured at | Gate | Boot smoke | `pnpm test` | Quarantine count |
|---|---|---:|---:|---:|---:|
| Previous | 2026-06-18T02:53:34.158Z | 8.5s | unavailable | 38.4s | 2 |
| Latest | 2026-06-18T03:04:28.794Z | 6.2s | 18.2s | 7.7s | 2 |
| Delta | — | -2.3s | n/a | -30.7s | 0 |
_Future weekly rows append to `scripts/test-velocity-history.json`; compare the latest row against the previous row before posting to #leads._
## Post to #leads
```text
FN-6612 weekly test velocity: gate 6.2s (-2.3s), boot smoke 18.2s (n/a), pnpm test 7.7s (-30.7s), quarantine ledger 2 (0). Slowest file: packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts at 13.9s. Deletion-due quarantines: 0.
```
## How to refresh
```bash
pnpm test:velocity -- --measure --write-report
```
Report-only regeneration is cheap and does not run any suite:
```bash
pnpm test:velocity
```

View File

@@ -225,6 +225,24 @@ shard artifacts into `.timings/` first (the default lookup directory), or pass
scheduled job can gate on freshness via `node scripts/ci-test-shard.mjs --check-timings-staleness`,
which exits non-zero when the snapshot is missing or older than the 30-day budget.
## Weekly test velocity baseline
FN-6612 tracks feedback-loop velocity as signal-per-second, not as a new blocking gate. Refresh the weekly baseline from a clean worktree with:
```bash
pnpm test:velocity -- --measure --write-report
```
The script runs `pnpm test:gate`, `pnpm smoke:boot`, and `pnpm test` with bounded async process supervision, then appends the measured row to `scripts/test-velocity-history.json` and rewrites the postable artifact at `docs/test-velocity-baseline.md`. It reads the slowest 20 files from the committed `scripts/test-timings.json` snapshot and the flake/quarantine count plus 14-day deletion-clock buckets directly from `scripts/lib/test-quarantine.json`; do not run the full suite just to populate the slowest-file table.
Use cheap report-only regeneration when measurements already exist:
```bash
pnpm test:velocity
```
Each week, copy the `Post to #leads` block from `docs/test-velocity-baseline.md`. If a measured command fails because the local environment is not ready, keep the failure recorded in the report instead of fabricating a time, then fix or rerun separately as appropriate. Do not wire `pnpm test:velocity`, `test:full`, or any slow-suite expansion into PR checks; the merge gate stays the thin Lint, Typecheck, Build, and Gate path.
## Targeted commands
```bash

View File

@@ -34,6 +34,7 @@
"test:scripts": "node --test scripts/__tests__/*.test.mjs",
"fn:cache-stats": "node scripts/cache-stats.mjs",
"test:full": "node scripts/test-changed.mjs --full --no-cache && pnpm --filter @fusion/engine test:slow",
"test:velocity": "node scripts/test-velocity-baseline.mjs",
"test:feedback-baseline": "node scripts/test-feedback-baseline.mjs",
"test:ci:shard": "node scripts/ci-test-shard.mjs",
"test:serial": "FUSION_TEST_CONCURRENCY=1 FUSION_TEST_WORKSPACE_CONCURRENCY=1 pnpm test:full",

View File

@@ -0,0 +1,121 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import {
readQuarantineCount,
renderReport,
topSlowestFiles,
} from "../test-velocity-baseline.mjs";
function makeTimings(count = 25) {
const files = {};
for (let index = 0; index < count; index += 1) {
files[`packages/a/src/__tests__/case-${String(index).padStart(2, "0")}.test.ts`] = 100 + index;
}
files["packages/a/src/__tests__/tie-z.test.ts"] = 500;
files["packages/a/src/__tests__/tie-a.test.ts"] = 500;
return {
packages: {
"@pkg/b": {
files: {
"packages/b/src/__tests__/winner.test.ts": 1000,
},
},
"@pkg/a": { files },
},
};
}
describe("topSlowestFiles", () => {
it("returns exactly 20 rows in descending duration order with package attribution and stable ties", () => {
const rows = topSlowestFiles(makeTimings(), 20);
assert.equal(rows.length, 20);
assert.deepEqual(rows[0], {
file: "packages/b/src/__tests__/winner.test.ts",
ms: 1000,
package: "@pkg/b",
});
assert.deepEqual(rows.slice(1, 3).map((row) => row.file), [
"packages/a/src/__tests__/tie-a.test.ts",
"packages/a/src/__tests__/tie-z.test.ts",
]);
assert.ok(rows.every((row, index) => index === 0 || rows[index - 1].ms >= row.ms));
assert.equal(rows[1].package, "@pkg/a");
});
});
describe("readQuarantineCount", () => {
it("counts entries by age bucket and flags deletion-due quarantines after 14 days", () => {
const result = readQuarantineCount(
{
entries: [
{ file: "fresh.test.ts", quarantinedAt: "2026-06-15" },
{ file: "warning.test.ts", quarantinedAt: "2026-06-08" },
{ file: "due.test.ts", quarantinedAt: "2026-06-01" },
{ file: "unknown.test.ts", quarantinedAt: "not-a-date" },
],
},
{ now: new Date("2026-06-17T12:00:00.000Z") },
);
assert.equal(result.total, 4);
assert.deepEqual(result.byAgeBucket, {
"0-6d": 1,
"7-13d": 1,
deletionDue: 1,
unknown: 1,
});
assert.deepEqual(result.deletionDueEntries, [
{ file: "due.test.ts", quarantinedAt: "2026-06-01", ageDays: 16 },
]);
});
});
describe("renderReport", () => {
it("includes metrics, slowest rows, quarantine count, and previous-run deltas", () => {
const report = renderReport({
gateMs: 12_000,
bootSmokeMs: 2_000,
testMs: 45_000,
capturedAt: "2026-06-17T12:00:00.000Z",
previous: {
capturedAt: "2026-06-10T12:00:00.000Z",
gateMs: 10_000,
bootSmokeMs: 3_000,
testMs: 50_000,
quarantineCount: 3,
},
slowest: [
{ file: "packages/a/src/__tests__/slow.test.ts", package: "@pkg/a", ms: 3210 },
],
quarantine: {
total: 2,
byAgeBucket: { "0-6d": 1, "7-13d": 1, deletionDue: 0, unknown: 0 },
deletionDueEntries: [],
deletionDueCount: 0,
},
});
assert.match(report, /\| Merge gate wall-time \(`pnpm test:gate`\) \| 12\.0s \| \+2\.0s \|/);
assert.match(report, /\| Boot smoke wall-time \(`pnpm smoke:boot`\) \| 2\.0s \| -1\.0s \|/);
assert.match(report, /\| Changed-only test wall-time \(`pnpm test`\) \| 45\.0s \| -5\.0s \|/);
assert.match(report, /\| Quarantine \/ flake count \| 2 \| -1 \|/);
assert.match(report, /`packages\/a\/src\/__tests__\/slow\.test\.ts` \| @pkg\/a \| 3\.2s/);
assert.match(report, /FN-6612 weekly test velocity: gate 12\.0s \(\+2\.0s\)/);
});
it("renders seed-baseline trend placeholders when there is no previous entry", () => {
const report = renderReport({
gateMs: 1_000,
bootSmokeMs: null,
testMs: 2_000,
capturedAt: "2026-06-17T12:00:00.000Z",
slowest: [],
quarantine: { total: 0, byAgeBucket: {}, deletionDueEntries: [], deletionDueCount: 0 },
});
assert.match(report, /\| Previous \| _\(seed baseline\)_ \| — \| — \| — \| — \|/);
assert.match(report, /\| Delta \| — \| n\/a \| n\/a \| n\/a \| n\/a \|/);
});
});

View File

@@ -0,0 +1,339 @@
#!/usr/bin/env node
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import path from "node:path";
import { spawn } from "node:child_process";
import { performance } from "node:perf_hooks";
import { fileURLToPath, pathToFileURL } from "node:url";
const currentFilePath = fileURLToPath(import.meta.url);
const repoRoot = path.resolve(path.dirname(currentFilePath), "..");
export const DEFAULT_TIMINGS_PATH = "scripts/test-timings.json";
export const DEFAULT_QUARANTINE_PATH = "scripts/lib/test-quarantine.json";
export const DEFAULT_HISTORY_PATH = "scripts/test-velocity-history.json";
export const DEFAULT_REPORT_PATH = "docs/test-velocity-baseline.md";
export const DEFAULT_MEASURE_TIMEOUT_MS = 10 * 60 * 1000;
export const DELETION_CLOCK_DAYS = 14;
const MEASURE_COMMANDS = [
{ key: "gateMs", label: "Merge gate (`pnpm test:gate`)", command: "pnpm", args: ["test:gate"] },
{ key: "bootSmokeMs", label: "Boot smoke (`pnpm smoke:boot`)", command: "pnpm", args: ["smoke:boot"] },
{ key: "testMs", label: "Changed-only tests (`pnpm test`)", command: "pnpm", args: ["test"] },
];
function readJson(relativePath, fallback = null, rootDir = repoRoot) {
const absolutePath = path.join(rootDir, relativePath);
if (!existsSync(absolutePath)) return fallback;
return JSON.parse(readFileSync(absolutePath, "utf8"));
}
function writeJson(relativePath, value, rootDir = repoRoot) {
const absolutePath = path.join(rootDir, relativePath);
mkdirSync(path.dirname(absolutePath), { recursive: true });
writeFileSync(absolutePath, `${JSON.stringify(value, null, 2)}\n`, "utf8");
}
function writeText(relativePath, value, rootDir = repoRoot) {
const absolutePath = path.join(rootDir, relativePath);
mkdirSync(path.dirname(absolutePath), { recursive: true });
writeFileSync(absolutePath, value, "utf8");
}
function normalizeMs(value) {
if (value == null || value === "") return null;
const parsed = Number(value);
if (!Number.isFinite(parsed) || parsed < 0) {
throw new Error(`Expected a non-negative millisecond value, got ${value}`);
}
return Math.round(parsed);
}
function toDate(value) {
if (!value) return null;
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? null : parsed;
}
function ageDays(quarantinedAt, now) {
const quarantinedAtDate = toDate(quarantinedAt);
if (!quarantinedAtDate) return null;
return Math.floor((now.getTime() - quarantinedAtDate.getTime()) / 86_400_000);
}
function isoWeek(date) {
const working = new Date(Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate()));
const day = working.getUTCDay() || 7;
working.setUTCDate(working.getUTCDate() + 4 - day);
const yearStart = new Date(Date.UTC(working.getUTCFullYear(), 0, 1));
const week = Math.ceil(((working - yearStart) / 86_400_000 + 1) / 7);
return `${working.getUTCFullYear()}-W${String(week).padStart(2, "0")}`;
}
export function formatDuration(ms) {
if (ms == null) return "unavailable";
const rounded = Math.round(ms);
const sign = rounded < 0 ? "-" : "";
const absoluteMs = Math.abs(rounded);
if (absoluteMs < 1000) return `${sign}${absoluteMs}ms`;
const seconds = absoluteMs / 1000;
if (seconds < 60) return `${sign}${seconds.toFixed(1)}s`;
const minutes = Math.floor(seconds / 60);
const remainingSeconds = Math.round(seconds - minutes * 60);
return `${sign}${minutes}m ${String(remainingSeconds).padStart(2, "0")}s`;
}
export function readQuarantineCount(json, { now = new Date() } = {}) {
const entries = Array.isArray(json?.entries) ? json.entries : [];
const buckets = {
"0-6d": 0,
"7-13d": 0,
deletionDue: 0,
unknown: 0,
};
const deletionDueEntries = [];
for (const entry of entries) {
const age = ageDays(entry?.quarantinedAt, now);
const normalized = {
file: entry?.file ?? "unknown",
quarantinedAt: entry?.quarantinedAt ?? null,
ageDays: age,
};
if (age == null || age < 0) {
buckets.unknown += 1;
} else if (age >= DELETION_CLOCK_DAYS) {
buckets.deletionDue += 1;
deletionDueEntries.push(normalized);
} else if (age >= 7) {
buckets["7-13d"] += 1;
} else {
buckets["0-6d"] += 1;
}
}
return {
total: entries.length,
byAgeBucket: buckets,
deletionDueEntries,
deletionDueCount: deletionDueEntries.length,
};
}
export function topSlowestFiles(timingsJson, n = 20) {
const rows = [];
for (const [packageName, packageTiming] of Object.entries(timingsJson?.packages ?? {})) {
for (const [file, ms] of Object.entries(packageTiming?.files ?? {})) {
const parsed = Number(ms);
rows.push({ file, ms: Number.isFinite(parsed) ? parsed : 0, package: packageName });
}
}
rows.sort((a, b) => b.ms - a.ms || a.package.localeCompare(b.package) || a.file.localeCompare(b.file));
return rows.slice(0, n);
}
function delta(latest, previous, field) {
if (!previous || latest?.[field] == null || previous?.[field] == null) return "n/a";
const diff = latest[field] - previous[field];
const sign = diff > 0 ? "+" : "";
return `${sign}${formatDuration(diff)}`;
}
function renderMetricRow(name, latest, previous, field) {
return `| ${name} | ${formatDuration(latest?.[field])} | ${delta(latest, previous, field)} |`;
}
function trendCell(current, prior) {
if (current == null || prior == null) return "n/a";
const diff = current - prior;
return `${diff > 0 ? "+" : ""}${diff}`;
}
export function renderReport({ gateMs, bootSmokeMs, testMs, slowest = [], quarantine, capturedAt, previous = null, measurementFailures = [], timingSnapshotCapturedAt = null } = {}) {
const latest = {
gateMs: normalizeMs(gateMs),
bootSmokeMs: normalizeMs(bootSmokeMs),
testMs: normalizeMs(testMs),
quarantineCount: quarantine?.total ?? quarantine?.quarantineCount ?? 0,
capturedAt: capturedAt ?? new Date().toISOString(),
};
const cycle = isoWeek(new Date(latest.capturedAt));
const slowRows = slowest
.map((row, index) => `| ${index + 1} | \`${row.file}\` | ${row.package} | ${formatDuration(row.ms)} |`)
.join("\n");
const dueRows = (quarantine?.deletionDueEntries ?? [])
.map((entry) => `| \`${entry.file}\` | ${entry.quarantinedAt ?? "unknown"} | ${entry.ageDays ?? "unknown"} |`)
.join("\n");
const failures = measurementFailures.length > 0
? measurementFailures.map((failure) => `- ${failure.label}: ${failure.status}`).join("\n")
: "- None recorded.";
const previousRows = previous
? `| Previous | ${previous.capturedAt ?? "unknown"} | ${formatDuration(previous.gateMs)} | ${formatDuration(previous.bootSmokeMs)} | ${formatDuration(previous.testMs)} | ${previous.quarantineCount ?? "n/a"} |\n| Latest | ${latest.capturedAt} | ${formatDuration(latest.gateMs)} | ${formatDuration(latest.bootSmokeMs)} | ${formatDuration(latest.testMs)} | ${latest.quarantineCount} |\n| Delta | — | ${delta(latest, previous, "gateMs")} | ${delta(latest, previous, "bootSmokeMs")} | ${delta(latest, previous, "testMs")} | ${trendCell(latest.quarantineCount, previous.quarantineCount)} |`
: `| Previous | _(seed baseline)_ | — | — | — | — |\n| Latest | ${latest.capturedAt} | ${formatDuration(latest.gateMs)} | ${formatDuration(latest.bootSmokeMs)} | ${formatDuration(latest.testMs)} | ${latest.quarantineCount} |\n| Delta | — | n/a | n/a | n/a | n/a |`;
return `# Test velocity baseline\n\n> Weekly FN-6612 signal-per-second baseline. Measure and report feedback-loop velocity; do **not** add slow tests or wire this report into blocking PR checks. The merge gate remains the existing thin Lint, Typecheck, Build, and Gate path.\n\n## Latest baseline\n\n- Cycle: **${cycle}**\n- Captured at: **${latest.capturedAt}**\n- Timing snapshot: \`${DEFAULT_TIMINGS_PATH}\`${timingSnapshotCapturedAt ? ` captured at **${timingSnapshotCapturedAt}**` : ""}\n- Quarantine ledger: \`${DEFAULT_QUARANTINE_PATH}\`\n\n## Metrics\n\n| Metric | Current | Delta vs previous |\n|---|---:|---:|\n${renderMetricRow("Merge gate wall-time (`pnpm test:gate`)", latest, previous, "gateMs")}\n${renderMetricRow("Boot smoke wall-time (`pnpm smoke:boot`)", latest, previous, "bootSmokeMs")}\n${renderMetricRow("Changed-only test wall-time (`pnpm test`)", latest, previous, "testMs")}\n| Quarantine / flake count | ${latest.quarantineCount} | ${trendCell(latest.quarantineCount, previous?.quarantineCount)} |\n| Deletion-due quarantines | ${quarantine?.deletionDueCount ?? 0} | n/a |\n\n## Measurement failures\n\n${failures}\n\n## Slowest 20 test files\n\n| Rank | File | Package | Duration |\n|---:|---|---|---:|\n${slowRows || "| — | — | — | — |"}\n\n## Quarantine age buckets\n\n| Age bucket | Count |\n|---|---:|\n| 0-6 days | ${quarantine?.byAgeBucket?.["0-6d"] ?? 0} |\n| 7-13 days | ${quarantine?.byAgeBucket?.["7-13d"] ?? 0} |\n| deletion due (>=14 days) | ${quarantine?.byAgeBucket?.deletionDue ?? 0} |\n| unknown/future | ${quarantine?.byAgeBucket?.unknown ?? 0} |\n\n### Deletion-due entries\n\n| File | Quarantined at | Age (days) |\n|---|---:|---:|\n${dueRows || "| — | — | — |"}\n\n## Before / after trend\n\n| Row | Captured at | Gate | Boot smoke | \`pnpm test\` | Quarantine count |\n|---|---|---:|---:|---:|---:|\n${previousRows}\n\n_Future weekly rows append to \`${DEFAULT_HISTORY_PATH}\`; compare the latest row against the previous row before posting to #leads._\n\n## Post to #leads\n\n\`\`\`text\nFN-6612 weekly test velocity: gate ${formatDuration(latest.gateMs)} (${delta(latest, previous, "gateMs")}), boot smoke ${formatDuration(latest.bootSmokeMs)} (${delta(latest, previous, "bootSmokeMs")}), pnpm test ${formatDuration(latest.testMs)} (${delta(latest, previous, "testMs")}), quarantine ledger ${latest.quarantineCount} (${trendCell(latest.quarantineCount, previous?.quarantineCount)}). Slowest file: ${slowest[0]?.file ?? "none"} at ${formatDuration(slowest[0]?.ms)}. Deletion-due quarantines: ${quarantine?.deletionDueCount ?? 0}.\n\`\`\`\n\n## How to refresh\n\n\`\`\`bash\npnpm test:velocity -- --measure --write-report\n\`\`\`\n\nReport-only regeneration is cheap and does not run any suite:\n\n\`\`\`bash\npnpm test:velocity\n\`\`\`\n`;
}
function historyEntries(history) {
if (Array.isArray(history)) return history;
if (Array.isArray(history?.entries)) return history.entries;
return [];
}
function createEntry({ capturedAt = new Date().toISOString(), gateMs = null, bootSmokeMs = null, testMs = null, quarantine, slowest, measurementFailures = [], timingSnapshotCapturedAt = null }) {
return {
capturedAt,
gateMs: normalizeMs(gateMs),
bootSmokeMs: normalizeMs(bootSmokeMs),
testMs: normalizeMs(testMs),
quarantineCount: quarantine?.total ?? 0,
slowestTop20: slowest,
measurementFailures,
timingSnapshotCapturedAt,
};
}
function parseArgs(argv) {
const args = { measure: false, writeReport: false, reportOnly: true, timeoutMs: DEFAULT_MEASURE_TIMEOUT_MS, help: false };
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--") continue;
else if (arg === "--measure") args.measure = true;
else if (arg === "--write-report") args.writeReport = true;
else if (arg === "--report-only") args.reportOnly = true;
else if (arg === "--timeout-ms") args.timeoutMs = Number(argv[++index]);
else if (arg === "--help" || arg === "-h") args.help = true;
else throw new Error(`Unknown argument: ${arg}`);
}
if (!Number.isFinite(args.timeoutMs) || args.timeoutMs <= 0) {
throw new Error(`Expected --timeout-ms to be a positive number, got ${args.timeoutMs}`);
}
return args;
}
async function timeCommand({ command, args, label, timeoutMs, cwd, stdout, stderr }) {
const started = performance.now();
stderr.write(`[test-velocity] measuring ${label}\n`);
return new Promise((resolve) => {
const child = spawn(command, args, { cwd, shell: false, stdio: ["ignore", "pipe", "pipe"] });
let settled = false;
const timer = setTimeout(() => {
if (settled) return;
settled = true;
child.kill("SIGTERM");
const elapsedMs = Math.round(performance.now() - started);
resolve({ ms: null, failure: { label, status: `timeout after ${formatDuration(timeoutMs)} (${elapsedMs}ms elapsed)` } });
}, timeoutMs);
child.stdout.on("data", (chunk) => stdout.write(chunk));
child.stderr.on("data", (chunk) => stderr.write(chunk));
child.on("error", (error) => {
if (settled) return;
settled = true;
clearTimeout(timer);
resolve({ ms: null, failure: { label, status: `spawn error: ${error.message}` } });
});
child.on("close", (code, signal) => {
if (settled) return;
settled = true;
clearTimeout(timer);
const elapsedMs = Math.round(performance.now() - started);
if (code === 0 && !signal) {
resolve({ ms: elapsedMs, failure: null });
} else {
resolve({ ms: null, failure: { label, status: signal ? `signal ${signal} after ${formatDuration(elapsedMs)}` : `exit ${code} after ${formatDuration(elapsedMs)}` } });
}
});
});
}
async function measureCommands({ timeoutMs, cwd, stdout, stderr }) {
const results = {};
const failures = [];
for (const measurement of MEASURE_COMMANDS) {
const result = await timeCommand({ ...measurement, timeoutMs, cwd, stdout, stderr });
results[measurement.key] = result.ms;
if (result.failure) failures.push(result.failure);
}
return { ...results, measurementFailures: failures };
}
function renderFromEntry(entry, previous, quarantine) {
return renderReport({
gateMs: entry?.gateMs,
bootSmokeMs: entry?.bootSmokeMs,
testMs: entry?.testMs,
slowest: entry?.slowestTop20 ?? [],
quarantine: quarantine ?? { total: entry?.quarantineCount ?? 0, byAgeBucket: {}, deletionDueEntries: [], deletionDueCount: 0 },
capturedAt: entry?.capturedAt,
previous,
measurementFailures: entry?.measurementFailures ?? [],
timingSnapshotCapturedAt: entry?.timingSnapshotCapturedAt ?? null,
});
}
export async function main(argv = process.argv.slice(2), { rootDir = repoRoot, stdout = process.stdout, stderr = process.stderr, now = new Date() } = {}) {
let args;
try {
args = parseArgs(argv);
} catch (error) {
stderr.write(`${error.message}\n`);
return 1;
}
if (args.help) {
stdout.write("Usage: node scripts/test-velocity-baseline.mjs [--measure] [--write-report] [--report-only] [--timeout-ms <ms>]\n");
return 0;
}
const history = readJson(DEFAULT_HISTORY_PATH, { entries: [] }, rootDir);
const entries = historyEntries(history);
const timings = readJson(DEFAULT_TIMINGS_PATH, { packages: {} }, rootDir);
const quarantineJson = readJson(DEFAULT_QUARANTINE_PATH, { entries: [] }, rootDir);
const quarantine = readQuarantineCount(quarantineJson, { now });
const slowest = topSlowestFiles(timings, 20);
if (args.measure) {
const measured = await measureCommands({ timeoutMs: args.timeoutMs, cwd: rootDir, stdout, stderr });
const entry = createEntry({
capturedAt: now.toISOString(),
gateMs: measured.gateMs,
bootSmokeMs: measured.bootSmokeMs,
testMs: measured.testMs,
quarantine,
slowest,
measurementFailures: measured.measurementFailures,
timingSnapshotCapturedAt: timings?.capturedAt ?? null,
});
entries.push(entry);
writeJson(DEFAULT_HISTORY_PATH, { entries }, rootDir);
}
const latest = entries.at(-1) ?? createEntry({
capturedAt: now.toISOString(),
quarantine,
slowest,
timingSnapshotCapturedAt: timings?.capturedAt ?? null,
});
const previous = entries.length > 1 ? entries.at(-2) : null;
const report = renderFromEntry(latest, previous, quarantine);
if (args.writeReport || args.reportOnly) {
writeText(DEFAULT_REPORT_PATH, report, rootDir);
stdout.write(`Updated ${DEFAULT_REPORT_PATH}\n`);
} else {
stdout.write(report);
}
return 0;
}
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
/*
FNXC:TestVelocityBaseline 2026-06-17-00:00:
FN-6612 requires a weekly signal-per-second baseline for gate time, boot smoke time, changed-only test time, slowest files, and quarantine age. This script measures and reports those values only; it must stay out of blocking PR checks so the merge gate remains thin.
*/
const exitCode = await main();
process.exitCode = exitCode;
}

View File

@@ -0,0 +1,231 @@
{
"entries": [
{
"capturedAt": "2026-06-18T02:53:34.158Z",
"gateMs": 8478,
"bootSmokeMs": null,
"testMs": 38405,
"quarantineCount": 2,
"slowestTop20": [
{
"file": "packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts",
"ms": 13900,
"package": "@fusion/engine"
},
{
"file": "packages/core/src/__tests__/agent-store.test.ts",
"ms": 11600,
"package": "@fusion/core"
},
{
"file": "packages/dashboard/src/__tests__/routes-agents.test.ts",
"ms": 11200,
"package": "@fusion/dashboard"
},
{
"file": "packages/core/src/__tests__/mission-store.test.ts",
"ms": 10700,
"package": "@fusion/core"
},
{
"file": "packages/core/src/__tests__/db.test.ts",
"ms": 10100,
"package": "@fusion/core"
},
{
"file": "packages/dashboard/src/__tests__/routes-git.test.ts",
"ms": 9400,
"package": "@fusion/dashboard"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts",
"ms": 9000,
"package": "@fusion/engine"
},
{
"file": "packages/engine/src/__tests__/merger-ai.test.ts",
"ms": 8700,
"package": "@fusion/engine"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts",
"ms": 8400,
"package": "@fusion/engine"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts",
"ms": 8400,
"package": "@fusion/engine"
},
{
"file": "packages/core/src/__tests__/task-documents.test.ts",
"ms": 8300,
"package": "@fusion/core"
},
{
"file": "packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts",
"ms": 7800,
"package": "@fusion/engine"
},
{
"file": "packages/cli/src/__tests__/extension.test.ts",
"ms": 7000,
"package": "@runfusion/fusion"
},
{
"file": "packages/core/src/__tests__/run-audit.test.ts",
"ms": 6900,
"package": "@fusion/core"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts",
"ms": 6100,
"package": "@fusion/engine"
},
{
"file": "packages/dashboard/src/__tests__/routes-planning.test.ts",
"ms": 5600,
"package": "@fusion/dashboard"
},
{
"file": "packages/core/src/__tests__/store-merge-queue.test.ts",
"ms": 5200,
"package": "@fusion/core"
},
{
"file": "packages/dashboard/app/components/__tests__/FileEditor.test.tsx",
"ms": 5100,
"package": "@fusion/dashboard"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts",
"ms": 4900,
"package": "@fusion/engine"
},
{
"file": "packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts",
"ms": 4900,
"package": "@fusion/engine"
}
],
"measurementFailures": [
{
"label": "Boot smoke (`pnpm smoke:boot`)",
"status": "exit 1 after 411ms"
}
],
"timingSnapshotCapturedAt": "2026-06-03T23:45:49.672Z"
},
{
"capturedAt": "2026-06-18T03:04:28.794Z",
"gateMs": 6177,
"bootSmokeMs": 18220,
"testMs": 7740,
"quarantineCount": 2,
"slowestTop20": [
{
"file": "packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts",
"ms": 13900,
"package": "@fusion/engine"
},
{
"file": "packages/core/src/__tests__/agent-store.test.ts",
"ms": 11600,
"package": "@fusion/core"
},
{
"file": "packages/dashboard/src/__tests__/routes-agents.test.ts",
"ms": 11200,
"package": "@fusion/dashboard"
},
{
"file": "packages/core/src/__tests__/mission-store.test.ts",
"ms": 10700,
"package": "@fusion/core"
},
{
"file": "packages/core/src/__tests__/db.test.ts",
"ms": 10100,
"package": "@fusion/core"
},
{
"file": "packages/dashboard/src/__tests__/routes-git.test.ts",
"ms": 9400,
"package": "@fusion/dashboard"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts",
"ms": 9000,
"package": "@fusion/engine"
},
{
"file": "packages/engine/src/__tests__/merger-ai.test.ts",
"ms": 8700,
"package": "@fusion/engine"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts",
"ms": 8400,
"package": "@fusion/engine"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts",
"ms": 8400,
"package": "@fusion/engine"
},
{
"file": "packages/core/src/__tests__/task-documents.test.ts",
"ms": 8300,
"package": "@fusion/core"
},
{
"file": "packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts",
"ms": 7800,
"package": "@fusion/engine"
},
{
"file": "packages/cli/src/__tests__/extension.test.ts",
"ms": 7000,
"package": "@runfusion/fusion"
},
{
"file": "packages/core/src/__tests__/run-audit.test.ts",
"ms": 6900,
"package": "@fusion/core"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts",
"ms": 6100,
"package": "@fusion/engine"
},
{
"file": "packages/dashboard/src/__tests__/routes-planning.test.ts",
"ms": 5600,
"package": "@fusion/dashboard"
},
{
"file": "packages/core/src/__tests__/store-merge-queue.test.ts",
"ms": 5200,
"package": "@fusion/core"
},
{
"file": "packages/dashboard/app/components/__tests__/FileEditor.test.tsx",
"ms": 5100,
"package": "@fusion/dashboard"
},
{
"file": "packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts",
"ms": 4900,
"package": "@fusion/engine"
},
{
"file": "packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts",
"ms": 4900,
"package": "@fusion/engine"
}
],
"measurementFailures": [],
"timingSnapshotCapturedAt": "2026-06-03T23:45:49.672Z"
}
]
}