FN-6617: add weekly test velocity baseline
Adds a report-only weekly baseline for test feedback-loop velocity. - Add a test:velocity script that measures gate, boot smoke, and changed-only test durations and regenerates a #leads-ready report. - Track committed baseline history, slowest-file summaries, and quarantine deletion-clock counts. - Document the weekly refresh workflow in testing guidance and agent instructions. Files changed: AGENTS.md | 4 +- docs/test-velocity-baseline.md | 92 ++++++ docs/testing.md | 18 ++ package.json | 1 + scripts/__tests__/test-velocity-baseline.test.mjs | 121 ++++++++ scripts/test-velocity-baseline.mjs | 339 ++++++++++++++++++++++ scripts/test-velocity-history.json | 231 +++++++++++++++ 7 files changed, 805 insertions(+), 1 deletion(-) Fusion-Task-Id: FN-6617 Fusion-Task-Lineage: 50a07298-74ec-4481-b288-4fadb7d3d29b
This commit is contained in:
@@ -86,6 +86,7 @@ The merge gate is thin and trusted: CI blocks PRs on exactly Lint, Typecheck, Bu
|
||||
pnpm test # gate suite + changed-only affected tests (bounded; never full-suite)
|
||||
pnpm test:gate # the merge gate: curated engine-core suite + CI-shape test
|
||||
pnpm smoke:boot # boot smoke: CLI --help + real serve /api/health
|
||||
pnpm test:velocity # weekly report-only test velocity baseline; use -- --measure --write-report to refresh
|
||||
pnpm test:full # full workspace suite — explicit opt-in only
|
||||
pnpm lint
|
||||
pnpm build
|
||||
@@ -195,7 +196,8 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
|
||||
## Reference docs (deeper detail)
|
||||
|
||||
- `./docs/architecture.md` — lifecycle invariants, self-healing rules, reliability interaction backstops, run-audit internals.
|
||||
- `./docs/testing.md` — full testing lanes, worker fanout guidance, test taxonomy, and file organization.
|
||||
- `./docs/testing.md` — full testing lanes, worker fanout guidance, test taxonomy, weekly velocity baseline, and file organization.
|
||||
- `./docs/test-velocity-baseline.md` — weekly #leads-ready test feedback-loop velocity report generated by `scripts/test-velocity-baseline.mjs`.
|
||||
- `./docs/dashboard-guide.md` — dashboard behavior and **Styling Guide** details. User-facing docs for Merge Advance Notice and Smart Pull live here.
|
||||
- `./docs/PLUGIN_AUTHORING.md` — plugin authoring guide, lifecycle hooks, routes, tools, and dashboard-extension surfaces.
|
||||
- `./docs/agents.md` — pi extension scope, coordination tools, checkout leasing, runtime config.
|
||||
|
||||
92
docs/test-velocity-baseline.md
Normal file
92
docs/test-velocity-baseline.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# Test velocity baseline
|
||||
|
||||
> Weekly FN-6612 signal-per-second baseline. Measure and report feedback-loop velocity; do **not** add slow tests or wire this report into blocking PR checks. The merge gate remains the existing thin Lint, Typecheck, Build, and Gate path.
|
||||
|
||||
## Latest baseline
|
||||
|
||||
- Cycle: **2026-W25**
|
||||
- Captured at: **2026-06-18T03:04:28.794Z**
|
||||
- Timing snapshot: `scripts/test-timings.json` captured at **2026-06-03T23:45:49.672Z**
|
||||
- Quarantine ledger: `scripts/lib/test-quarantine.json`
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Current | Delta vs previous |
|
||||
|---|---:|---:|
|
||||
| Merge gate wall-time (`pnpm test:gate`) | 6.2s | -2.3s |
|
||||
| Boot smoke wall-time (`pnpm smoke:boot`) | 18.2s | n/a |
|
||||
| Changed-only test wall-time (`pnpm test`) | 7.7s | -30.7s |
|
||||
| Quarantine / flake count | 2 | 0 |
|
||||
| Deletion-due quarantines | 0 | n/a |
|
||||
|
||||
## Measurement failures
|
||||
|
||||
- None recorded.
|
||||
|
||||
## Slowest 20 test files
|
||||
|
||||
| Rank | File | Package | Duration |
|
||||
|---:|---|---|---:|
|
||||
| 1 | `packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts` | @fusion/engine | 13.9s |
|
||||
| 2 | `packages/core/src/__tests__/agent-store.test.ts` | @fusion/core | 11.6s |
|
||||
| 3 | `packages/dashboard/src/__tests__/routes-agents.test.ts` | @fusion/dashboard | 11.2s |
|
||||
| 4 | `packages/core/src/__tests__/mission-store.test.ts` | @fusion/core | 10.7s |
|
||||
| 5 | `packages/core/src/__tests__/db.test.ts` | @fusion/core | 10.1s |
|
||||
| 6 | `packages/dashboard/src/__tests__/routes-git.test.ts` | @fusion/dashboard | 9.4s |
|
||||
| 7 | `packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts` | @fusion/engine | 9.0s |
|
||||
| 8 | `packages/engine/src/__tests__/merger-ai.test.ts` | @fusion/engine | 8.7s |
|
||||
| 9 | `packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts` | @fusion/engine | 8.4s |
|
||||
| 10 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts` | @fusion/engine | 8.4s |
|
||||
| 11 | `packages/core/src/__tests__/task-documents.test.ts` | @fusion/core | 8.3s |
|
||||
| 12 | `packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts` | @fusion/engine | 7.8s |
|
||||
| 13 | `packages/cli/src/__tests__/extension.test.ts` | @runfusion/fusion | 7.0s |
|
||||
| 14 | `packages/core/src/__tests__/run-audit.test.ts` | @fusion/core | 6.9s |
|
||||
| 15 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts` | @fusion/engine | 6.1s |
|
||||
| 16 | `packages/dashboard/src/__tests__/routes-planning.test.ts` | @fusion/dashboard | 5.6s |
|
||||
| 17 | `packages/core/src/__tests__/store-merge-queue.test.ts` | @fusion/core | 5.2s |
|
||||
| 18 | `packages/dashboard/app/components/__tests__/FileEditor.test.tsx` | @fusion/dashboard | 5.1s |
|
||||
| 19 | `packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts` | @fusion/engine | 4.9s |
|
||||
| 20 | `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` | @fusion/engine | 4.9s |
|
||||
|
||||
## Quarantine age buckets
|
||||
|
||||
| Age bucket | Count |
|
||||
|---|---:|
|
||||
| 0-6 days | 2 |
|
||||
| 7-13 days | 0 |
|
||||
| deletion due (>=14 days) | 0 |
|
||||
| unknown/future | 0 |
|
||||
|
||||
### Deletion-due entries
|
||||
|
||||
| File | Quarantined at | Age (days) |
|
||||
|---|---:|---:|
|
||||
| — | — | — |
|
||||
|
||||
## Before / after trend
|
||||
|
||||
| Row | Captured at | Gate | Boot smoke | `pnpm test` | Quarantine count |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| Previous | 2026-06-18T02:53:34.158Z | 8.5s | unavailable | 38.4s | 2 |
|
||||
| Latest | 2026-06-18T03:04:28.794Z | 6.2s | 18.2s | 7.7s | 2 |
|
||||
| Delta | — | -2.3s | n/a | -30.7s | 0 |
|
||||
|
||||
_Future weekly rows append to `scripts/test-velocity-history.json`; compare the latest row against the previous row before posting to #leads._
|
||||
|
||||
## Post to #leads
|
||||
|
||||
```text
|
||||
FN-6612 weekly test velocity: gate 6.2s (-2.3s), boot smoke 18.2s (n/a), pnpm test 7.7s (-30.7s), quarantine ledger 2 (0). Slowest file: packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts at 13.9s. Deletion-due quarantines: 0.
|
||||
```
|
||||
|
||||
## How to refresh
|
||||
|
||||
```bash
|
||||
pnpm test:velocity -- --measure --write-report
|
||||
```
|
||||
|
||||
Report-only regeneration is cheap and does not run any suite:
|
||||
|
||||
```bash
|
||||
pnpm test:velocity
|
||||
```
|
||||
@@ -225,6 +225,24 @@ shard artifacts into `.timings/` first (the default lookup directory), or pass
|
||||
scheduled job can gate on freshness via `node scripts/ci-test-shard.mjs --check-timings-staleness`,
|
||||
which exits non-zero when the snapshot is missing or older than the 30-day budget.
|
||||
|
||||
## Weekly test velocity baseline
|
||||
|
||||
FN-6612 tracks feedback-loop velocity as signal-per-second, not as a new blocking gate. Refresh the weekly baseline from a clean worktree with:
|
||||
|
||||
```bash
|
||||
pnpm test:velocity -- --measure --write-report
|
||||
```
|
||||
|
||||
The script runs `pnpm test:gate`, `pnpm smoke:boot`, and `pnpm test` with bounded async process supervision, then appends the measured row to `scripts/test-velocity-history.json` and rewrites the postable artifact at `docs/test-velocity-baseline.md`. It reads the slowest 20 files from the committed `scripts/test-timings.json` snapshot and the flake/quarantine count plus 14-day deletion-clock buckets directly from `scripts/lib/test-quarantine.json`; do not run the full suite just to populate the slowest-file table.
|
||||
|
||||
Use cheap report-only regeneration when measurements already exist:
|
||||
|
||||
```bash
|
||||
pnpm test:velocity
|
||||
```
|
||||
|
||||
Each week, copy the `Post to #leads` block from `docs/test-velocity-baseline.md`. If a measured command fails because the local environment is not ready, keep the failure recorded in the report instead of fabricating a time, then fix or rerun separately as appropriate. Do not wire `pnpm test:velocity`, `test:full`, or any slow-suite expansion into PR checks; the merge gate stays the thin Lint, Typecheck, Build, and Gate path.
|
||||
|
||||
## Targeted commands
|
||||
|
||||
```bash
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
"test:scripts": "node --test scripts/__tests__/*.test.mjs",
|
||||
"fn:cache-stats": "node scripts/cache-stats.mjs",
|
||||
"test:full": "node scripts/test-changed.mjs --full --no-cache && pnpm --filter @fusion/engine test:slow",
|
||||
"test:velocity": "node scripts/test-velocity-baseline.mjs",
|
||||
"test:feedback-baseline": "node scripts/test-feedback-baseline.mjs",
|
||||
"test:ci:shard": "node scripts/ci-test-shard.mjs",
|
||||
"test:serial": "FUSION_TEST_CONCURRENCY=1 FUSION_TEST_WORKSPACE_CONCURRENCY=1 pnpm test:full",
|
||||
|
||||
121
scripts/__tests__/test-velocity-baseline.test.mjs
Normal file
121
scripts/__tests__/test-velocity-baseline.test.mjs
Normal file
@@ -0,0 +1,121 @@
|
||||
import { describe, it } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
readQuarantineCount,
|
||||
renderReport,
|
||||
topSlowestFiles,
|
||||
} from "../test-velocity-baseline.mjs";
|
||||
|
||||
function makeTimings(count = 25) {
|
||||
const files = {};
|
||||
for (let index = 0; index < count; index += 1) {
|
||||
files[`packages/a/src/__tests__/case-${String(index).padStart(2, "0")}.test.ts`] = 100 + index;
|
||||
}
|
||||
files["packages/a/src/__tests__/tie-z.test.ts"] = 500;
|
||||
files["packages/a/src/__tests__/tie-a.test.ts"] = 500;
|
||||
return {
|
||||
packages: {
|
||||
"@pkg/b": {
|
||||
files: {
|
||||
"packages/b/src/__tests__/winner.test.ts": 1000,
|
||||
},
|
||||
},
|
||||
"@pkg/a": { files },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("topSlowestFiles", () => {
|
||||
it("returns exactly 20 rows in descending duration order with package attribution and stable ties", () => {
|
||||
const rows = topSlowestFiles(makeTimings(), 20);
|
||||
|
||||
assert.equal(rows.length, 20);
|
||||
assert.deepEqual(rows[0], {
|
||||
file: "packages/b/src/__tests__/winner.test.ts",
|
||||
ms: 1000,
|
||||
package: "@pkg/b",
|
||||
});
|
||||
assert.deepEqual(rows.slice(1, 3).map((row) => row.file), [
|
||||
"packages/a/src/__tests__/tie-a.test.ts",
|
||||
"packages/a/src/__tests__/tie-z.test.ts",
|
||||
]);
|
||||
assert.ok(rows.every((row, index) => index === 0 || rows[index - 1].ms >= row.ms));
|
||||
assert.equal(rows[1].package, "@pkg/a");
|
||||
});
|
||||
});
|
||||
|
||||
describe("readQuarantineCount", () => {
|
||||
it("counts entries by age bucket and flags deletion-due quarantines after 14 days", () => {
|
||||
const result = readQuarantineCount(
|
||||
{
|
||||
entries: [
|
||||
{ file: "fresh.test.ts", quarantinedAt: "2026-06-15" },
|
||||
{ file: "warning.test.ts", quarantinedAt: "2026-06-08" },
|
||||
{ file: "due.test.ts", quarantinedAt: "2026-06-01" },
|
||||
{ file: "unknown.test.ts", quarantinedAt: "not-a-date" },
|
||||
],
|
||||
},
|
||||
{ now: new Date("2026-06-17T12:00:00.000Z") },
|
||||
);
|
||||
|
||||
assert.equal(result.total, 4);
|
||||
assert.deepEqual(result.byAgeBucket, {
|
||||
"0-6d": 1,
|
||||
"7-13d": 1,
|
||||
deletionDue: 1,
|
||||
unknown: 1,
|
||||
});
|
||||
assert.deepEqual(result.deletionDueEntries, [
|
||||
{ file: "due.test.ts", quarantinedAt: "2026-06-01", ageDays: 16 },
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("renderReport", () => {
|
||||
it("includes metrics, slowest rows, quarantine count, and previous-run deltas", () => {
|
||||
const report = renderReport({
|
||||
gateMs: 12_000,
|
||||
bootSmokeMs: 2_000,
|
||||
testMs: 45_000,
|
||||
capturedAt: "2026-06-17T12:00:00.000Z",
|
||||
previous: {
|
||||
capturedAt: "2026-06-10T12:00:00.000Z",
|
||||
gateMs: 10_000,
|
||||
bootSmokeMs: 3_000,
|
||||
testMs: 50_000,
|
||||
quarantineCount: 3,
|
||||
},
|
||||
slowest: [
|
||||
{ file: "packages/a/src/__tests__/slow.test.ts", package: "@pkg/a", ms: 3210 },
|
||||
],
|
||||
quarantine: {
|
||||
total: 2,
|
||||
byAgeBucket: { "0-6d": 1, "7-13d": 1, deletionDue: 0, unknown: 0 },
|
||||
deletionDueEntries: [],
|
||||
deletionDueCount: 0,
|
||||
},
|
||||
});
|
||||
|
||||
assert.match(report, /\| Merge gate wall-time \(`pnpm test:gate`\) \| 12\.0s \| \+2\.0s \|/);
|
||||
assert.match(report, /\| Boot smoke wall-time \(`pnpm smoke:boot`\) \| 2\.0s \| -1\.0s \|/);
|
||||
assert.match(report, /\| Changed-only test wall-time \(`pnpm test`\) \| 45\.0s \| -5\.0s \|/);
|
||||
assert.match(report, /\| Quarantine \/ flake count \| 2 \| -1 \|/);
|
||||
assert.match(report, /`packages\/a\/src\/__tests__\/slow\.test\.ts` \| @pkg\/a \| 3\.2s/);
|
||||
assert.match(report, /FN-6612 weekly test velocity: gate 12\.0s \(\+2\.0s\)/);
|
||||
});
|
||||
|
||||
it("renders seed-baseline trend placeholders when there is no previous entry", () => {
|
||||
const report = renderReport({
|
||||
gateMs: 1_000,
|
||||
bootSmokeMs: null,
|
||||
testMs: 2_000,
|
||||
capturedAt: "2026-06-17T12:00:00.000Z",
|
||||
slowest: [],
|
||||
quarantine: { total: 0, byAgeBucket: {}, deletionDueEntries: [], deletionDueCount: 0 },
|
||||
});
|
||||
|
||||
assert.match(report, /\| Previous \| _\(seed baseline\)_ \| — \| — \| — \| — \|/);
|
||||
assert.match(report, /\| Delta \| — \| n\/a \| n\/a \| n\/a \| n\/a \|/);
|
||||
});
|
||||
});
|
||||
339
scripts/test-velocity-baseline.mjs
Executable file
339
scripts/test-velocity-baseline.mjs
Executable file
@@ -0,0 +1,339 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { spawn } from "node:child_process";
|
||||
import { performance } from "node:perf_hooks";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
|
||||
const currentFilePath = fileURLToPath(import.meta.url);
|
||||
const repoRoot = path.resolve(path.dirname(currentFilePath), "..");
|
||||
|
||||
export const DEFAULT_TIMINGS_PATH = "scripts/test-timings.json";
|
||||
export const DEFAULT_QUARANTINE_PATH = "scripts/lib/test-quarantine.json";
|
||||
export const DEFAULT_HISTORY_PATH = "scripts/test-velocity-history.json";
|
||||
export const DEFAULT_REPORT_PATH = "docs/test-velocity-baseline.md";
|
||||
export const DEFAULT_MEASURE_TIMEOUT_MS = 10 * 60 * 1000;
|
||||
export const DELETION_CLOCK_DAYS = 14;
|
||||
|
||||
const MEASURE_COMMANDS = [
|
||||
{ key: "gateMs", label: "Merge gate (`pnpm test:gate`)", command: "pnpm", args: ["test:gate"] },
|
||||
{ key: "bootSmokeMs", label: "Boot smoke (`pnpm smoke:boot`)", command: "pnpm", args: ["smoke:boot"] },
|
||||
{ key: "testMs", label: "Changed-only tests (`pnpm test`)", command: "pnpm", args: ["test"] },
|
||||
];
|
||||
|
||||
function readJson(relativePath, fallback = null, rootDir = repoRoot) {
|
||||
const absolutePath = path.join(rootDir, relativePath);
|
||||
if (!existsSync(absolutePath)) return fallback;
|
||||
return JSON.parse(readFileSync(absolutePath, "utf8"));
|
||||
}
|
||||
|
||||
function writeJson(relativePath, value, rootDir = repoRoot) {
|
||||
const absolutePath = path.join(rootDir, relativePath);
|
||||
mkdirSync(path.dirname(absolutePath), { recursive: true });
|
||||
writeFileSync(absolutePath, `${JSON.stringify(value, null, 2)}\n`, "utf8");
|
||||
}
|
||||
|
||||
function writeText(relativePath, value, rootDir = repoRoot) {
|
||||
const absolutePath = path.join(rootDir, relativePath);
|
||||
mkdirSync(path.dirname(absolutePath), { recursive: true });
|
||||
writeFileSync(absolutePath, value, "utf8");
|
||||
}
|
||||
|
||||
function normalizeMs(value) {
|
||||
if (value == null || value === "") return null;
|
||||
const parsed = Number(value);
|
||||
if (!Number.isFinite(parsed) || parsed < 0) {
|
||||
throw new Error(`Expected a non-negative millisecond value, got ${value}`);
|
||||
}
|
||||
return Math.round(parsed);
|
||||
}
|
||||
|
||||
function toDate(value) {
|
||||
if (!value) return null;
|
||||
const parsed = new Date(value);
|
||||
return Number.isNaN(parsed.getTime()) ? null : parsed;
|
||||
}
|
||||
|
||||
function ageDays(quarantinedAt, now) {
|
||||
const quarantinedAtDate = toDate(quarantinedAt);
|
||||
if (!quarantinedAtDate) return null;
|
||||
return Math.floor((now.getTime() - quarantinedAtDate.getTime()) / 86_400_000);
|
||||
}
|
||||
|
||||
function isoWeek(date) {
|
||||
const working = new Date(Date.UTC(date.getUTCFullYear(), date.getUTCMonth(), date.getUTCDate()));
|
||||
const day = working.getUTCDay() || 7;
|
||||
working.setUTCDate(working.getUTCDate() + 4 - day);
|
||||
const yearStart = new Date(Date.UTC(working.getUTCFullYear(), 0, 1));
|
||||
const week = Math.ceil(((working - yearStart) / 86_400_000 + 1) / 7);
|
||||
return `${working.getUTCFullYear()}-W${String(week).padStart(2, "0")}`;
|
||||
}
|
||||
|
||||
export function formatDuration(ms) {
|
||||
if (ms == null) return "unavailable";
|
||||
const rounded = Math.round(ms);
|
||||
const sign = rounded < 0 ? "-" : "";
|
||||
const absoluteMs = Math.abs(rounded);
|
||||
if (absoluteMs < 1000) return `${sign}${absoluteMs}ms`;
|
||||
const seconds = absoluteMs / 1000;
|
||||
if (seconds < 60) return `${sign}${seconds.toFixed(1)}s`;
|
||||
const minutes = Math.floor(seconds / 60);
|
||||
const remainingSeconds = Math.round(seconds - minutes * 60);
|
||||
return `${sign}${minutes}m ${String(remainingSeconds).padStart(2, "0")}s`;
|
||||
}
|
||||
|
||||
export function readQuarantineCount(json, { now = new Date() } = {}) {
|
||||
const entries = Array.isArray(json?.entries) ? json.entries : [];
|
||||
const buckets = {
|
||||
"0-6d": 0,
|
||||
"7-13d": 0,
|
||||
deletionDue: 0,
|
||||
unknown: 0,
|
||||
};
|
||||
const deletionDueEntries = [];
|
||||
|
||||
for (const entry of entries) {
|
||||
const age = ageDays(entry?.quarantinedAt, now);
|
||||
const normalized = {
|
||||
file: entry?.file ?? "unknown",
|
||||
quarantinedAt: entry?.quarantinedAt ?? null,
|
||||
ageDays: age,
|
||||
};
|
||||
if (age == null || age < 0) {
|
||||
buckets.unknown += 1;
|
||||
} else if (age >= DELETION_CLOCK_DAYS) {
|
||||
buckets.deletionDue += 1;
|
||||
deletionDueEntries.push(normalized);
|
||||
} else if (age >= 7) {
|
||||
buckets["7-13d"] += 1;
|
||||
} else {
|
||||
buckets["0-6d"] += 1;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
total: entries.length,
|
||||
byAgeBucket: buckets,
|
||||
deletionDueEntries,
|
||||
deletionDueCount: deletionDueEntries.length,
|
||||
};
|
||||
}
|
||||
|
||||
export function topSlowestFiles(timingsJson, n = 20) {
|
||||
const rows = [];
|
||||
for (const [packageName, packageTiming] of Object.entries(timingsJson?.packages ?? {})) {
|
||||
for (const [file, ms] of Object.entries(packageTiming?.files ?? {})) {
|
||||
const parsed = Number(ms);
|
||||
rows.push({ file, ms: Number.isFinite(parsed) ? parsed : 0, package: packageName });
|
||||
}
|
||||
}
|
||||
|
||||
rows.sort((a, b) => b.ms - a.ms || a.package.localeCompare(b.package) || a.file.localeCompare(b.file));
|
||||
return rows.slice(0, n);
|
||||
}
|
||||
|
||||
function delta(latest, previous, field) {
|
||||
if (!previous || latest?.[field] == null || previous?.[field] == null) return "n/a";
|
||||
const diff = latest[field] - previous[field];
|
||||
const sign = diff > 0 ? "+" : "";
|
||||
return `${sign}${formatDuration(diff)}`;
|
||||
}
|
||||
|
||||
function renderMetricRow(name, latest, previous, field) {
|
||||
return `| ${name} | ${formatDuration(latest?.[field])} | ${delta(latest, previous, field)} |`;
|
||||
}
|
||||
|
||||
function trendCell(current, prior) {
|
||||
if (current == null || prior == null) return "n/a";
|
||||
const diff = current - prior;
|
||||
return `${diff > 0 ? "+" : ""}${diff}`;
|
||||
}
|
||||
|
||||
export function renderReport({ gateMs, bootSmokeMs, testMs, slowest = [], quarantine, capturedAt, previous = null, measurementFailures = [], timingSnapshotCapturedAt = null } = {}) {
|
||||
const latest = {
|
||||
gateMs: normalizeMs(gateMs),
|
||||
bootSmokeMs: normalizeMs(bootSmokeMs),
|
||||
testMs: normalizeMs(testMs),
|
||||
quarantineCount: quarantine?.total ?? quarantine?.quarantineCount ?? 0,
|
||||
capturedAt: capturedAt ?? new Date().toISOString(),
|
||||
};
|
||||
const cycle = isoWeek(new Date(latest.capturedAt));
|
||||
const slowRows = slowest
|
||||
.map((row, index) => `| ${index + 1} | \`${row.file}\` | ${row.package} | ${formatDuration(row.ms)} |`)
|
||||
.join("\n");
|
||||
const dueRows = (quarantine?.deletionDueEntries ?? [])
|
||||
.map((entry) => `| \`${entry.file}\` | ${entry.quarantinedAt ?? "unknown"} | ${entry.ageDays ?? "unknown"} |`)
|
||||
.join("\n");
|
||||
const failures = measurementFailures.length > 0
|
||||
? measurementFailures.map((failure) => `- ${failure.label}: ${failure.status}`).join("\n")
|
||||
: "- None recorded.";
|
||||
const previousRows = previous
|
||||
? `| Previous | ${previous.capturedAt ?? "unknown"} | ${formatDuration(previous.gateMs)} | ${formatDuration(previous.bootSmokeMs)} | ${formatDuration(previous.testMs)} | ${previous.quarantineCount ?? "n/a"} |\n| Latest | ${latest.capturedAt} | ${formatDuration(latest.gateMs)} | ${formatDuration(latest.bootSmokeMs)} | ${formatDuration(latest.testMs)} | ${latest.quarantineCount} |\n| Delta | — | ${delta(latest, previous, "gateMs")} | ${delta(latest, previous, "bootSmokeMs")} | ${delta(latest, previous, "testMs")} | ${trendCell(latest.quarantineCount, previous.quarantineCount)} |`
|
||||
: `| Previous | _(seed baseline)_ | — | — | — | — |\n| Latest | ${latest.capturedAt} | ${formatDuration(latest.gateMs)} | ${formatDuration(latest.bootSmokeMs)} | ${formatDuration(latest.testMs)} | ${latest.quarantineCount} |\n| Delta | — | n/a | n/a | n/a | n/a |`;
|
||||
|
||||
return `# Test velocity baseline\n\n> Weekly FN-6612 signal-per-second baseline. Measure and report feedback-loop velocity; do **not** add slow tests or wire this report into blocking PR checks. The merge gate remains the existing thin Lint, Typecheck, Build, and Gate path.\n\n## Latest baseline\n\n- Cycle: **${cycle}**\n- Captured at: **${latest.capturedAt}**\n- Timing snapshot: \`${DEFAULT_TIMINGS_PATH}\`${timingSnapshotCapturedAt ? ` captured at **${timingSnapshotCapturedAt}**` : ""}\n- Quarantine ledger: \`${DEFAULT_QUARANTINE_PATH}\`\n\n## Metrics\n\n| Metric | Current | Delta vs previous |\n|---|---:|---:|\n${renderMetricRow("Merge gate wall-time (`pnpm test:gate`)", latest, previous, "gateMs")}\n${renderMetricRow("Boot smoke wall-time (`pnpm smoke:boot`)", latest, previous, "bootSmokeMs")}\n${renderMetricRow("Changed-only test wall-time (`pnpm test`)", latest, previous, "testMs")}\n| Quarantine / flake count | ${latest.quarantineCount} | ${trendCell(latest.quarantineCount, previous?.quarantineCount)} |\n| Deletion-due quarantines | ${quarantine?.deletionDueCount ?? 0} | n/a |\n\n## Measurement failures\n\n${failures}\n\n## Slowest 20 test files\n\n| Rank | File | Package | Duration |\n|---:|---|---|---:|\n${slowRows || "| — | — | — | — |"}\n\n## Quarantine age buckets\n\n| Age bucket | Count |\n|---|---:|\n| 0-6 days | ${quarantine?.byAgeBucket?.["0-6d"] ?? 0} |\n| 7-13 days | ${quarantine?.byAgeBucket?.["7-13d"] ?? 0} |\n| deletion due (>=14 days) | ${quarantine?.byAgeBucket?.deletionDue ?? 0} |\n| unknown/future | ${quarantine?.byAgeBucket?.unknown ?? 0} |\n\n### Deletion-due entries\n\n| File | Quarantined at | Age (days) |\n|---|---:|---:|\n${dueRows || "| — | — | — |"}\n\n## Before / after trend\n\n| Row | Captured at | Gate | Boot smoke | \`pnpm test\` | Quarantine count |\n|---|---|---:|---:|---:|---:|\n${previousRows}\n\n_Future weekly rows append to \`${DEFAULT_HISTORY_PATH}\`; compare the latest row against the previous row before posting to #leads._\n\n## Post to #leads\n\n\`\`\`text\nFN-6612 weekly test velocity: gate ${formatDuration(latest.gateMs)} (${delta(latest, previous, "gateMs")}), boot smoke ${formatDuration(latest.bootSmokeMs)} (${delta(latest, previous, "bootSmokeMs")}), pnpm test ${formatDuration(latest.testMs)} (${delta(latest, previous, "testMs")}), quarantine ledger ${latest.quarantineCount} (${trendCell(latest.quarantineCount, previous?.quarantineCount)}). Slowest file: ${slowest[0]?.file ?? "none"} at ${formatDuration(slowest[0]?.ms)}. Deletion-due quarantines: ${quarantine?.deletionDueCount ?? 0}.\n\`\`\`\n\n## How to refresh\n\n\`\`\`bash\npnpm test:velocity -- --measure --write-report\n\`\`\`\n\nReport-only regeneration is cheap and does not run any suite:\n\n\`\`\`bash\npnpm test:velocity\n\`\`\`\n`;
|
||||
}
|
||||
|
||||
function historyEntries(history) {
|
||||
if (Array.isArray(history)) return history;
|
||||
if (Array.isArray(history?.entries)) return history.entries;
|
||||
return [];
|
||||
}
|
||||
|
||||
function createEntry({ capturedAt = new Date().toISOString(), gateMs = null, bootSmokeMs = null, testMs = null, quarantine, slowest, measurementFailures = [], timingSnapshotCapturedAt = null }) {
|
||||
return {
|
||||
capturedAt,
|
||||
gateMs: normalizeMs(gateMs),
|
||||
bootSmokeMs: normalizeMs(bootSmokeMs),
|
||||
testMs: normalizeMs(testMs),
|
||||
quarantineCount: quarantine?.total ?? 0,
|
||||
slowestTop20: slowest,
|
||||
measurementFailures,
|
||||
timingSnapshotCapturedAt,
|
||||
};
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = { measure: false, writeReport: false, reportOnly: true, timeoutMs: DEFAULT_MEASURE_TIMEOUT_MS, help: false };
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index];
|
||||
if (arg === "--") continue;
|
||||
else if (arg === "--measure") args.measure = true;
|
||||
else if (arg === "--write-report") args.writeReport = true;
|
||||
else if (arg === "--report-only") args.reportOnly = true;
|
||||
else if (arg === "--timeout-ms") args.timeoutMs = Number(argv[++index]);
|
||||
else if (arg === "--help" || arg === "-h") args.help = true;
|
||||
else throw new Error(`Unknown argument: ${arg}`);
|
||||
}
|
||||
if (!Number.isFinite(args.timeoutMs) || args.timeoutMs <= 0) {
|
||||
throw new Error(`Expected --timeout-ms to be a positive number, got ${args.timeoutMs}`);
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
async function timeCommand({ command, args, label, timeoutMs, cwd, stdout, stderr }) {
|
||||
const started = performance.now();
|
||||
stderr.write(`[test-velocity] measuring ${label}\n`);
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn(command, args, { cwd, shell: false, stdio: ["ignore", "pipe", "pipe"] });
|
||||
let settled = false;
|
||||
const timer = setTimeout(() => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
child.kill("SIGTERM");
|
||||
const elapsedMs = Math.round(performance.now() - started);
|
||||
resolve({ ms: null, failure: { label, status: `timeout after ${formatDuration(timeoutMs)} (${elapsedMs}ms elapsed)` } });
|
||||
}, timeoutMs);
|
||||
|
||||
child.stdout.on("data", (chunk) => stdout.write(chunk));
|
||||
child.stderr.on("data", (chunk) => stderr.write(chunk));
|
||||
child.on("error", (error) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
resolve({ ms: null, failure: { label, status: `spawn error: ${error.message}` } });
|
||||
});
|
||||
child.on("close", (code, signal) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
const elapsedMs = Math.round(performance.now() - started);
|
||||
if (code === 0 && !signal) {
|
||||
resolve({ ms: elapsedMs, failure: null });
|
||||
} else {
|
||||
resolve({ ms: null, failure: { label, status: signal ? `signal ${signal} after ${formatDuration(elapsedMs)}` : `exit ${code} after ${formatDuration(elapsedMs)}` } });
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function measureCommands({ timeoutMs, cwd, stdout, stderr }) {
|
||||
const results = {};
|
||||
const failures = [];
|
||||
for (const measurement of MEASURE_COMMANDS) {
|
||||
const result = await timeCommand({ ...measurement, timeoutMs, cwd, stdout, stderr });
|
||||
results[measurement.key] = result.ms;
|
||||
if (result.failure) failures.push(result.failure);
|
||||
}
|
||||
return { ...results, measurementFailures: failures };
|
||||
}
|
||||
|
||||
function renderFromEntry(entry, previous, quarantine) {
|
||||
return renderReport({
|
||||
gateMs: entry?.gateMs,
|
||||
bootSmokeMs: entry?.bootSmokeMs,
|
||||
testMs: entry?.testMs,
|
||||
slowest: entry?.slowestTop20 ?? [],
|
||||
quarantine: quarantine ?? { total: entry?.quarantineCount ?? 0, byAgeBucket: {}, deletionDueEntries: [], deletionDueCount: 0 },
|
||||
capturedAt: entry?.capturedAt,
|
||||
previous,
|
||||
measurementFailures: entry?.measurementFailures ?? [],
|
||||
timingSnapshotCapturedAt: entry?.timingSnapshotCapturedAt ?? null,
|
||||
});
|
||||
}
|
||||
|
||||
export async function main(argv = process.argv.slice(2), { rootDir = repoRoot, stdout = process.stdout, stderr = process.stderr, now = new Date() } = {}) {
|
||||
let args;
|
||||
try {
|
||||
args = parseArgs(argv);
|
||||
} catch (error) {
|
||||
stderr.write(`${error.message}\n`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (args.help) {
|
||||
stdout.write("Usage: node scripts/test-velocity-baseline.mjs [--measure] [--write-report] [--report-only] [--timeout-ms <ms>]\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
const history = readJson(DEFAULT_HISTORY_PATH, { entries: [] }, rootDir);
|
||||
const entries = historyEntries(history);
|
||||
const timings = readJson(DEFAULT_TIMINGS_PATH, { packages: {} }, rootDir);
|
||||
const quarantineJson = readJson(DEFAULT_QUARANTINE_PATH, { entries: [] }, rootDir);
|
||||
const quarantine = readQuarantineCount(quarantineJson, { now });
|
||||
const slowest = topSlowestFiles(timings, 20);
|
||||
|
||||
if (args.measure) {
|
||||
const measured = await measureCommands({ timeoutMs: args.timeoutMs, cwd: rootDir, stdout, stderr });
|
||||
const entry = createEntry({
|
||||
capturedAt: now.toISOString(),
|
||||
gateMs: measured.gateMs,
|
||||
bootSmokeMs: measured.bootSmokeMs,
|
||||
testMs: measured.testMs,
|
||||
quarantine,
|
||||
slowest,
|
||||
measurementFailures: measured.measurementFailures,
|
||||
timingSnapshotCapturedAt: timings?.capturedAt ?? null,
|
||||
});
|
||||
entries.push(entry);
|
||||
writeJson(DEFAULT_HISTORY_PATH, { entries }, rootDir);
|
||||
}
|
||||
|
||||
const latest = entries.at(-1) ?? createEntry({
|
||||
capturedAt: now.toISOString(),
|
||||
quarantine,
|
||||
slowest,
|
||||
timingSnapshotCapturedAt: timings?.capturedAt ?? null,
|
||||
});
|
||||
const previous = entries.length > 1 ? entries.at(-2) : null;
|
||||
const report = renderFromEntry(latest, previous, quarantine);
|
||||
|
||||
if (args.writeReport || args.reportOnly) {
|
||||
writeText(DEFAULT_REPORT_PATH, report, rootDir);
|
||||
stdout.write(`Updated ${DEFAULT_REPORT_PATH}\n`);
|
||||
} else {
|
||||
stdout.write(report);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
/*
|
||||
FNXC:TestVelocityBaseline 2026-06-17-00:00:
|
||||
FN-6612 requires a weekly signal-per-second baseline for gate time, boot smoke time, changed-only test time, slowest files, and quarantine age. This script measures and reports those values only; it must stay out of blocking PR checks so the merge gate remains thin.
|
||||
*/
|
||||
const exitCode = await main();
|
||||
process.exitCode = exitCode;
|
||||
}
|
||||
231
scripts/test-velocity-history.json
Normal file
231
scripts/test-velocity-history.json
Normal file
@@ -0,0 +1,231 @@
|
||||
{
|
||||
"entries": [
|
||||
{
|
||||
"capturedAt": "2026-06-18T02:53:34.158Z",
|
||||
"gateMs": 8478,
|
||||
"bootSmokeMs": null,
|
||||
"testMs": 38405,
|
||||
"quarantineCount": 2,
|
||||
"slowestTop20": [
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts",
|
||||
"ms": 13900,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/agent-store.test.ts",
|
||||
"ms": 11600,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/dashboard/src/__tests__/routes-agents.test.ts",
|
||||
"ms": 11200,
|
||||
"package": "@fusion/dashboard"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/mission-store.test.ts",
|
||||
"ms": 10700,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/db.test.ts",
|
||||
"ms": 10100,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/dashboard/src/__tests__/routes-git.test.ts",
|
||||
"ms": 9400,
|
||||
"package": "@fusion/dashboard"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts",
|
||||
"ms": 9000,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/merger-ai.test.ts",
|
||||
"ms": 8700,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts",
|
||||
"ms": 8400,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts",
|
||||
"ms": 8400,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/task-documents.test.ts",
|
||||
"ms": 8300,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts",
|
||||
"ms": 7800,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/cli/src/__tests__/extension.test.ts",
|
||||
"ms": 7000,
|
||||
"package": "@runfusion/fusion"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/run-audit.test.ts",
|
||||
"ms": 6900,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts",
|
||||
"ms": 6100,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/dashboard/src/__tests__/routes-planning.test.ts",
|
||||
"ms": 5600,
|
||||
"package": "@fusion/dashboard"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/store-merge-queue.test.ts",
|
||||
"ms": 5200,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/dashboard/app/components/__tests__/FileEditor.test.tsx",
|
||||
"ms": 5100,
|
||||
"package": "@fusion/dashboard"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts",
|
||||
"ms": 4900,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts",
|
||||
"ms": 4900,
|
||||
"package": "@fusion/engine"
|
||||
}
|
||||
],
|
||||
"measurementFailures": [
|
||||
{
|
||||
"label": "Boot smoke (`pnpm smoke:boot`)",
|
||||
"status": "exit 1 after 411ms"
|
||||
}
|
||||
],
|
||||
"timingSnapshotCapturedAt": "2026-06-03T23:45:49.672Z"
|
||||
},
|
||||
{
|
||||
"capturedAt": "2026-06-18T03:04:28.794Z",
|
||||
"gateMs": 6177,
|
||||
"bootSmokeMs": 18220,
|
||||
"testMs": 7740,
|
||||
"quarantineCount": 2,
|
||||
"slowestTop20": [
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts",
|
||||
"ms": 13900,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/agent-store.test.ts",
|
||||
"ms": 11600,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/dashboard/src/__tests__/routes-agents.test.ts",
|
||||
"ms": 11200,
|
||||
"package": "@fusion/dashboard"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/mission-store.test.ts",
|
||||
"ms": 10700,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/db.test.ts",
|
||||
"ms": 10100,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/dashboard/src/__tests__/routes-git.test.ts",
|
||||
"ms": 9400,
|
||||
"package": "@fusion/dashboard"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts",
|
||||
"ms": 9000,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/merger-ai.test.ts",
|
||||
"ms": 8700,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts",
|
||||
"ms": 8400,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts",
|
||||
"ms": 8400,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/task-documents.test.ts",
|
||||
"ms": 8300,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts",
|
||||
"ms": 7800,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/cli/src/__tests__/extension.test.ts",
|
||||
"ms": 7000,
|
||||
"package": "@runfusion/fusion"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/run-audit.test.ts",
|
||||
"ms": 6900,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts",
|
||||
"ms": 6100,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/dashboard/src/__tests__/routes-planning.test.ts",
|
||||
"ms": 5600,
|
||||
"package": "@fusion/dashboard"
|
||||
},
|
||||
{
|
||||
"file": "packages/core/src/__tests__/store-merge-queue.test.ts",
|
||||
"ms": 5200,
|
||||
"package": "@fusion/core"
|
||||
},
|
||||
{
|
||||
"file": "packages/dashboard/app/components/__tests__/FileEditor.test.tsx",
|
||||
"ms": 5100,
|
||||
"package": "@fusion/dashboard"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts",
|
||||
"ms": 4900,
|
||||
"package": "@fusion/engine"
|
||||
},
|
||||
{
|
||||
"file": "packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts",
|
||||
"ms": 4900,
|
||||
"package": "@fusion/engine"
|
||||
}
|
||||
],
|
||||
"measurementFailures": [],
|
||||
"timingSnapshotCapturedAt": "2026-06-03T23:45:49.672Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user