FN-6617: add weekly test velocity baseline

Adds a report-only weekly baseline for test feedback-loop velocity.

- Add a test:velocity script that measures gate, boot smoke, and changed-only test durations and regenerates a #leads-ready report.
- Track committed baseline history, slowest-file summaries, and quarantine deletion-clock counts.
- Document the weekly refresh workflow in testing guidance and agent instructions.

Files changed:
 AGENTS.md                                         |   4 +-
 docs/test-velocity-baseline.md                    |  92 ++++++
 docs/testing.md                                   |  18 ++
 package.json                                      |   1 +
 scripts/__tests__/test-velocity-baseline.test.mjs | 121 ++++++++
 scripts/test-velocity-baseline.mjs                | 339 ++++++++++++++++++++++
 scripts/test-velocity-history.json                | 231 +++++++++++++++
 7 files changed, 805 insertions(+), 1 deletion(-)

Fusion-Task-Id: FN-6617

Fusion-Task-Lineage: 50a07298-74ec-4481-b288-4fadb7d3d29b
This commit is contained in:
gsxdsm
2026-06-17 20:09:05 -07:00
parent a013bc0309
commit fbd9ad2f3b
7 changed files with 805 additions and 1 deletions

View File

@@ -0,0 +1,92 @@
# Test velocity baseline
> Weekly FN-6612 signal-per-second baseline. Measure and report feedback-loop velocity; do **not** add slow tests or wire this report into blocking PR checks. The merge gate remains the existing thin Lint, Typecheck, Build, and Gate path.
## Latest baseline
- Cycle: **2026-W25**
- Captured at: **2026-06-18T03:04:28.794Z**
- Timing snapshot: `scripts/test-timings.json` captured at **2026-06-03T23:45:49.672Z**
- Quarantine ledger: `scripts/lib/test-quarantine.json`
## Metrics
| Metric | Current | Delta vs previous |
|---|---:|---:|
| Merge gate wall-time (`pnpm test:gate`) | 6.2s | -2.3s |
| Boot smoke wall-time (`pnpm smoke:boot`) | 18.2s | n/a |
| Changed-only test wall-time (`pnpm test`) | 7.7s | -30.7s |
| Quarantine / flake count | 2 | 0 |
| Deletion-due quarantines | 0 | n/a |
## Measurement failures
- None recorded.
## Slowest 20 test files
| Rank | File | Package | Duration |
|---:|---|---|---:|
| 1 | `packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts` | @fusion/engine | 13.9s |
| 2 | `packages/core/src/__tests__/agent-store.test.ts` | @fusion/core | 11.6s |
| 3 | `packages/dashboard/src/__tests__/routes-agents.test.ts` | @fusion/dashboard | 11.2s |
| 4 | `packages/core/src/__tests__/mission-store.test.ts` | @fusion/core | 10.7s |
| 5 | `packages/core/src/__tests__/db.test.ts` | @fusion/core | 10.1s |
| 6 | `packages/dashboard/src/__tests__/routes-git.test.ts` | @fusion/dashboard | 9.4s |
| 7 | `packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts` | @fusion/engine | 9.0s |
| 8 | `packages/engine/src/__tests__/merger-ai.test.ts` | @fusion/engine | 8.7s |
| 9 | `packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts` | @fusion/engine | 8.4s |
| 10 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts` | @fusion/engine | 8.4s |
| 11 | `packages/core/src/__tests__/task-documents.test.ts` | @fusion/core | 8.3s |
| 12 | `packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts` | @fusion/engine | 7.8s |
| 13 | `packages/cli/src/__tests__/extension.test.ts` | @runfusion/fusion | 7.0s |
| 14 | `packages/core/src/__tests__/run-audit.test.ts` | @fusion/core | 6.9s |
| 15 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts` | @fusion/engine | 6.1s |
| 16 | `packages/dashboard/src/__tests__/routes-planning.test.ts` | @fusion/dashboard | 5.6s |
| 17 | `packages/core/src/__tests__/store-merge-queue.test.ts` | @fusion/core | 5.2s |
| 18 | `packages/dashboard/app/components/__tests__/FileEditor.test.tsx` | @fusion/dashboard | 5.1s |
| 19 | `packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts` | @fusion/engine | 4.9s |
| 20 | `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` | @fusion/engine | 4.9s |
## Quarantine age buckets
| Age bucket | Count |
|---|---:|
| 0-6 days | 2 |
| 7-13 days | 0 |
| deletion due (>=14 days) | 0 |
| unknown/future | 0 |
### Deletion-due entries
| File | Quarantined at | Age (days) |
|---|---:|---:|
| — | — | — |
## Before / after trend
| Row | Captured at | Gate | Boot smoke | `pnpm test` | Quarantine count |
|---|---|---:|---:|---:|---:|
| Previous | 2026-06-18T02:53:34.158Z | 8.5s | unavailable | 38.4s | 2 |
| Latest | 2026-06-18T03:04:28.794Z | 6.2s | 18.2s | 7.7s | 2 |
| Delta | — | -2.3s | n/a | -30.7s | 0 |
_Future weekly rows append to `scripts/test-velocity-history.json`; compare the latest row against the previous row before posting to #leads._
## Post to #leads
```text
FN-6612 weekly test velocity: gate 6.2s (-2.3s), boot smoke 18.2s (n/a), pnpm test 7.7s (-30.7s), quarantine ledger 2 (0). Slowest file: packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts at 13.9s. Deletion-due quarantines: 0.
```
## How to refresh
```bash
pnpm test:velocity -- --measure --write-report
```
Report-only regeneration is cheap and does not run any suite:
```bash
pnpm test:velocity
```

View File

@@ -225,6 +225,24 @@ shard artifacts into `.timings/` first (the default lookup directory), or pass
scheduled job can gate on freshness via `node scripts/ci-test-shard.mjs --check-timings-staleness`,
which exits non-zero when the snapshot is missing or older than the 30-day budget.
## Weekly test velocity baseline
FN-6612 tracks feedback-loop velocity as signal-per-second, not as a new blocking gate. Refresh the weekly baseline from a clean worktree with:
```bash
pnpm test:velocity -- --measure --write-report
```
The script runs `pnpm test:gate`, `pnpm smoke:boot`, and `pnpm test` with bounded async process supervision, then appends the measured row to `scripts/test-velocity-history.json` and rewrites the postable artifact at `docs/test-velocity-baseline.md`. It reads the slowest 20 files from the committed `scripts/test-timings.json` snapshot and the flake/quarantine count plus 14-day deletion-clock buckets directly from `scripts/lib/test-quarantine.json`; do not run the full suite just to populate the slowest-file table.
Use cheap report-only regeneration when measurements already exist:
```bash
pnpm test:velocity
```
Each week, copy the `Post to #leads` block from `docs/test-velocity-baseline.md`. If a measured command fails because the local environment is not ready, keep the failure recorded in the report instead of fabricating a time, then fix or rerun separately as appropriate. Do not wire `pnpm test:velocity`, `test:full`, or any slow-suite expansion into PR checks; the merge gate stays the thin Lint, Typecheck, Build, and Gate path.
## Targeted commands
```bash