FN-6617: add weekly test velocity baseline
Adds a report-only weekly baseline for test feedback-loop velocity. - Add a test:velocity script that measures gate, boot smoke, and changed-only test durations and regenerates a #leads-ready report. - Track committed baseline history, slowest-file summaries, and quarantine deletion-clock counts. - Document the weekly refresh workflow in testing guidance and agent instructions. Files changed: AGENTS.md | 4 +- docs/test-velocity-baseline.md | 92 ++++++ docs/testing.md | 18 ++ package.json | 1 + scripts/__tests__/test-velocity-baseline.test.mjs | 121 ++++++++ scripts/test-velocity-baseline.mjs | 339 ++++++++++++++++++++++ scripts/test-velocity-history.json | 231 +++++++++++++++ 7 files changed, 805 insertions(+), 1 deletion(-) Fusion-Task-Id: FN-6617 Fusion-Task-Lineage: 50a07298-74ec-4481-b288-4fadb7d3d29b
This commit is contained in:
92
docs/test-velocity-baseline.md
Normal file
92
docs/test-velocity-baseline.md
Normal file
@@ -0,0 +1,92 @@
|
||||
# Test velocity baseline
|
||||
|
||||
> Weekly FN-6612 signal-per-second baseline. Measure and report feedback-loop velocity; do **not** add slow tests or wire this report into blocking PR checks. The merge gate remains the existing thin Lint, Typecheck, Build, and Gate path.
|
||||
|
||||
## Latest baseline
|
||||
|
||||
- Cycle: **2026-W25**
|
||||
- Captured at: **2026-06-18T03:04:28.794Z**
|
||||
- Timing snapshot: `scripts/test-timings.json` captured at **2026-06-03T23:45:49.672Z**
|
||||
- Quarantine ledger: `scripts/lib/test-quarantine.json`
|
||||
|
||||
## Metrics
|
||||
|
||||
| Metric | Current | Delta vs previous |
|
||||
|---|---:|---:|
|
||||
| Merge gate wall-time (`pnpm test:gate`) | 6.2s | -2.3s |
|
||||
| Boot smoke wall-time (`pnpm smoke:boot`) | 18.2s | n/a |
|
||||
| Changed-only test wall-time (`pnpm test`) | 7.7s | -30.7s |
|
||||
| Quarantine / flake count | 2 | 0 |
|
||||
| Deletion-due quarantines | 0 | n/a |
|
||||
|
||||
## Measurement failures
|
||||
|
||||
- None recorded.
|
||||
|
||||
## Slowest 20 test files
|
||||
|
||||
| Rank | File | Package | Duration |
|
||||
|---:|---|---|---:|
|
||||
| 1 | `packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts` | @fusion/engine | 13.9s |
|
||||
| 2 | `packages/core/src/__tests__/agent-store.test.ts` | @fusion/core | 11.6s |
|
||||
| 3 | `packages/dashboard/src/__tests__/routes-agents.test.ts` | @fusion/dashboard | 11.2s |
|
||||
| 4 | `packages/core/src/__tests__/mission-store.test.ts` | @fusion/core | 10.7s |
|
||||
| 5 | `packages/core/src/__tests__/db.test.ts` | @fusion/core | 10.1s |
|
||||
| 6 | `packages/dashboard/src/__tests__/routes-git.test.ts` | @fusion/dashboard | 9.4s |
|
||||
| 7 | `packages/engine/src/__tests__/reliability-interactions/branch-group-automerge-precedence.test.ts` | @fusion/engine | 9.0s |
|
||||
| 8 | `packages/engine/src/__tests__/merger-ai.test.ts` | @fusion/engine | 8.7s |
|
||||
| 9 | `packages/engine/src/__tests__/reliability-interactions/branch-group-merge-routing.test.ts` | @fusion/engine | 8.4s |
|
||||
| 10 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion-gate.test.ts` | @fusion/engine | 8.4s |
|
||||
| 11 | `packages/core/src/__tests__/task-documents.test.ts` | @fusion/core | 8.3s |
|
||||
| 12 | `packages/engine/src/runtimes/__tests__/in-process-runtime.test.ts` | @fusion/engine | 7.8s |
|
||||
| 13 | `packages/cli/src/__tests__/extension.test.ts` | @runfusion/fusion | 7.0s |
|
||||
| 14 | `packages/core/src/__tests__/run-audit.test.ts` | @fusion/core | 6.9s |
|
||||
| 15 | `packages/engine/src/__tests__/reliability-interactions/branch-group-promotion.test.ts` | @fusion/engine | 6.1s |
|
||||
| 16 | `packages/dashboard/src/__tests__/routes-planning.test.ts` | @fusion/dashboard | 5.6s |
|
||||
| 17 | `packages/core/src/__tests__/store-merge-queue.test.ts` | @fusion/core | 5.2s |
|
||||
| 18 | `packages/dashboard/app/components/__tests__/FileEditor.test.tsx` | @fusion/dashboard | 5.1s |
|
||||
| 19 | `packages/engine/src/__tests__/reliability-interactions/integration-worktree-state.test.ts` | @fusion/engine | 4.9s |
|
||||
| 20 | `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` | @fusion/engine | 4.9s |
|
||||
|
||||
## Quarantine age buckets
|
||||
|
||||
| Age bucket | Count |
|
||||
|---|---:|
|
||||
| 0-6 days | 2 |
|
||||
| 7-13 days | 0 |
|
||||
| deletion due (>=14 days) | 0 |
|
||||
| unknown/future | 0 |
|
||||
|
||||
### Deletion-due entries
|
||||
|
||||
| File | Quarantined at | Age (days) |
|
||||
|---|---:|---:|
|
||||
| — | — | — |
|
||||
|
||||
## Before / after trend
|
||||
|
||||
| Row | Captured at | Gate | Boot smoke | `pnpm test` | Quarantine count |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| Previous | 2026-06-18T02:53:34.158Z | 8.5s | unavailable | 38.4s | 2 |
|
||||
| Latest | 2026-06-18T03:04:28.794Z | 6.2s | 18.2s | 7.7s | 2 |
|
||||
| Delta | — | -2.3s | n/a | -30.7s | 0 |
|
||||
|
||||
_Future weekly rows append to `scripts/test-velocity-history.json`; compare the latest row against the previous row before posting to #leads._
|
||||
|
||||
## Post to #leads
|
||||
|
||||
```text
|
||||
FN-6612 weekly test velocity: gate 6.2s (-2.3s), boot smoke 18.2s (n/a), pnpm test 7.7s (-30.7s), quarantine ledger 2 (0). Slowest file: packages/engine/src/__tests__/reliability-interactions/shared-branch-group-lifecycle.test.ts at 13.9s. Deletion-due quarantines: 0.
|
||||
```
|
||||
|
||||
## How to refresh
|
||||
|
||||
```bash
|
||||
pnpm test:velocity -- --measure --write-report
|
||||
```
|
||||
|
||||
Report-only regeneration is cheap and does not run any suite:
|
||||
|
||||
```bash
|
||||
pnpm test:velocity
|
||||
```
|
||||
@@ -225,6 +225,24 @@ shard artifacts into `.timings/` first (the default lookup directory), or pass
|
||||
scheduled job can gate on freshness via `node scripts/ci-test-shard.mjs --check-timings-staleness`,
|
||||
which exits non-zero when the snapshot is missing or older than the 30-day budget.
|
||||
|
||||
## Weekly test velocity baseline
|
||||
|
||||
FN-6612 tracks feedback-loop velocity as signal-per-second, not as a new blocking gate. Refresh the weekly baseline from a clean worktree with:
|
||||
|
||||
```bash
|
||||
pnpm test:velocity -- --measure --write-report
|
||||
```
|
||||
|
||||
The script runs `pnpm test:gate`, `pnpm smoke:boot`, and `pnpm test` with bounded async process supervision, then appends the measured row to `scripts/test-velocity-history.json` and rewrites the postable artifact at `docs/test-velocity-baseline.md`. It reads the slowest 20 files from the committed `scripts/test-timings.json` snapshot and the flake/quarantine count plus 14-day deletion-clock buckets directly from `scripts/lib/test-quarantine.json`; do not run the full suite just to populate the slowest-file table.
|
||||
|
||||
Use cheap report-only regeneration when measurements already exist:
|
||||
|
||||
```bash
|
||||
pnpm test:velocity
|
||||
```
|
||||
|
||||
Each week, copy the `Post to #leads` block from `docs/test-velocity-baseline.md`. If a measured command fails because the local environment is not ready, keep the failure recorded in the report instead of fabricating a time, then fix or rerun separately as appropriate. Do not wire `pnpm test:velocity`, `test:full`, or any slow-suite expansion into PR checks; the merge gate stays the thin Lint, Typecheck, Build, and Gate path.
|
||||
|
||||
## Targeted commands
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user