chore: add test value audit (deletion-candidate evidence base)
Add scripts/test-value-audit.mjs + scripts/lib/test-value-audit-lib.mjs: a heuristic that scores every test file under packages/*/src/**/__tests__/** and packages/dashboard/app/**/__tests__/** from git history, classifying commits as positive (fix+source co-change, Symptom Verification regressions, added-with-source) vs negative (flake/quarantine/timeout churn, test-only modifies, quarantine-ledger membership). Joins per-file durations from scripts/test-timings.json so SLOW + LOW-VALUE files rank first as deletion candidates. Emits docs/test-value-audit.json + docs/test-value-audit.md (top 40 + methodology + caveats). Pure scoring logic is unit-tested with synthetic commit records. The script never deletes tests — evidence only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gsxdsm
129726
docs/test-value-audit.json
Normal file
129726
docs/test-value-audit.json
Normal file
File diff suppressed because it is too large
Load Diff
144
docs/test-value-audit.md
Normal file
144
docs/test-value-audit.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# Test Value Audit
|
||||
|
||||
> Generated by `scripts/test-value-audit.mjs` on 2026-06-26T00:03:00.110Z.
|
||||
>
|
||||
> **This is a HEURISTIC, not ground truth.** It is an evidence base for a human
|
||||
> deletion decision. The script does **not** delete any tests. See Methodology + Caveats.
|
||||
|
||||
## Summary
|
||||
|
||||
- Files analyzed: **2051**
|
||||
- With timing data: **1979**
|
||||
- `delete` candidates (valueScore ≤ 0 or quarantined): **288**
|
||||
- `review` candidates (0 < valueScore ≤ 3): **839**
|
||||
- `keep` (valueScore > 3): **924**
|
||||
- "Safe delete" under the deletion-ratchet (zero positive evidence + churn/quarantine): **54**
|
||||
- In quarantine ledger (current or historical): **59**
|
||||
|
||||
## Methodology
|
||||
|
||||
For every audited test file we run a single whole-history `git log --name-status`
|
||||
pass and classify each commit that touched the file (renames followed backward):
|
||||
|
||||
**Positive signal** (encoded/caught a real bug):
|
||||
- `fix(...)`/`fix:` subject **with** a sibling source change in the same package (+3)
|
||||
- `fix(...)`/`fix:` subject alone (+2)
|
||||
- `## Symptom Verification` regression marker in the commit body (+3, FN-5893)
|
||||
- test file first **added** together with source (+2), or added alone (+1)
|
||||
- a plain test+source co-change modify (+1.5)
|
||||
|
||||
**Negative signal** (low value / churn):
|
||||
- subject/body mentions `flake`, `flaky`, `deflake`, `quarantine`, `stabiliz`, `appease`, `timeout`, `retry` (−3)
|
||||
- a modify that touched **only** the test file, no source (−1)
|
||||
- appears in `scripts/lib/test-quarantine.json` history (−5)
|
||||
|
||||
The per-file **valueScore** is the sum of commit weights (plus the quarantine penalty).
|
||||
**deletionPriority** = `durationMs / (1 + max(0, valueScore))` with a small boost for
|
||||
net-negative files — so the ranking surfaces **slow AND low-value** files first
|
||||
(most CI time saved per unit of lost signal). Recommendation: `delete` (≤0), `review`
|
||||
(≤3), else `keep`.
|
||||
|
||||
## Caveats (read before deleting anything)
|
||||
|
||||
- **Heuristic, not truth.** A quiet, never-modified test can still be load-bearing;
|
||||
a high-churn test can still be valuable. Use this to *prioritize human review*.
|
||||
- **`git log --follow` / rename limits.** Renames are followed only through linear
|
||||
`R`/`C` name-status chains; squash-merges collapse multi-commit history into one
|
||||
subject, so per-commit signal is lost for squashed work (this repo defaults to
|
||||
squash merges — a major reason to treat scores as lower bounds on value).
|
||||
- **Subjects lie.** `fix(...)` is trusted as a positive even if the test was unrelated;
|
||||
conversely a real bug fixed under a `feat(...)`/`FN-` subject without source co-change
|
||||
may be undercounted.
|
||||
- **Timing is a snapshot** from `scripts/test-timings.json`; files with no entry show
|
||||
`n/a` duration and get deletionPriority 0 (cost unknown, not necessarily cheap).
|
||||
- **Not a green light.** Deleting a gate test still requires the gate-eviction process
|
||||
(AGENTS.md). "Safe delete" only flags files that meet the ratchet's churn/quarantine bar.
|
||||
|
||||
## Top 40 deletion candidates
|
||||
|
||||
| # | File | Tests | Duration | Value | Priority | Rec | Why (recent commits) |
|
||||
|---|------|------:|---------:|------:|---------:|-----|----------------------|
|
||||
| 1 | `packages/engine/src/__tests__/workspace-merger-idempotency.test.ts` | 8 | 12.7s | -1 | 13335 | delete (safe-delete) | `754434632` feat(workspace): Phase C U2 — per-repo landed pr _(churn-keyword,added-with-source)_<br>`627bdcfb0` fix(review): Phase C merge-loop hardening — doub _(churn-keyword,fix+source)_<br>`3a7123762` fix(review): address PR #1717 Phase C merge-loop _(churn-keyword,fix+source)_ |
|
||||
| 2 | `packages/dashboard/src/__tests__/github-tracking-delete.test.ts` | 9 | 8.7s | -7.5 | 11962.5 | delete | `d0c9e47f6` feat(FN-4253): close linked GitHub issues on tas _(added-with-source)_<br>`9dfefc551` feat(FN-4941): close or delete GitHub issues whe _(test+source)_<br>`46fb3f0ef` feat(FN-5305): fix async synchronization in gith _(test-only-churn)_ |
|
||||
| 3 | `packages/engine/src/__tests__/reliability-interactions/merge-runner-spawn-enoent-prevention.test.ts` | 5 | 7.9s | -2 | 8690 | delete (safe-delete) | `7763ba5fa` FN-6278: preflight reusable merge worktree cwd _(churn-keyword,added-with-source)_<br>`a768d36da` FN-6817: root reliability fixtures under worker _(test-only-churn)_ |
|
||||
| 4 | `packages/dashboard/src/__tests__/session-reconnect.test.ts` | 4 | 6.1s | -6.5 | 8082.5 | delete (quarantined) | `bf141c47d` test(FN-1156): add AI session lifecycle and reco _(added)_<br>`84ed84313` feat(FN-2123): merge fusion/fn-2123 _(test-only-churn)_<br>`6714f7654` refactor(FN-2161): standardize on createFnAgent _(test+source)_ |
|
||||
| 5 | `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` | 7 | 4.6s | -12 | 7360 | delete (quarantined) | `602a7da51` feat(FN-3865): complete Steps 3-6 — recover alre _(added-with-source)_<br>`39e0b3c6a` fix(FN-3865): align merge recovery types and rea _(fix+source)_<br>`6fc804483` feat(FN-3940): expose github tracking controls i _(test-only-churn)_ |
|
||||
| 6 | `packages/core/src/__tests__/run-audit.integration.test.ts` | 24 | 5.8s | -2 | 6380 | delete (quarantined) | `c2d0ac572` test(FN-1414): add run-audit integration tests f _(churn-keyword,added)_<br>`84ed84313` feat(FN-2123): merge fusion/fn-2123 _(test+source)_<br>`a269f4829` refactor: remove legacy kb compatibility _(test+source)_ |
|
||||
| 7 | `packages/engine/src/__tests__/workspace-merger-lease.test.ts` | 4 | 5.8s | -1 | 6090 | delete (safe-delete) | `64e87f9a1` feat(workspace): Phase C U3 — per-repo land leas _(churn-keyword,added-with-source)_<br>`627bdcfb0` fix(review): Phase C merge-loop hardening — doub _(churn-keyword,fix+source)_ |
|
||||
| 8 | `packages/engine/src/__tests__/workspace-merger.test.ts` | 7 | 8.0s | 0.5 | 5333.3 | review | `744ed098a` feat(workspace): Phase C U1 — per-repo merge loo _(churn-keyword,added-with-source)_<br>`754434632` feat(workspace): Phase C U2 — per-repo landed pr _(churn-keyword,test+source)_<br>`3a7123762` fix(review): address PR #1717 Phase C merge-loop _(churn-keyword,fix+source)_ |
|
||||
| 9 | `packages/cli/src/__tests__/extension-task-tools.test.ts` | 4 | 3.8s | -7 | 5130 | delete (quarantined) | `47eaf089b` feat(FN-4904): complete Step 3 — align engine an _(added-with-source)_<br>`0dc4c9c6a` test(FN-4904): cover worktree-root lookup in ext _(test-only-churn)_<br>`b80b517a4` test(FN-4927): cover no-task fallback for task t _(test-only-churn)_ |
|
||||
| 10 | `packages/engine/src/__tests__/worktree-db-hydrate.test.ts` | 12 | 7.1s | 0.5 | 4733.3 | review | `e59a740d0` feat(FN-3841): add worktree database hydration t _(added-with-source)_<br>`6acc51cf0` feat(FN-4039): recover worktree db scratch boots _(test+source)_<br>`93dc5a727` feat(FN-4083): add WAL enforcement, immediate wr _(test+source)_ |
|
||||
| 11 | `packages/core/src/__tests__/activity-analytics.test.ts` | 25 | 4.6s | 0 | 4600 | delete (quarantined) | `53bb1d8f3` feat(analytics): U2 — core date-range aggregator _(added-with-source)_<br>`5bc8901f0` feat(command-center): U7 — SDLC funnel + through _(test+source)_<br>`f5bd86214` feat(monitor): U13 — monitor stage (deployments, _(test+source)_ |
|
||||
| 12 | `packages/dashboard/src/routes/__tests__/board-workflows-route.test.ts` | 4 | 3.6s | -5.5 | 4590 | delete | `010976cee` test: promote/board-workflows routes, concurrent _(added-with-source)_<br>`f77aa073c` FN-6025: fix builtin coding auto-merge review fl _(test-only-churn)_<br>`edde74d56` FN-6146: harden dashboard test mocks and request _(churn-keyword,test+source)_ |
|
||||
| 13 | `packages/dashboard/app/components/__tests__/TaskDetailModal.responsive-and-dependencies.test.tsx` | 44 | 6.6s | 0.5 | 4400 | review | `ed52c3dff` test(FN-3591): split TaskDetailModal coverage in _(added)_<br>`ecbf1f829` feat(FN-3276): add task review tab with metadata _(test+source)_<br>`309e3f91a` feat(FN-3897): add blocker fan-out hook, badge, _(test+source)_ |
|
||||
| 14 | `packages/core/src/__tests__/todo-store.test.ts` | 17 | 3.8s | -3 | 4370 | delete (quarantined) | `6f4b42a89` feat(FN-2575): merge fusion/fn-2575 _(added-with-source)_ |
|
||||
| 15 | `packages/engine/src/__tests__/reliability-interactions/soft-blocker-auto-finalize-interactions.real-git.test.ts` | 7 | 3.4s | -4 | 4080 | delete | `208753fb8` test(FN-4653): complete Step 1 — add merge-path _(added)_<br>`875b32266` test(FN-4653): complete Step 2 — cover mixed blo _(test-only-churn)_<br>`5548b6f4a` test(FN-4653): complete Step 3 — add scheduler s _(test-only-churn)_ |
|
||||
| 16 | `packages/core/src/__tests__/move-task-characterization.test.ts` | 10 | 8.1s | 1 | 4050 | review | `4e1b0fab0` feat(core): workflow-resolved transitions behind _(added-with-source)_<br>`0b7549a53` FN-6126: enable workflow experimental flags by d _(test+source)_<br>`9a7881441` FN-6245: keep default auto-merge tasks on live s _(test+source)_ |
|
||||
| 17 | `packages/engine/src/__tests__/pr-response-run.test.ts` | 23 | 11.6s | 2 | 3866.7 | review | `31d4b5335` feat(pr): security-hardened review-response run _(added-with-source)_ |
|
||||
| 18 | `packages/engine/src/__tests__/merger-autostash-orphan-surface.test.ts` | 6 | 3.7s | 0 | 3700 | delete | `d604d090a` feat(FN-3863): add stash recovery dashboard surf _(added-with-source)_<br>`bcb4107a1` feat(FN-3932): add orphaned finalize-reset autos _(test+source)_<br>`1cf987208` feat(FN-4018): isolate split merger temp workspa _(test-only-churn)_ |
|
||||
| 19 | `packages/engine/src/__tests__/merger-empty-cherry-pick-fallback.test.ts` | 2 | 3.0s | -1.5 | 3225 | delete | `653500959` test(FN-4475): add fallback empty cherry-pick re _(added)_<br>`ebd79d21b` feat(FN-4475): remove unstable merger empty cher _(churn-keyword,test-only-churn)_<br>`22d59b5f9` feat(FN-5279): add merge integration worktree fe _(test+source)_ |
|
||||
| 20 | `packages/engine/src/__tests__/merger-ai-cleanup.test.ts` | 15 | 2.8s | -3 | 3220 | delete (quarantined) | `4032a35a2` FN-6188: harden AI merge temp worktree cleanup _(added-with-source)_<br>`d75f861f2` FN-6199: harden AI merge worktree cleanup _(test+source)_<br>`3e69d3990` FN-6244: protect active AI merge worktrees _(churn-keyword,test+source)_ |
|
||||
| 21 | `packages/dashboard/src/__tests__/routes-secrets-sync.test.ts` | 17 | 2.7s | -3 | 3105 | delete | `cca96a760` test(FN-4913): complete Step 7 — add secrets syn _(added-with-source)_<br>`b5e674059` test(FN-4980): cover sync-export auth rejection _(test-only-churn)_<br>`47775e942` test(FN-4981): complete Step 2 — add push missin _(test-only-churn)_ |
|
||||
| 22 | `packages/core/src/__tests__/store-concurrent-writes.test.ts` | 6 | 2.8s | -1.5 | 3010 | delete (quarantined) | `93dc5a727` feat(FN-4083): add WAL enforcement, immediate wr _(added-with-source)_<br>`67c680c8d` fix(FN-4122): use unique tmp filename for task.j _(fix+source)_<br>`a38752c6b` FN-6486: rescue quarantined flaky tests _(churn-keyword,test+source)_ |
|
||||
| 23 | `packages/core/src/__tests__/store-archive-search.test.ts` | 63 | 3.0s | 0 | 3000 | delete | `ac6aaca2c` feat(FN-3982): split monolithic store.test.ts in _(added)_<br>`139e25fcf` feat(FN-4837): complete Step 2 — migrate store t _(test-only-churn)_<br>`294209f64` FN-5943: maintain tasks FTS5 indexes automatical _(test+source)_ |
|
||||
| 24 | `packages/engine/src/__tests__/reliability-interactions/in-review-branch-rebind.test.ts` | 10 | 7.4s | 1.5 | 2960 | review | `5c3096c7d` feat(FN-5083): complete Steps 5-7 rebind UI and _(added)_<br>`b3ac0beaa` test(FN-5083): tighten rebind reliability and qu _(test-only-churn)_<br>`3566cf8a1` FN-6695: block unsafe in-review branch rebinds _(test+source)_ |
|
||||
| 25 | `packages/core/src/__tests__/store-activity.test.ts` | 49 | 5.9s | 1 | 2950 | review | `36e130937` refactor(FN-4021): split core store tests into d _(added)_<br>`287ebaf10` feat(FN-4053): unify task ID allocation with sto _(test+source)_<br>`a0f2bc530` feat(FN-4429): complete Step 1 — plumb moveSourc _(test+source)_ |
|
||||
| 26 | `packages/dashboard/app/components/__tests__/AgentDetailView.mobile-scroll.test.tsx` | 4 | 2.5s | -3 | 2875 | delete | `3a4397215` feat(FN-4231): restore mobile scroll in AgentDet _(added)_<br>`d68fe3e9f` FN-6209: fix tablet tab overflow in agent detail _(test-only-churn)_<br>`b7c23c09c` FN-6450: enable touch scrolling for agent detail _(test-only-churn)_ |
|
||||
| 27 | `packages/dashboard/src/__tests__/insights-routes.test.ts` | 24 | 26.5s | 8.5 | 2789.5 | keep | `629398483` fix(FN-1909): reorder insights routes to prevent _(fix+source,added-with-source)_<br>`8de7feeb1` feat(FN-1974): merge fusion/fn-1974 _(test+source)_<br>`6714f7654` refactor(FN-2161): standardize on createFnAgent _(test+source)_ |
|
||||
| 28 | `packages/engine/src/__tests__/branch-conflicts-ghost-references.test.ts` | 5 | 2.5s | -2 | 2750 | delete | `663e5d2cd` feat(FN-4508): complete Step 1 — harden inspectB _(added-with-source)_<br>`5cd0e6665` feat(FN-4839): harden real-git test timeouts acr _(churn-keyword,test-only-churn)_ |
|
||||
| 29 | `packages/core/src/__tests__/soft-delete-lineage-children.test.ts` | 11 | 2.6s | 0 | 2600 | delete | `e0c0745c0` test(FN-5129): add lineage soft-delete and archi _(added-with-source)_<br>`2c7ae1c1a` feat(FN-5131): add lineage-unlink flag to triage _(test-only-churn)_<br>`4955ccf6e` feat(FN-5132): align soft-delete-lineage-childre _(test-only-churn)_ |
|
||||
| 30 | `packages/core/src/__tests__/store-handoff-to-review.test.ts` | 8 | 2.6s | 0 | 2600 | delete (quarantined) | `93b11c6c0` feat(FN-5241): add atomic in-review handoff seam _(added-with-source)_<br>`2d425b1e2` fix: scrub queued/blockedBy/overlapBlockedBy on _(fix+source)_ |
|
||||
| 31 | `packages/engine/src/__tests__/branch-conflicts-zero-unique.test.ts` | 4 | 2.2s | -2.5 | 2475 | delete | `bcb0035d1` feat(FN-4500): complete Step 2 — harden zero-uni _(added-with-source)_<br>`14ef1041a` feat(FN-4500): complete Step 3 — add live zero-c _(test+source)_<br>`5ce377afe` feat(FN-4508): complete Step 7 — docs and compat _(test-only-churn)_ |
|
||||
| 32 | `packages/engine/src/__tests__/workspace-e2e.test.ts` | 2 | 4.8s | 1 | 2400 | review | `78d7a28f1` test(workspace): Phase D U2 — e2e merge + recove _(added)_ |
|
||||
| 33 | `packages/cli/src/__tests__/research-extension-tools.test.ts` | 10 | 2.1s | -2.5 | 2362.5 | delete (quarantined) | `6670837b7` feat(FN-2996): add research agent tools with ext _(added-with-source)_<br>`f2fa44e27` feat(FN-2999): harden research lifecycle with id _(churn-keyword,test+source)_<br>`f1ee69e59` feat(FN-3014): document research recovery semant _(test+source)_ |
|
||||
| 34 | `packages/cli/src/__tests__/extension-mission-goal-tools.test.ts` | 4 | 1.7s | -7 | 2295 | delete (quarantined) | `93e8bd994` FN-5899: add mission-goal linking commands and t _(added-with-source)_<br>`26bc80a0a` FN-5958: add mission goal linking to create and _(test+source)_<br>`259cdfb53` FN-6734: stabilize CLI extension test cleanup _(churn-keyword,test-only-churn)_ |
|
||||
| 35 | `packages/engine/src/__tests__/worktree-acquisition-workspace.test.ts` | 8 | 6.6s | 2 | 2200 | review | `12d33c512` feat(workspace): Phase A U2 — per-repo acquisiti _(added-with-source)_<br>`d5fa8654f` fix(review): Phase A workspace hardening — tool _(churn-keyword,fix+source)_ |
|
||||
| 36 | `packages/dashboard/src/routes/__tests__/workflow-import-export.test.ts` | 10 | 6.5s | 2 | 2166.7 | review | `407ce03bf` feat(dashboard): workflow and template JSON impo _(added-with-source)_ |
|
||||
| 37 | `packages/cli/src/__tests__/extension-goal-tools.test.ts` | 7 | 1.6s | -7 | 2160 | delete (quarantined) | `b335f3d7c` FN-5652: add goal retrieval tool for agent execu _(added-with-source)_<br>`fab8a62b5` FN-5977: expose goal retrieval tools across engi _(test+source)_<br>`259cdfb53` FN-6734: stabilize CLI extension test cleanup _(churn-keyword,test-only-churn)_ |
|
||||
| 38 | `packages/dashboard/app/components/__tests__/AgentDetailView.settings.test.tsx` | 25 | 10.7s | 4 | 2140 | keep | `d98c053ed` feat(FN-4088): split AgentDetailView tests into _(added)_<br>`91931bd5d` feat(FN-4394): complete Step 5 — add dashboard s _(test+source)_<br>`6b2607dae` feat(FN-4400): complete Step 7 — add prompt-size _(test+source)_ |
|
||||
| 39 | `packages/engine/src/cli-agent/__tests__/session-manager.test.ts` | 19 | 4.2s | 1 | 2100 | review | `4fc1a9dd4` feat(engine): add CliAgentAdapter interface and _(added-with-source)_<br>`773ba7620` test(engine): self-skip real-PTY e2e when PTY I/ _(test-only-churn)_ |
|
||||
| 40 | `packages/engine/src/__tests__/reliability-interactions/ai-merge-worktree-cleanup.test.ts` | 7 | 6.3s | 2 | 2100 | review | `ac342f501` FN-6220: harden AI merge worktree cleanup _(churn-keyword,added-with-source)_<br>`2085610e9` FN-6246: move AI merge clean rooms into repo sto _(test+source)_<br>`dc4c2b220` FN-6453: clean up AI-merge worktrees after setup _(test+source)_ |
|
||||
|
||||
### One-line recommendations
|
||||
|
||||
1. `packages/engine/src/__tests__/workspace-merger-idempotency.test.ts` — **delete**: no positive (bug-encoding) signal in history; pure churn — strong delete candidate (slow: 12.7s — high CI time payoff)
|
||||
2. `packages/dashboard/src/__tests__/github-tracking-delete.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 8.7s — high CI time payoff)
|
||||
3. `packages/engine/src/__tests__/reliability-interactions/merge-runner-spawn-enoent-prevention.test.ts` — **delete**: no positive (bug-encoding) signal in history; pure churn — strong delete candidate (slow: 7.9s — high CI time payoff)
|
||||
4. `packages/dashboard/src/__tests__/session-reconnect.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 6.1s — high CI time payoff)
|
||||
5. `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 4.6s — high CI time payoff)
|
||||
6. `packages/core/src/__tests__/run-audit.integration.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 5.8s — high CI time payoff)
|
||||
7. `packages/engine/src/__tests__/workspace-merger-lease.test.ts` — **delete**: no positive (bug-encoding) signal in history; pure churn — strong delete candidate (slow: 5.8s — high CI time payoff)
|
||||
8. `packages/engine/src/__tests__/workspace-merger.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 8.0s — high CI time payoff)
|
||||
9. `packages/cli/src/__tests__/extension-task-tools.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 3.8s — high CI time payoff)
|
||||
10. `packages/engine/src/__tests__/worktree-db-hydrate.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 7.1s — high CI time payoff)
|
||||
11. `packages/core/src/__tests__/activity-analytics.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 4.6s — high CI time payoff)
|
||||
12. `packages/dashboard/src/routes/__tests__/board-workflows-route.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.6s — high CI time payoff)
|
||||
13. `packages/dashboard/app/components/__tests__/TaskDetailModal.responsive-and-dependencies.test.tsx` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 6.6s — high CI time payoff)
|
||||
14. `packages/core/src/__tests__/todo-store.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 3.8s — high CI time payoff)
|
||||
15. `packages/engine/src/__tests__/reliability-interactions/soft-blocker-auto-finalize-interactions.real-git.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.4s — high CI time payoff)
|
||||
16. `packages/core/src/__tests__/move-task-characterization.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 8.1s — high CI time payoff)
|
||||
17. `packages/engine/src/__tests__/pr-response-run.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 11.6s — high CI time payoff)
|
||||
18. `packages/engine/src/__tests__/merger-autostash-orphan-surface.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.7s — high CI time payoff)
|
||||
19. `packages/engine/src/__tests__/merger-empty-cherry-pick-fallback.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.0s — high CI time payoff)
|
||||
20. `packages/engine/src/__tests__/merger-ai-cleanup.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 2.8s — high CI time payoff)
|
||||
21. `packages/dashboard/src/__tests__/routes-secrets-sync.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.7s — high CI time payoff)
|
||||
22. `packages/core/src/__tests__/store-concurrent-writes.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 2.8s — high CI time payoff)
|
||||
23. `packages/core/src/__tests__/store-archive-search.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.0s — high CI time payoff)
|
||||
24. `packages/engine/src/__tests__/reliability-interactions/in-review-branch-rebind.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 7.4s — high CI time payoff)
|
||||
25. `packages/core/src/__tests__/store-activity.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 5.9s — high CI time payoff)
|
||||
26. `packages/dashboard/app/components/__tests__/AgentDetailView.mobile-scroll.test.tsx` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.5s — high CI time payoff)
|
||||
27. `packages/dashboard/src/__tests__/insights-routes.test.ts` — **keep**: carries real bug-fix / source-coupled signal — keep
|
||||
28. `packages/engine/src/__tests__/branch-conflicts-ghost-references.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.5s — high CI time payoff)
|
||||
29. `packages/core/src/__tests__/soft-delete-lineage-children.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.6s — high CI time payoff)
|
||||
30. `packages/core/src/__tests__/store-handoff-to-review.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 2.6s — high CI time payoff)
|
||||
31. `packages/engine/src/__tests__/branch-conflicts-zero-unique.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.2s — high CI time payoff)
|
||||
32. `packages/engine/src/__tests__/workspace-e2e.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 4.8s — high CI time payoff)
|
||||
33. `packages/cli/src/__tests__/research-extension-tools.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 2.1s — high CI time payoff)
|
||||
34. `packages/cli/src/__tests__/extension-mission-goal-tools.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 1.7s — high CI time payoff)
|
||||
35. `packages/engine/src/__tests__/worktree-acquisition-workspace.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 6.6s — high CI time payoff)
|
||||
36. `packages/dashboard/src/routes/__tests__/workflow-import-export.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 6.5s — high CI time payoff)
|
||||
37. `packages/cli/src/__tests__/extension-goal-tools.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 1.6s — high CI time payoff)
|
||||
38. `packages/dashboard/app/components/__tests__/AgentDetailView.settings.test.tsx` — **keep**: carries real bug-fix / source-coupled signal — keep
|
||||
39. `packages/engine/src/cli-agent/__tests__/session-manager.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 4.2s — high CI time payoff)
|
||||
40. `packages/engine/src/__tests__/reliability-interactions/ai-merge-worktree-cleanup.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 6.3s — high CI time payoff)
|
||||
|
||||
171
scripts/__tests__/test-value-audit.test.mjs
Normal file
171
scripts/__tests__/test-value-audit.test.mjs
Normal file
@@ -0,0 +1,171 @@
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
|
||||
import {
|
||||
classifyCommit,
|
||||
scoreFile,
|
||||
isFixSubject,
|
||||
WEIGHTS,
|
||||
NEGATIVE_KEYWORDS,
|
||||
} from "../lib/test-value-audit-lib.mjs";
|
||||
|
||||
/*
|
||||
FNXC:TestInfrastructure 2026-06-25-00:00:
|
||||
Unit coverage for the test-value-audit classifier. Feeds synthetic commit records and
|
||||
asserts the scoring so the heuristic's behavior is pinned independent of git/FS IO.
|
||||
*/
|
||||
|
||||
// --- isFixSubject ---
|
||||
test("isFixSubject matches conventional fix subjects incl. fix(FN-XXXX)", () => {
|
||||
assert.ok(isFixSubject("fix(FN-1234): correct merge race"));
|
||||
assert.ok(isFixSubject("fix: handle null"));
|
||||
assert.ok(isFixSubject("fix!: breaking repair"));
|
||||
assert.ok(!isFixSubject("feat(FN-1): add thing"));
|
||||
assert.ok(!isFixSubject("prefix change"));
|
||||
});
|
||||
|
||||
// --- classifyCommit positive signals ---
|
||||
test("fix subject + source change is strongest positive", () => {
|
||||
const c = classifyCommit({ subject: "fix(FN-9): bug", body: "", status: "M", touchedSource: true });
|
||||
assert.equal(c.category, "positive");
|
||||
assert.equal(c.weight, WEIGHTS.fixWithSource);
|
||||
assert.ok(c.signals.includes("fix+source"));
|
||||
});
|
||||
|
||||
test("fix subject without source change is a weaker positive", () => {
|
||||
const c = classifyCommit({ subject: "fix: bug", body: "", status: "M", touchedSource: false });
|
||||
assert.equal(c.weight, WEIGHTS.fix);
|
||||
assert.ok(c.signals.includes("fix"));
|
||||
});
|
||||
|
||||
test("symptom verification marker adds positive weight on top of fix", () => {
|
||||
const c = classifyCommit({
|
||||
subject: "fix(FN-5893): invariant",
|
||||
body: "## Symptom Verification\nOriginal symptom: ...",
|
||||
status: "M",
|
||||
touchedSource: true,
|
||||
});
|
||||
assert.equal(c.weight, WEIGHTS.fixWithSource + WEIGHTS.symptomVerification);
|
||||
assert.ok(c.signals.includes("symptom-verification"));
|
||||
});
|
||||
|
||||
test("added with source is positive; added alone is weak positive", () => {
|
||||
const withSrc = classifyCommit({ subject: "feat: x", status: "A", touchedSource: true });
|
||||
assert.equal(withSrc.weight, WEIGHTS.addedWithSource);
|
||||
assert.ok(withSrc.signals.includes("added-with-source"));
|
||||
|
||||
const alone = classifyCommit({ subject: "test: x", status: "A", touchedSource: false });
|
||||
assert.equal(alone.weight, WEIGHTS.added);
|
||||
assert.ok(alone.signals.includes("added"));
|
||||
});
|
||||
|
||||
test("non-fix test+source modify is a mild positive", () => {
|
||||
const c = classifyCommit({ subject: "refactor: rework", status: "M", touchedSource: true });
|
||||
assert.equal(c.weight, WEIGHTS.testPlusSource);
|
||||
assert.ok(c.signals.includes("test+source"));
|
||||
});
|
||||
|
||||
// --- classifyCommit negative signals ---
|
||||
test("flake/quarantine keywords produce negative weight", () => {
|
||||
for (const kw of NEGATIVE_KEYWORDS) {
|
||||
const c = classifyCommit({ subject: `chore: ${kw} the test`, status: "M", touchedSource: false });
|
||||
assert.ok(c.weight <= WEIGHTS.churnKeyword, `keyword ${kw} should be negative`);
|
||||
assert.ok(c.signals.includes("churn-keyword"));
|
||||
}
|
||||
});
|
||||
|
||||
test("test-only modify (no source, no fix) is churn", () => {
|
||||
const c = classifyCommit({ subject: "chore: tidy assertions", status: "M", touchedSource: false });
|
||||
assert.equal(c.category, "negative");
|
||||
assert.equal(c.weight, WEIGHTS.testOnlyChurn);
|
||||
assert.ok(c.signals.includes("test-only-churn"));
|
||||
});
|
||||
|
||||
test("a fix that also mentions a churn keyword nets toward neutral (mixed signal)", () => {
|
||||
const c = classifyCommit({
|
||||
subject: "fix(FN-1): widen timeout to stabilize merge",
|
||||
status: "M",
|
||||
touchedSource: true,
|
||||
});
|
||||
// churnKeyword(-3) + fixWithSource(+3) = 0 -> neutral, both signals recorded.
|
||||
assert.equal(c.weight, 0);
|
||||
assert.equal(c.category, "neutral");
|
||||
assert.ok(c.signals.includes("churn-keyword"));
|
||||
assert.ok(c.signals.includes("fix+source"));
|
||||
});
|
||||
|
||||
// --- scoreFile aggregation ---
|
||||
test("scoreFile sums weights and recommends keep for net-positive files", () => {
|
||||
const commits = [
|
||||
classifyCommit({ subject: "test: add", status: "A", touchedSource: true }), // +2
|
||||
classifyCommit({ subject: "fix(FN-2): real bug", status: "M", touchedSource: true }), // +3
|
||||
];
|
||||
const s = scoreFile({ commits, durationMs: 500, testCount: 4 });
|
||||
assert.equal(s.valueScore, 5);
|
||||
assert.equal(s.positiveCount, 2);
|
||||
assert.equal(s.recommendation, "keep");
|
||||
assert.equal(s.safeDelete, false);
|
||||
});
|
||||
|
||||
test("scoreFile recommends delete + safeDelete for pure churn with no positives", () => {
|
||||
const commits = [
|
||||
classifyCommit({ subject: "chore: tweak", status: "M", touchedSource: false }), // -1
|
||||
classifyCommit({ subject: "chore: deflake", status: "M", touchedSource: false }), // -3 (and -1 churn? no: keyword path only)
|
||||
];
|
||||
const s = scoreFile({ commits, durationMs: 9000, testCount: 3 });
|
||||
assert.ok(s.valueScore < 0);
|
||||
assert.equal(s.positiveCount, 0);
|
||||
assert.equal(s.recommendation, "delete");
|
||||
assert.equal(s.safeDelete, true);
|
||||
});
|
||||
|
||||
test("quarantine ledger membership forces delete and applies the penalty", () => {
|
||||
const commits = [classifyCommit({ subject: "fix(FN-3): bug", status: "M", touchedSource: true })]; // +3
|
||||
const s = scoreFile({ commits, durationMs: 100, testCount: 2, quarantined: true });
|
||||
// +3 then -5 quarantine = -2 => delete; but positiveCount=1 so NOT safeDelete.
|
||||
assert.equal(s.valueScore, WEIGHTS.fixWithSource + WEIGHTS.quarantineLedger);
|
||||
assert.equal(s.recommendation, "delete");
|
||||
assert.equal(s.quarantined, true);
|
||||
assert.equal(s.safeDelete, false);
|
||||
});
|
||||
|
||||
test("deletionPriority surfaces slow + low-value above slow + valuable", () => {
|
||||
const lowValueSlow = scoreFile({
|
||||
commits: [classifyCommit({ subject: "chore: tweak", status: "M", touchedSource: false })],
|
||||
durationMs: 10_000,
|
||||
testCount: 1,
|
||||
});
|
||||
const highValueSlow = scoreFile({
|
||||
commits: [
|
||||
classifyCommit({ subject: "fix(FN-4): bug", status: "M", touchedSource: true }),
|
||||
classifyCommit({ subject: "fix(FN-5): bug2", status: "M", touchedSource: true }),
|
||||
],
|
||||
durationMs: 10_000,
|
||||
testCount: 1,
|
||||
});
|
||||
assert.ok(
|
||||
lowValueSlow.deletionPriority > highValueSlow.deletionPriority,
|
||||
"low-value slow file must outrank high-value slow file",
|
||||
);
|
||||
});
|
||||
|
||||
test("missing duration yields zero deletionPriority (unknown cost)", () => {
|
||||
const s = scoreFile({
|
||||
commits: [classifyCommit({ subject: "chore: tweak", status: "M", touchedSource: false })],
|
||||
durationMs: null,
|
||||
testCount: 1,
|
||||
});
|
||||
assert.equal(s.deletionPriority, 0);
|
||||
assert.equal(s.durationMs, null);
|
||||
});
|
||||
|
||||
test("testOnlyRatio reflects churn fraction", () => {
|
||||
const commits = [
|
||||
classifyCommit({ subject: "test: add", status: "A", touchedSource: true }), // not churn
|
||||
classifyCommit({ subject: "chore: tidy", status: "M", touchedSource: false }), // churn
|
||||
classifyCommit({ subject: "chore: tidy2", status: "M", touchedSource: false }), // churn
|
||||
];
|
||||
const s = scoreFile({ commits, durationMs: 1, testCount: 1 });
|
||||
assert.equal(s.testOnlyCount, 2);
|
||||
assert.equal(s.testOnlyRatio, Number((2 / 3).toFixed(3)));
|
||||
});
|
||||
203
scripts/lib/test-value-audit-lib.mjs
Normal file
203
scripts/lib/test-value-audit-lib.mjs
Normal file
@@ -0,0 +1,203 @@
|
||||
/*
|
||||
FNXC:TestInfrastructure 2026-06-25-00:00:
|
||||
Heuristic "test value audit" scoring library. Pure functions only — all git/FS IO
|
||||
lives in scripts/test-value-audit.mjs so the classification logic stays unit-testable
|
||||
with synthetic commit records.
|
||||
|
||||
Why this exists (requirement): the suite has rotted before (FN-5048 slow tests, the
|
||||
quarantine deletion-ratchet in AGENTS.md). We want a data-driven signal for WHICH test
|
||||
files have actually encoded real bugs vs. only ever churned/flaked, so a human can drive
|
||||
aggressive deletion of low-signal-yet-slow tests. This is a HEURISTIC, never ground truth:
|
||||
git history is lossy (renames, squashes), commit subjects lie, and a quiet test can still
|
||||
be load-bearing. Treat the output as evidence, not a verdict.
|
||||
|
||||
Scoring model (transparent + auditable on purpose):
|
||||
- POSITIVE signal = the commit encoded/caught a real behavioral bug. Strongest when a
|
||||
`fix(...)` / `fix:` subject lands alongside a sibling SOURCE change, or carries a
|
||||
`## Symptom Verification` regression marker (FN-5893), or the test file was first ADDED
|
||||
together with source.
|
||||
- NEGATIVE signal = low value / churn. Commits whose subject/body mention flake words
|
||||
(flake/quarantine/stabiliz/appease/timeout/retry/flaky/deflake) or that repeatedly
|
||||
touch ONLY the test file with no source change. Appearing in the quarantine ledger
|
||||
history is the strongest negative.
|
||||
*/
|
||||
|
||||
/** Churn / appeasement keywords. A commit mentioning any of these is a negative signal. */
|
||||
export const NEGATIVE_KEYWORDS = [
|
||||
"flake",
|
||||
"flaky",
|
||||
"deflake",
|
||||
"quarantine",
|
||||
"stabiliz", // stabilize / stabilization / stabilise
|
||||
"appease",
|
||||
"timeout",
|
||||
"retry",
|
||||
];
|
||||
|
||||
/** Per-commit signed weights. Documented so the report can explain every score. */
|
||||
export const WEIGHTS = {
|
||||
churnKeyword: -3,
|
||||
fixWithSource: 3,
|
||||
fix: 2,
|
||||
symptomVerification: 3,
|
||||
addedWithSource: 2,
|
||||
added: 1,
|
||||
testPlusSource: 1.5,
|
||||
testOnlyChurn: -1,
|
||||
quarantineLedger: -5,
|
||||
};
|
||||
|
||||
/** Recommendation thresholds on the final summed value score. */
|
||||
export const THRESHOLDS = {
|
||||
delete: 0, // valueScore <= 0 => delete candidate
|
||||
review: 3, // 0 < valueScore <= 3 => review
|
||||
// valueScore > 3 => keep
|
||||
};
|
||||
|
||||
/**
|
||||
* True when the subject is a conventional `fix(...)` / `fix:` / `fix!:` commit.
|
||||
* The repo convention is `fix(FN-XXXX):` so this also captures task bug fixes.
|
||||
* @param {string} subject
|
||||
*/
|
||||
export function isFixSubject(subject) {
|
||||
return /^\s*fix(\(|:|!)/i.test(subject ?? "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a single commit that touched a given test file.
|
||||
*
|
||||
* @param {object} record
|
||||
* @param {string} [record.subject] commit subject line
|
||||
* @param {string} [record.body] commit body
|
||||
* @param {string} [record.status] git name-status of the TEST file in this commit ("A","M","D","R"...)
|
||||
* @param {boolean} [record.touchedSource] did the commit also change a non-test source file in the same package?
|
||||
* @returns {{category:"positive"|"negative"|"neutral", weight:number, signals:string[]}}
|
||||
*/
|
||||
export function classifyCommit(record) {
|
||||
const subject = record.subject ?? "";
|
||||
const body = record.body ?? "";
|
||||
const status = record.status ?? "M";
|
||||
const touchedSource = Boolean(record.touchedSource);
|
||||
const text = `${subject}\n${body}`.toLowerCase();
|
||||
|
||||
const isAdd = status.startsWith("A");
|
||||
const isFix = isFixSubject(subject);
|
||||
const hasSymptom = /symptom verification/i.test(body) || /symptom verification/i.test(subject);
|
||||
const churnKeyword = NEGATIVE_KEYWORDS.some((k) => text.includes(k));
|
||||
|
||||
let weight = 0;
|
||||
const signals = [];
|
||||
|
||||
if (churnKeyword) {
|
||||
weight += WEIGHTS.churnKeyword;
|
||||
signals.push("churn-keyword");
|
||||
}
|
||||
|
||||
if (isFix && touchedSource) {
|
||||
weight += WEIGHTS.fixWithSource;
|
||||
signals.push("fix+source");
|
||||
} else if (isFix) {
|
||||
weight += WEIGHTS.fix;
|
||||
signals.push("fix");
|
||||
}
|
||||
|
||||
if (hasSymptom) {
|
||||
weight += WEIGHTS.symptomVerification;
|
||||
signals.push("symptom-verification");
|
||||
}
|
||||
|
||||
if (isAdd && touchedSource) {
|
||||
weight += WEIGHTS.addedWithSource;
|
||||
signals.push("added-with-source");
|
||||
} else if (isAdd) {
|
||||
weight += WEIGHTS.added;
|
||||
signals.push("added");
|
||||
}
|
||||
|
||||
// Test+source co-change (non-fix, non-add modify) is real behavioral coverage.
|
||||
if (!isAdd && !isFix && touchedSource) {
|
||||
weight += WEIGHTS.testPlusSource;
|
||||
signals.push("test+source");
|
||||
}
|
||||
|
||||
// A modify that touched ONLY the test file (no source, not a fix, not the add) is churn.
|
||||
if (!isAdd && !isFix && !touchedSource && !hasSymptom) {
|
||||
weight += WEIGHTS.testOnlyChurn;
|
||||
signals.push("test-only-churn");
|
||||
}
|
||||
|
||||
let category = "neutral";
|
||||
if (weight > 0) category = "positive";
|
||||
else if (weight < 0) category = "negative";
|
||||
|
||||
return { category, weight, signals };
|
||||
}
|
||||
|
||||
/**
|
||||
* Aggregate per-commit classifications into a file-level value score + recommendation.
|
||||
*
|
||||
* @param {object} input
|
||||
* @param {Array<{category:string, weight:number, signals:string[]}>} input.commits classified commits
|
||||
* @param {number|null} [input.durationMs] per-file test duration from scripts/test-timings.json
|
||||
* @param {number} [input.testCount] number of it()/test() cases in the file
|
||||
* @param {boolean} [input.quarantined] does the file appear in the quarantine ledger (current or historical)?
|
||||
*/
|
||||
export function scoreFile({ commits = [], durationMs = null, testCount = 0, quarantined = false }) {
|
||||
let valueScore = 0;
|
||||
let positiveCount = 0;
|
||||
let negativeCount = 0;
|
||||
let neutralCount = 0;
|
||||
let testOnlyCount = 0;
|
||||
const signalTally = {};
|
||||
|
||||
for (const c of commits) {
|
||||
valueScore += c.weight;
|
||||
if (c.category === "positive") positiveCount += 1;
|
||||
else if (c.category === "negative") negativeCount += 1;
|
||||
else neutralCount += 1;
|
||||
for (const s of c.signals ?? []) {
|
||||
signalTally[s] = (signalTally[s] ?? 0) + 1;
|
||||
if (s === "test-only-churn") testOnlyCount += 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (quarantined) {
|
||||
valueScore += WEIGHTS.quarantineLedger;
|
||||
signalTally["quarantine-ledger"] = (signalTally["quarantine-ledger"] ?? 0) + 1;
|
||||
}
|
||||
|
||||
const timesTouched = commits.length;
|
||||
const testOnlyRatio = timesTouched > 0 ? testOnlyCount / timesTouched : 0;
|
||||
|
||||
let recommendation = "keep";
|
||||
if (quarantined || valueScore <= THRESHOLDS.delete) recommendation = "delete";
|
||||
else if (valueScore <= THRESHOLDS.review) recommendation = "review";
|
||||
|
||||
// Deletion priority surfaces SLOW + LOW-VALUE files first (most time saved per unit of
|
||||
// lost signal). Valuable files divide their cost down; non-positive files keep full cost
|
||||
// and get a small extra nudge proportional to how negative they are.
|
||||
const cost = typeof durationMs === "number" && Number.isFinite(durationMs) ? durationMs : 0;
|
||||
const negativityBoost = 1 + Math.max(0, -valueScore) * 0.05;
|
||||
const deletionPriority = (cost / (1 + Math.max(0, valueScore))) * negativityBoost;
|
||||
|
||||
// Under the deletion-ratchet (AGENTS.md), a file is a "safe delete" candidate when it
|
||||
// carries zero positive evidence and is either quarantined or pure churn.
|
||||
const safeDelete = positiveCount === 0 && (quarantined || valueScore <= 0);
|
||||
|
||||
return {
|
||||
valueScore: Number(valueScore.toFixed(2)),
|
||||
positiveCount,
|
||||
negativeCount,
|
||||
neutralCount,
|
||||
timesTouched,
|
||||
testOnlyCount,
|
||||
testOnlyRatio: Number(testOnlyRatio.toFixed(3)),
|
||||
signalTally,
|
||||
durationMs: cost || null,
|
||||
testCount,
|
||||
quarantined: Boolean(quarantined),
|
||||
recommendation,
|
||||
safeDelete,
|
||||
deletionPriority: Number(deletionPriority.toFixed(1)),
|
||||
};
|
||||
}
|
||||
429
scripts/test-value-audit.mjs
Normal file
429
scripts/test-value-audit.mjs
Normal file
@@ -0,0 +1,429 @@
|
||||
#!/usr/bin/env node
|
||||
/*
|
||||
FNXC:TestInfrastructure 2026-06-25-00:00:
|
||||
Test Value Audit. Builds a heuristic "value score" for every test file under
|
||||
- packages/(asterisk)/src/(asterisk)(asterisk)/__tests__/(asterisk)(asterisk)
|
||||
- packages/dashboard/app/(asterisk)(asterisk)/__tests__/(asterisk)(asterisk)
|
||||
from git history, joins it with per-file durations (scripts/test-timings.json), and ranks
|
||||
the SLOW + LOW-VALUE files first as deletion candidates. Emits docs/test-value-audit.json
|
||||
(machine artifact) and docs/test-value-audit.md (human report). This is an EVIDENCE BASE
|
||||
for a human/follow-up deletion decision — the script never deletes tests.
|
||||
|
||||
Requirement context: AGENTS.md "Do Not Add Slow Tests" (FN-5048) + the quarantine deletion
|
||||
ratchet. We need to know which slow tests are also low-signal so they can be cut without
|
||||
losing real regression coverage.
|
||||
|
||||
HEURISTIC, not ground truth. Caveats live in the generated report's Methodology section.
|
||||
|
||||
Usage:
|
||||
node scripts/test-value-audit.mjs # full run, writes artifacts, prints top 15
|
||||
node scripts/test-value-audit.mjs --top 30 # change how many rows are printed/written to md
|
||||
node scripts/test-value-audit.mjs --json-only # skip markdown
|
||||
*/
|
||||
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFileSync, writeFileSync, existsSync } from "node:fs";
|
||||
import { resolve, relative } from "node:path";
|
||||
import { fileURLToPath, URL } from "node:url";
|
||||
|
||||
import { classifyCommit, scoreFile, WEIGHTS, NEGATIVE_KEYWORDS } from "./lib/test-value-audit-lib.mjs";
|
||||
|
||||
const repoRoot = fileURLToPath(new URL("..", import.meta.url));
|
||||
|
||||
function arg(name, fallback) {
|
||||
const i = process.argv.indexOf(name);
|
||||
if (i === -1) return fallback;
|
||||
const v = process.argv[i + 1];
|
||||
return v && !v.startsWith("--") ? v : true;
|
||||
}
|
||||
|
||||
const TOP_N = Number(arg("--top", 40)) || 40;
|
||||
const JSON_ONLY = process.argv.includes("--json-only");
|
||||
|
||||
function git(args) {
|
||||
return execFileSync("git", args, { cwd: repoRoot, encoding: "utf8", maxBuffer: 512 * 1024 * 1024 });
|
||||
}
|
||||
|
||||
// --- Target test files -------------------------------------------------------
|
||||
// Globs (POSIX-style, repo-relative) for the two audited surfaces.
|
||||
function isTargetTestFile(path) {
|
||||
if (!/\.(test|spec)\.(ts|tsx|mts|cts|js|mjs|cjs)$/.test(path)) return false;
|
||||
if (!path.includes("/__tests__/")) return false;
|
||||
if (/^packages\/[^/]+\/src\//.test(path)) return true;
|
||||
if (path.startsWith("packages/dashboard/app/")) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
function packageRootOf(path) {
|
||||
const m = path.match(/^(packages\/[^/]+)\//);
|
||||
return m ? m[1] : null;
|
||||
}
|
||||
|
||||
// A non-test source file in the same package (used for "test + source changed together").
|
||||
function isSourceFileInPackage(path, pkgRoot) {
|
||||
if (!pkgRoot || !path.startsWith(pkgRoot + "/")) return false;
|
||||
if (!/\.(ts|tsx|mts|cts|js|mjs|cjs)$/.test(path)) return false;
|
||||
if (/\.(test|spec)\./.test(path)) return false;
|
||||
if (path.includes("/__tests__/")) return false;
|
||||
if (path.includes("/__test-utils__/") || path.includes("/__mocks__/")) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
// --- Timings -----------------------------------------------------------------
|
||||
function loadTimings() {
|
||||
const p = resolve(repoRoot, "scripts/test-timings.json");
|
||||
const map = new Map();
|
||||
if (!existsSync(p)) return map;
|
||||
const data = JSON.parse(readFileSync(p, "utf8"));
|
||||
for (const pkg of Object.values(data.packages ?? {})) {
|
||||
for (const [file, ms] of Object.entries(pkg.files ?? {})) {
|
||||
// Keep the max if a path appears twice across packages.
|
||||
map.set(file, Math.max(map.get(file) ?? 0, Number(ms) || 0));
|
||||
}
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
// --- Quarantine ledger (current + historical) --------------------------------
|
||||
function loadQuarantinePaths() {
|
||||
const set = new Set();
|
||||
const ledger = "scripts/lib/test-quarantine.json";
|
||||
try {
|
||||
const current = JSON.parse(readFileSync(resolve(repoRoot, ledger), "utf8"));
|
||||
for (const e of current.entries ?? []) if (e.file) set.add(e.file);
|
||||
} catch {
|
||||
/* ledger may not exist */
|
||||
}
|
||||
// Historical entries: scan every past revision of the ledger for "file": "<path>".
|
||||
try {
|
||||
const blob = git(["log", "-p", "--format=", "--", ledger]);
|
||||
for (const m of blob.matchAll(/"file"\s*:\s*"([^"]+)"/g)) set.add(m[1]);
|
||||
} catch {
|
||||
/* no history */
|
||||
}
|
||||
return set;
|
||||
}
|
||||
|
||||
// --- Whole-history parse -----------------------------------------------------
|
||||
// One git invocation yields every commit's subject/body/time + name-status file list.
|
||||
// Field sep \x1f, body terminator \x02, commit start \x01.
|
||||
function loadHistory() {
|
||||
const SEP = "\x1f";
|
||||
const BODY_END = "\x02";
|
||||
const COMMIT_START = "\x01";
|
||||
const raw = git([
|
||||
"log",
|
||||
"--no-color",
|
||||
"--name-status",
|
||||
`--format=${COMMIT_START}%H${SEP}%ct${SEP}%s${SEP}%b${BODY_END}`,
|
||||
]);
|
||||
|
||||
const commits = [];
|
||||
// newPath -> oldPath rename map (last writer wins; good enough for linear chains).
|
||||
const renameMap = new Map();
|
||||
// path -> array of commit indices that touched it (as add/modify/rename target).
|
||||
const byPath = new Map();
|
||||
|
||||
const chunks = raw.split(COMMIT_START);
|
||||
for (const chunk of chunks) {
|
||||
if (!chunk.trim()) continue;
|
||||
const bodyEnd = chunk.indexOf(BODY_END);
|
||||
if (bodyEnd === -1) continue;
|
||||
const head = chunk.slice(0, bodyEnd);
|
||||
const [sha, ct, subject, ...bodyParts] = head.split(SEP);
|
||||
const body = bodyParts.join(SEP);
|
||||
const nameStatusBlock = chunk.slice(bodyEnd + 1);
|
||||
|
||||
const files = [];
|
||||
for (const line of nameStatusBlock.split("\n")) {
|
||||
const t = line.trim();
|
||||
if (!t) continue;
|
||||
const cols = t.split("\t");
|
||||
const statusRaw = cols[0];
|
||||
if (!statusRaw) continue;
|
||||
const status = statusRaw[0]; // A/M/D/R/C/T
|
||||
if ((status === "R" || status === "C") && cols.length >= 3) {
|
||||
const oldPath = cols[1];
|
||||
const newPath = cols[2];
|
||||
files.push({ status, path: newPath, oldPath });
|
||||
renameMap.set(newPath, oldPath);
|
||||
} else if (cols.length >= 2) {
|
||||
files.push({ status, path: cols[1] });
|
||||
}
|
||||
}
|
||||
|
||||
const idx = commits.length;
|
||||
commits.push({
|
||||
sha,
|
||||
time: Number(ct) * 1000,
|
||||
subject: subject ?? "",
|
||||
body: body ?? "",
|
||||
files,
|
||||
});
|
||||
for (const f of files) {
|
||||
if (f.status === "D") continue;
|
||||
if (!byPath.has(f.path)) byPath.set(f.path, []);
|
||||
byPath.get(f.path).push(idx);
|
||||
}
|
||||
}
|
||||
|
||||
return { commits, renameMap, byPath };
|
||||
}
|
||||
|
||||
// Follow renames backward to collect every historical path this file lived at.
|
||||
function historicalPaths(currentPath, renameMap) {
|
||||
const paths = new Set([currentPath]);
|
||||
let p = currentPath;
|
||||
let guard = 0;
|
||||
while (renameMap.has(p) && guard < 100) {
|
||||
p = renameMap.get(p);
|
||||
if (paths.has(p)) break;
|
||||
paths.add(p);
|
||||
guard += 1;
|
||||
}
|
||||
return paths;
|
||||
}
|
||||
|
||||
function countTestCases(absPath) {
|
||||
try {
|
||||
const src = readFileSync(absPath, "utf8");
|
||||
let n = 0;
|
||||
for (const line of src.split("\n")) {
|
||||
if (/^\s*(it|test)\s*(\.\w+)?\s*\(/.test(line)) n += 1;
|
||||
}
|
||||
return n;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
function fmtMs(ms) {
|
||||
if (ms == null) return "n/a";
|
||||
if (ms >= 1000) return `${(ms / 1000).toFixed(1)}s`;
|
||||
return `${ms}ms`;
|
||||
}
|
||||
|
||||
// --- Main --------------------------------------------------------------------
|
||||
function main() {
|
||||
const started = Date.now();
|
||||
const timings = loadTimings();
|
||||
const quarantinePaths = loadQuarantinePaths();
|
||||
const { commits, renameMap, byPath } = loadHistory();
|
||||
|
||||
const tracked = git(["ls-files"]).split("\n").filter(Boolean);
|
||||
const testFiles = tracked.filter(isTargetTestFile).sort();
|
||||
|
||||
const rows = [];
|
||||
for (const file of testFiles) {
|
||||
const pkgRoot = packageRootOf(file) ?? "packages/dashboard";
|
||||
const paths = historicalPaths(file, renameMap);
|
||||
|
||||
// Gather candidate commit indices across all historical paths, dedup.
|
||||
const idxSet = new Set();
|
||||
for (const p of paths) for (const i of byPath.get(p) ?? []) idxSet.add(i);
|
||||
|
||||
const classified = [];
|
||||
const evidence = [];
|
||||
for (const i of [...idxSet].sort((a, b) => b - a)) {
|
||||
const c = commits[i];
|
||||
const fileEntry = c.files.find((f) => paths.has(f.path) && f.status !== "D");
|
||||
if (!fileEntry) continue;
|
||||
const status = fileEntry.status;
|
||||
const touchedSource = c.files.some(
|
||||
(f) => f.status !== "D" && isSourceFileInPackage(f.path, pkgRoot),
|
||||
);
|
||||
const cls = classifyCommit({ subject: c.subject, body: c.body, status, touchedSource });
|
||||
classified.push(cls);
|
||||
evidence.push({
|
||||
sha: c.sha.slice(0, 9),
|
||||
time: c.time,
|
||||
subject: c.subject,
|
||||
status,
|
||||
touchedSource,
|
||||
category: cls.category,
|
||||
weight: cls.weight,
|
||||
signals: cls.signals,
|
||||
});
|
||||
}
|
||||
|
||||
const durationMs = timings.has(file) ? timings.get(file) : null;
|
||||
const testCount = countTestCases(resolve(repoRoot, file));
|
||||
const quarantined = [...paths].some((p) => quarantinePaths.has(p));
|
||||
|
||||
const score = scoreFile({ commits: classified, durationMs, testCount, quarantined });
|
||||
|
||||
const times = evidence.map((e) => e.time).filter(Boolean);
|
||||
const firstSeen = times.length ? Math.min(...times) : null;
|
||||
const lastSeen = times.length ? Math.max(...times) : null;
|
||||
const ageDays = firstSeen ? Math.round((Date.now() - firstSeen) / 86_400_000) : null;
|
||||
|
||||
rows.push({
|
||||
file,
|
||||
package: pkgRoot,
|
||||
...score,
|
||||
ageDays,
|
||||
lastTouched: lastSeen ? new Date(lastSeen).toISOString().slice(0, 10) : null,
|
||||
evidence: evidence.slice(0, 6), // most-recent commits as the "why"
|
||||
});
|
||||
}
|
||||
|
||||
// Rank: slow + low-value first.
|
||||
rows.sort((a, b) => b.deletionPriority - a.deletionPriority || a.valueScore - b.valueScore);
|
||||
|
||||
const summary = {
|
||||
total: rows.length,
|
||||
deleteCandidates: rows.filter((r) => r.recommendation === "delete").length,
|
||||
reviewCandidates: rows.filter((r) => r.recommendation === "review").length,
|
||||
keep: rows.filter((r) => r.recommendation === "keep").length,
|
||||
safeDeletes: rows.filter((r) => r.safeDelete).length,
|
||||
withTiming: rows.filter((r) => r.durationMs != null).length,
|
||||
quarantined: rows.filter((r) => r.quarantined).length,
|
||||
};
|
||||
|
||||
const artifact = {
|
||||
generatedAt: new Date().toISOString(),
|
||||
heuristic: true,
|
||||
note: "HEURISTIC, not ground truth. Do not auto-delete from this file. See docs/test-value-audit.md methodology + caveats.",
|
||||
weights: WEIGHTS,
|
||||
negativeKeywords: NEGATIVE_KEYWORDS,
|
||||
timingsCapturedFrom: "scripts/test-timings.json",
|
||||
summary,
|
||||
rows,
|
||||
};
|
||||
|
||||
const jsonPath = resolve(repoRoot, "docs/test-value-audit.json");
|
||||
writeFileSync(jsonPath, JSON.stringify(artifact, null, 2) + "\n");
|
||||
|
||||
if (!JSON_ONLY) writeMarkdown(rows, summary);
|
||||
|
||||
// Console: top 15 rows.
|
||||
const top = rows.slice(0, 15);
|
||||
const tbl = top.map((r, i) => ({
|
||||
"#": i + 1,
|
||||
file: r.file.replace(/^packages\//, "").length > 58 ? "…" + r.file.slice(-57) : r.file.replace(/^packages\//, ""),
|
||||
tests: r.testCount,
|
||||
dur: fmtMs(r.durationMs),
|
||||
score: r.valueScore,
|
||||
rec: r.recommendation,
|
||||
}));
|
||||
console.log(`\nTest Value Audit — ${rows.length} files analyzed in ${((Date.now() - started) / 1000).toFixed(1)}s`);
|
||||
console.log(
|
||||
`delete=${summary.deleteCandidates} review=${summary.reviewCandidates} keep=${summary.keep} ` +
|
||||
`safeDelete=${summary.safeDeletes} withTiming=${summary.withTiming}\n`,
|
||||
);
|
||||
console.log("Top 15 deletion candidates (slow + low value first):");
|
||||
console.table(tbl);
|
||||
console.log(`\nArtifacts:\n ${relative(repoRoot, jsonPath)}`);
|
||||
if (!JSON_ONLY) console.log(` docs/test-value-audit.md`);
|
||||
}
|
||||
|
||||
function recEmoji(rec) {
|
||||
return rec === "delete" ? "delete" : rec === "review" ? "review" : "keep";
|
||||
}
|
||||
|
||||
function writeMarkdown(rows, summary) {
|
||||
const top = rows.slice(0, TOP_N);
|
||||
const L = [];
|
||||
L.push("# Test Value Audit");
|
||||
L.push("");
|
||||
L.push(`> Generated by \`scripts/test-value-audit.mjs\` on ${new Date().toISOString()}.`);
|
||||
L.push(">");
|
||||
L.push("> **This is a HEURISTIC, not ground truth.** It is an evidence base for a human");
|
||||
L.push("> deletion decision. The script does **not** delete any tests. See Methodology + Caveats.");
|
||||
L.push("");
|
||||
L.push("## Summary");
|
||||
L.push("");
|
||||
L.push(`- Files analyzed: **${summary.total}**`);
|
||||
L.push(`- With timing data: **${summary.withTiming}**`);
|
||||
L.push(`- \`delete\` candidates (valueScore ≤ 0 or quarantined): **${summary.deleteCandidates}**`);
|
||||
L.push(`- \`review\` candidates (0 < valueScore ≤ 3): **${summary.reviewCandidates}**`);
|
||||
L.push(`- \`keep\` (valueScore > 3): **${summary.keep}**`);
|
||||
L.push(`- "Safe delete" under the deletion-ratchet (zero positive evidence + churn/quarantine): **${summary.safeDeletes}**`);
|
||||
L.push(`- In quarantine ledger (current or historical): **${summary.quarantined}**`);
|
||||
L.push("");
|
||||
L.push("## Methodology");
|
||||
L.push("");
|
||||
L.push("For every audited test file we run a single whole-history `git log --name-status`");
|
||||
L.push("pass and classify each commit that touched the file (renames followed backward):");
|
||||
L.push("");
|
||||
L.push("**Positive signal** (encoded/caught a real bug):");
|
||||
L.push("- `fix(...)`/`fix:` subject **with** a sibling source change in the same package (+3)");
|
||||
L.push("- `fix(...)`/`fix:` subject alone (+2)");
|
||||
L.push("- `## Symptom Verification` regression marker in the commit body (+3, FN-5893)");
|
||||
L.push("- test file first **added** together with source (+2), or added alone (+1)");
|
||||
L.push("- a plain test+source co-change modify (+1.5)");
|
||||
L.push("");
|
||||
L.push("**Negative signal** (low value / churn):");
|
||||
L.push("- subject/body mentions " + NEGATIVE_KEYWORDS.map((k) => `\`${k}\``).join(", ") + " (−3)");
|
||||
L.push("- a modify that touched **only** the test file, no source (−1)");
|
||||
L.push("- appears in `scripts/lib/test-quarantine.json` history (−5)");
|
||||
L.push("");
|
||||
L.push("The per-file **valueScore** is the sum of commit weights (plus the quarantine penalty).");
|
||||
L.push("**deletionPriority** = `durationMs / (1 + max(0, valueScore))` with a small boost for");
|
||||
L.push("net-negative files — so the ranking surfaces **slow AND low-value** files first");
|
||||
L.push("(most CI time saved per unit of lost signal). Recommendation: `delete` (≤0), `review`");
|
||||
L.push("(≤3), else `keep`.");
|
||||
L.push("");
|
||||
L.push("## Caveats (read before deleting anything)");
|
||||
L.push("");
|
||||
L.push("- **Heuristic, not truth.** A quiet, never-modified test can still be load-bearing;");
|
||||
L.push(" a high-churn test can still be valuable. Use this to *prioritize human review*.");
|
||||
L.push("- **`git log --follow` / rename limits.** Renames are followed only through linear");
|
||||
L.push(" `R`/`C` name-status chains; squash-merges collapse multi-commit history into one");
|
||||
L.push(" subject, so per-commit signal is lost for squashed work (this repo defaults to");
|
||||
L.push(" squash merges — a major reason to treat scores as lower bounds on value).");
|
||||
L.push("- **Subjects lie.** `fix(...)` is trusted as a positive even if the test was unrelated;");
|
||||
L.push(" conversely a real bug fixed under a `feat(...)`/`FN-` subject without source co-change");
|
||||
L.push(" may be undercounted.");
|
||||
L.push("- **Timing is a snapshot** from `scripts/test-timings.json`; files with no entry show");
|
||||
L.push(" `n/a` duration and get deletionPriority 0 (cost unknown, not necessarily cheap).");
|
||||
L.push("- **Not a green light.** Deleting a gate test still requires the gate-eviction process");
|
||||
L.push(" (AGENTS.md). \"Safe delete\" only flags files that meet the ratchet's churn/quarantine bar.");
|
||||
L.push("");
|
||||
L.push(`## Top ${top.length} deletion candidates`);
|
||||
L.push("");
|
||||
L.push("| # | File | Tests | Duration | Value | Priority | Rec | Why (recent commits) |");
|
||||
L.push("|---|------|------:|---------:|------:|---------:|-----|----------------------|");
|
||||
top.forEach((r, i) => {
|
||||
const why = r.evidence
|
||||
.slice(0, 3)
|
||||
.map((e) => {
|
||||
const subj = (e.subject || "").replace(/\|/g, "\\|").slice(0, 48);
|
||||
return `\`${e.sha}\` ${subj} _(${e.signals.join(",") || "neutral"})_`;
|
||||
})
|
||||
.join("<br>");
|
||||
const flags = [];
|
||||
if (r.quarantined) flags.push("quarantined");
|
||||
if (r.safeDelete) flags.push("safe-delete");
|
||||
const rec = recEmoji(r.recommendation) + (flags.length ? ` (${flags.join(", ")})` : "");
|
||||
L.push(
|
||||
`| ${i + 1} | \`${r.file}\` | ${r.testCount} | ${fmtMs(r.durationMs)} | ${r.valueScore} | ${r.deletionPriority} | ${rec} | ${why || "_no git evidence_"} |`,
|
||||
);
|
||||
});
|
||||
L.push("");
|
||||
L.push("### One-line recommendations");
|
||||
L.push("");
|
||||
top.forEach((r, i) => {
|
||||
let rationale;
|
||||
if (r.recommendation === "delete") {
|
||||
rationale = r.quarantined
|
||||
? "in quarantine ledger — delete per ratchet once expiry passes"
|
||||
: r.positiveCount === 0
|
||||
? "no positive (bug-encoding) signal in history; pure churn — strong delete candidate"
|
||||
: "net-negative value (churn outweighs signal) — delete candidate";
|
||||
} else if (r.recommendation === "review") {
|
||||
rationale = "thin positive signal; confirm it asserts a real invariant before trimming";
|
||||
} else {
|
||||
rationale = "carries real bug-fix / source-coupled signal — keep";
|
||||
}
|
||||
if (r.durationMs != null && r.durationMs >= 1000 && r.recommendation !== "keep") {
|
||||
rationale += ` (slow: ${fmtMs(r.durationMs)} — high CI time payoff)`;
|
||||
}
|
||||
L.push(`${i + 1}. \`${r.file}\` — **${r.recommendation}**: ${rationale}`);
|
||||
});
|
||||
L.push("");
|
||||
|
||||
writeFileSync(resolve(repoRoot, "docs/test-value-audit.md"), L.join("\n") + "\n");
|
||||
}
|
||||
|
||||
main();
|
||||
Reference in New Issue
Block a user