chore: test value audit (deletion-candidate evidence base) (#1776)

## What

A **test value audit**: a heuristic that scores every test file under
`packages/*/src/**/__tests__/**` and
`packages/dashboard/app/**/__tests__/**` by how much real regression
signal it has encoded over its git history, then ranks **slow AND
low-value** files first as deletion candidates. This is the **evidence
base** for a human/follow-up deletion decision — **the script deletes
nothing**.

Motivated by AGENTS.md "Do Not Add Slow Tests" (FN-5048) + the
quarantine deletion ratchet: we want data on which slow tests are also
low-signal so they can be cut without losing coverage.

## How it scores (HEURISTIC, not ground truth)

Single whole-history `git log --name-status` pass; per-commit
classification (renames followed backward):

- **Positive** — `fix(...)`/`fix:` with sibling source change (+3), fix
alone (+2), `## Symptom Verification` regression marker (+3, FN-5893),
test added-with-source (+2)/alone (+1), plain test+source co-change
(+1.5).
- **Negative** — subject/body churn keywords
`flake/flaky/deflake/quarantine/stabiliz/appease/timeout/retry` (−3),
test-only modify with no source (−1), quarantine-ledger membership
current+historical (−5).

`valueScore` = sum of weights. `deletionPriority = durationMs / (1 +
max(0, valueScore))` (+ small net-negative boost), joined with
`scripts/test-timings.json` so slow + low-value surfaces first.
Recommendation: `delete` (≤0) / `review` (≤3) / `keep` (>3). A
`safeDelete` flag marks files meeting the ratchet's churn/quarantine
bar.

## Deliverables

- `scripts/test-value-audit.mjs` — runner (git IO + report generation)
- `scripts/lib/test-value-audit-lib.mjs` — pure, unit-tested scoring
logic
- `scripts/__tests__/test-value-audit.test.mjs` — 15 synthetic-record
unit tests
- `docs/test-value-audit.json` + `docs/test-value-audit.md` — generated
artifacts (top 40 + methodology + honest caveats: heuristic limits,
git-follow/squash-merge blind spots, lying subjects, timing snapshot)

## Verification

- `node scripts/test-value-audit.mjs` runs end-to-end (~1s), 2051 files
analyzed, writes both artifacts.
- `node --test scripts/__tests__/test-value-audit.test.mjs` → 15/15
pass.
- `eslint` clean on all three new source files.

No changeset (scripts + docs only; `@runfusion/fusion` runtime
unaffected).

🤖 Generated with [Claude Code](https://claude.com/claude-code)


<!-- stage-review-badge-begin -->

---

<a href="https://stagereview.app/Runfusion/Fusion/pull/1776">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://stagereview.app/assets/gh-open-in-stage-dark.svg">
<img src="https://stagereview.app/assets/gh-open-in-stage-light.svg"
alt="Open in Stage">
  </picture>
</a>

<!-- stage-review-badge-end -->
This commit is contained in:
gsxdsm
2026-06-25 17:23:03 -07:00
committed by GitHub
5 changed files with 130673 additions and 0 deletions

129726
docs/test-value-audit.json Normal file

File diff suppressed because it is too large Load Diff

144
docs/test-value-audit.md Normal file
View File

@@ -0,0 +1,144 @@
# Test Value Audit
> Generated by `scripts/test-value-audit.mjs` on 2026-06-26T00:03:00.110Z.
>
> **This is a HEURISTIC, not ground truth.** It is an evidence base for a human
> deletion decision. The script does **not** delete any tests. See Methodology + Caveats.
## Summary
- Files analyzed: **2051**
- With timing data: **1979**
- `delete` candidates (valueScore ≤ 0 or quarantined): **288**
- `review` candidates (0 < valueScore ≤ 3): **839**
- `keep` (valueScore > 3): **924**
- "Safe delete" under the deletion-ratchet (zero positive evidence + churn/quarantine): **54**
- In quarantine ledger (current or historical): **59**
## Methodology
For every audited test file we run a single whole-history `git log --name-status`
pass and classify each commit that touched the file (renames followed backward):
**Positive signal** (encoded/caught a real bug):
- `fix(...)`/`fix:` subject **with** a sibling source change in the same package (+3)
- `fix(...)`/`fix:` subject alone (+2)
- `## Symptom Verification` regression marker in the commit body (+3, FN-5893)
- test file first **added** together with source (+2), or added alone (+1)
- a plain test+source co-change modify (+1.5)
**Negative signal** (low value / churn):
- subject/body mentions `flake`, `flaky`, `deflake`, `quarantine`, `stabiliz`, `appease`, `timeout`, `retry` (−3)
- a modify that touched **only** the test file, no source (−1)
- appears in `scripts/lib/test-quarantine.json` history (−5)
The per-file **valueScore** is the sum of commit weights (plus the quarantine penalty).
**deletionPriority** = `durationMs / (1 + max(0, valueScore))` with a small boost for
net-negative files — so the ranking surfaces **slow AND low-value** files first
(most CI time saved per unit of lost signal). Recommendation: `delete` (≤0), `review`
(≤3), else `keep`.
## Caveats (read before deleting anything)
- **Heuristic, not truth.** A quiet, never-modified test can still be load-bearing;
a high-churn test can still be valuable. Use this to *prioritize human review*.
- **`git log --follow` / rename limits.** Renames are followed only through linear
`R`/`C` name-status chains; squash-merges collapse multi-commit history into one
subject, so per-commit signal is lost for squashed work (this repo defaults to
squash merges — a major reason to treat scores as lower bounds on value).
- **Subjects lie.** `fix(...)` is trusted as a positive even if the test was unrelated;
conversely a real bug fixed under a `feat(...)`/`FN-` subject without source co-change
may be undercounted.
- **Timing is a snapshot** from `scripts/test-timings.json`; files with no entry show
`n/a` duration and get deletionPriority 0 (cost unknown, not necessarily cheap).
- **Not a green light.** Deleting a gate test still requires the gate-eviction process
(AGENTS.md). "Safe delete" only flags files that meet the ratchet's churn/quarantine bar.
## Top 40 deletion candidates
| # | File | Tests | Duration | Value | Priority | Rec | Why (recent commits) |
|---|------|------:|---------:|------:|---------:|-----|----------------------|
| 1 | `packages/engine/src/__tests__/workspace-merger-idempotency.test.ts` | 8 | 12.7s | -1 | 13335 | delete (safe-delete) | `754434632` feat(workspace): Phase C U2 — per-repo landed pr _(churn-keyword,added-with-source)_<br>`627bdcfb0` fix(review): Phase C merge-loop hardening — doub _(churn-keyword,fix+source)_<br>`3a7123762` fix(review): address PR #1717 Phase C merge-loop _(churn-keyword,fix+source)_ |
| 2 | `packages/dashboard/src/__tests__/github-tracking-delete.test.ts` | 9 | 8.7s | -7.5 | 11962.5 | delete | `d0c9e47f6` feat(FN-4253): close linked GitHub issues on tas _(added-with-source)_<br>`9dfefc551` feat(FN-4941): close or delete GitHub issues whe _(test+source)_<br>`46fb3f0ef` feat(FN-5305): fix async synchronization in gith _(test-only-churn)_ |
| 3 | `packages/engine/src/__tests__/reliability-interactions/merge-runner-spawn-enoent-prevention.test.ts` | 5 | 7.9s | -2 | 8690 | delete (safe-delete) | `7763ba5fa` FN-6278: preflight reusable merge worktree cwd _(churn-keyword,added-with-source)_<br>`a768d36da` FN-6817: root reliability fixtures under worker _(test-only-churn)_ |
| 4 | `packages/dashboard/src/__tests__/session-reconnect.test.ts` | 4 | 6.1s | -6.5 | 8082.5 | delete (quarantined) | `bf141c47d` test(FN-1156): add AI session lifecycle and reco _(added)_<br>`84ed84313` feat(FN-2123): merge fusion/fn-2123 _(test-only-churn)_<br>`6714f7654` refactor(FN-2161): standardize on createFnAgent _(test+source)_ |
| 5 | `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` | 7 | 4.6s | -12 | 7360 | delete (quarantined) | `602a7da51` feat(FN-3865): complete Steps 3-6 — recover alre _(added-with-source)_<br>`39e0b3c6a` fix(FN-3865): align merge recovery types and rea _(fix+source)_<br>`6fc804483` feat(FN-3940): expose github tracking controls i _(test-only-churn)_ |
| 6 | `packages/core/src/__tests__/run-audit.integration.test.ts` | 24 | 5.8s | -2 | 6380 | delete (quarantined) | `c2d0ac572` test(FN-1414): add run-audit integration tests f _(churn-keyword,added)_<br>`84ed84313` feat(FN-2123): merge fusion/fn-2123 _(test+source)_<br>`a269f4829` refactor: remove legacy kb compatibility _(test+source)_ |
| 7 | `packages/engine/src/__tests__/workspace-merger-lease.test.ts` | 4 | 5.8s | -1 | 6090 | delete (safe-delete) | `64e87f9a1` feat(workspace): Phase C U3 — per-repo land leas _(churn-keyword,added-with-source)_<br>`627bdcfb0` fix(review): Phase C merge-loop hardening — doub _(churn-keyword,fix+source)_ |
| 8 | `packages/engine/src/__tests__/workspace-merger.test.ts` | 7 | 8.0s | 0.5 | 5333.3 | review | `744ed098a` feat(workspace): Phase C U1 — per-repo merge loo _(churn-keyword,added-with-source)_<br>`754434632` feat(workspace): Phase C U2 — per-repo landed pr _(churn-keyword,test+source)_<br>`3a7123762` fix(review): address PR #1717 Phase C merge-loop _(churn-keyword,fix+source)_ |
| 9 | `packages/cli/src/__tests__/extension-task-tools.test.ts` | 4 | 3.8s | -7 | 5130 | delete (quarantined) | `47eaf089b` feat(FN-4904): complete Step 3 — align engine an _(added-with-source)_<br>`0dc4c9c6a` test(FN-4904): cover worktree-root lookup in ext _(test-only-churn)_<br>`b80b517a4` test(FN-4927): cover no-task fallback for task t _(test-only-churn)_ |
| 10 | `packages/engine/src/__tests__/worktree-db-hydrate.test.ts` | 12 | 7.1s | 0.5 | 4733.3 | review | `e59a740d0` feat(FN-3841): add worktree database hydration t _(added-with-source)_<br>`6acc51cf0` feat(FN-4039): recover worktree db scratch boots _(test+source)_<br>`93dc5a727` feat(FN-4083): add WAL enforcement, immediate wr _(test+source)_ |
| 11 | `packages/core/src/__tests__/activity-analytics.test.ts` | 25 | 4.6s | 0 | 4600 | delete (quarantined) | `53bb1d8f3` feat(analytics): U2 — core date-range aggregator _(added-with-source)_<br>`5bc8901f0` feat(command-center): U7 — SDLC funnel + through _(test+source)_<br>`f5bd86214` feat(monitor): U13 — monitor stage (deployments, _(test+source)_ |
| 12 | `packages/dashboard/src/routes/__tests__/board-workflows-route.test.ts` | 4 | 3.6s | -5.5 | 4590 | delete | `010976cee` test: promote/board-workflows routes, concurrent _(added-with-source)_<br>`f77aa073c` FN-6025: fix builtin coding auto-merge review fl _(test-only-churn)_<br>`edde74d56` FN-6146: harden dashboard test mocks and request _(churn-keyword,test+source)_ |
| 13 | `packages/dashboard/app/components/__tests__/TaskDetailModal.responsive-and-dependencies.test.tsx` | 44 | 6.6s | 0.5 | 4400 | review | `ed52c3dff` test(FN-3591): split TaskDetailModal coverage in _(added)_<br>`ecbf1f829` feat(FN-3276): add task review tab with metadata _(test+source)_<br>`309e3f91a` feat(FN-3897): add blocker fan-out hook, badge, _(test+source)_ |
| 14 | `packages/core/src/__tests__/todo-store.test.ts` | 17 | 3.8s | -3 | 4370 | delete (quarantined) | `6f4b42a89` feat(FN-2575): merge fusion/fn-2575 _(added-with-source)_ |
| 15 | `packages/engine/src/__tests__/reliability-interactions/soft-blocker-auto-finalize-interactions.real-git.test.ts` | 7 | 3.4s | -4 | 4080 | delete | `208753fb8` test(FN-4653): complete Step 1 — add merge-path _(added)_<br>`875b32266` test(FN-4653): complete Step 2 — cover mixed blo _(test-only-churn)_<br>`5548b6f4a` test(FN-4653): complete Step 3 — add scheduler s _(test-only-churn)_ |
| 16 | `packages/core/src/__tests__/move-task-characterization.test.ts` | 10 | 8.1s | 1 | 4050 | review | `4e1b0fab0` feat(core): workflow-resolved transitions behind _(added-with-source)_<br>`0b7549a53` FN-6126: enable workflow experimental flags by d _(test+source)_<br>`9a7881441` FN-6245: keep default auto-merge tasks on live s _(test+source)_ |
| 17 | `packages/engine/src/__tests__/pr-response-run.test.ts` | 23 | 11.6s | 2 | 3866.7 | review | `31d4b5335` feat(pr): security-hardened review-response run _(added-with-source)_ |
| 18 | `packages/engine/src/__tests__/merger-autostash-orphan-surface.test.ts` | 6 | 3.7s | 0 | 3700 | delete | `d604d090a` feat(FN-3863): add stash recovery dashboard surf _(added-with-source)_<br>`bcb4107a1` feat(FN-3932): add orphaned finalize-reset autos _(test+source)_<br>`1cf987208` feat(FN-4018): isolate split merger temp workspa _(test-only-churn)_ |
| 19 | `packages/engine/src/__tests__/merger-empty-cherry-pick-fallback.test.ts` | 2 | 3.0s | -1.5 | 3225 | delete | `653500959` test(FN-4475): add fallback empty cherry-pick re _(added)_<br>`ebd79d21b` feat(FN-4475): remove unstable merger empty cher _(churn-keyword,test-only-churn)_<br>`22d59b5f9` feat(FN-5279): add merge integration worktree fe _(test+source)_ |
| 20 | `packages/engine/src/__tests__/merger-ai-cleanup.test.ts` | 15 | 2.8s | -3 | 3220 | delete (quarantined) | `4032a35a2` FN-6188: harden AI merge temp worktree cleanup _(added-with-source)_<br>`d75f861f2` FN-6199: harden AI merge worktree cleanup _(test+source)_<br>`3e69d3990` FN-6244: protect active AI merge worktrees _(churn-keyword,test+source)_ |
| 21 | `packages/dashboard/src/__tests__/routes-secrets-sync.test.ts` | 17 | 2.7s | -3 | 3105 | delete | `cca96a760` test(FN-4913): complete Step 7 — add secrets syn _(added-with-source)_<br>`b5e674059` test(FN-4980): cover sync-export auth rejection _(test-only-churn)_<br>`47775e942` test(FN-4981): complete Step 2 — add push missin _(test-only-churn)_ |
| 22 | `packages/core/src/__tests__/store-concurrent-writes.test.ts` | 6 | 2.8s | -1.5 | 3010 | delete (quarantined) | `93dc5a727` feat(FN-4083): add WAL enforcement, immediate wr _(added-with-source)_<br>`67c680c8d` fix(FN-4122): use unique tmp filename for task.j _(fix+source)_<br>`a38752c6b` FN-6486: rescue quarantined flaky tests _(churn-keyword,test+source)_ |
| 23 | `packages/core/src/__tests__/store-archive-search.test.ts` | 63 | 3.0s | 0 | 3000 | delete | `ac6aaca2c` feat(FN-3982): split monolithic store.test.ts in _(added)_<br>`139e25fcf` feat(FN-4837): complete Step 2 — migrate store t _(test-only-churn)_<br>`294209f64` FN-5943: maintain tasks FTS5 indexes automatical _(test+source)_ |
| 24 | `packages/engine/src/__tests__/reliability-interactions/in-review-branch-rebind.test.ts` | 10 | 7.4s | 1.5 | 2960 | review | `5c3096c7d` feat(FN-5083): complete Steps 5-7 rebind UI and _(added)_<br>`b3ac0beaa` test(FN-5083): tighten rebind reliability and qu _(test-only-churn)_<br>`3566cf8a1` FN-6695: block unsafe in-review branch rebinds _(test+source)_ |
| 25 | `packages/core/src/__tests__/store-activity.test.ts` | 49 | 5.9s | 1 | 2950 | review | `36e130937` refactor(FN-4021): split core store tests into d _(added)_<br>`287ebaf10` feat(FN-4053): unify task ID allocation with sto _(test+source)_<br>`a0f2bc530` feat(FN-4429): complete Step 1 — plumb moveSourc _(test+source)_ |
| 26 | `packages/dashboard/app/components/__tests__/AgentDetailView.mobile-scroll.test.tsx` | 4 | 2.5s | -3 | 2875 | delete | `3a4397215` feat(FN-4231): restore mobile scroll in AgentDet _(added)_<br>`d68fe3e9f` FN-6209: fix tablet tab overflow in agent detail _(test-only-churn)_<br>`b7c23c09c` FN-6450: enable touch scrolling for agent detail _(test-only-churn)_ |
| 27 | `packages/dashboard/src/__tests__/insights-routes.test.ts` | 24 | 26.5s | 8.5 | 2789.5 | keep | `629398483` fix(FN-1909): reorder insights routes to prevent _(fix+source,added-with-source)_<br>`8de7feeb1` feat(FN-1974): merge fusion/fn-1974 _(test+source)_<br>`6714f7654` refactor(FN-2161): standardize on createFnAgent _(test+source)_ |
| 28 | `packages/engine/src/__tests__/branch-conflicts-ghost-references.test.ts` | 5 | 2.5s | -2 | 2750 | delete | `663e5d2cd` feat(FN-4508): complete Step 1 — harden inspectB _(added-with-source)_<br>`5cd0e6665` feat(FN-4839): harden real-git test timeouts acr _(churn-keyword,test-only-churn)_ |
| 29 | `packages/core/src/__tests__/soft-delete-lineage-children.test.ts` | 11 | 2.6s | 0 | 2600 | delete | `e0c0745c0` test(FN-5129): add lineage soft-delete and archi _(added-with-source)_<br>`2c7ae1c1a` feat(FN-5131): add lineage-unlink flag to triage _(test-only-churn)_<br>`4955ccf6e` feat(FN-5132): align soft-delete-lineage-childre _(test-only-churn)_ |
| 30 | `packages/core/src/__tests__/store-handoff-to-review.test.ts` | 8 | 2.6s | 0 | 2600 | delete (quarantined) | `93b11c6c0` feat(FN-5241): add atomic in-review handoff seam _(added-with-source)_<br>`2d425b1e2` fix: scrub queued/blockedBy/overlapBlockedBy on _(fix+source)_ |
| 31 | `packages/engine/src/__tests__/branch-conflicts-zero-unique.test.ts` | 4 | 2.2s | -2.5 | 2475 | delete | `bcb0035d1` feat(FN-4500): complete Step 2 — harden zero-uni _(added-with-source)_<br>`14ef1041a` feat(FN-4500): complete Step 3 — add live zero-c _(test+source)_<br>`5ce377afe` feat(FN-4508): complete Step 7 — docs and compat _(test-only-churn)_ |
| 32 | `packages/engine/src/__tests__/workspace-e2e.test.ts` | 2 | 4.8s | 1 | 2400 | review | `78d7a28f1` test(workspace): Phase D U2 — e2e merge + recove _(added)_ |
| 33 | `packages/cli/src/__tests__/research-extension-tools.test.ts` | 10 | 2.1s | -2.5 | 2362.5 | delete (quarantined) | `6670837b7` feat(FN-2996): add research agent tools with ext _(added-with-source)_<br>`f2fa44e27` feat(FN-2999): harden research lifecycle with id _(churn-keyword,test+source)_<br>`f1ee69e59` feat(FN-3014): document research recovery semant _(test+source)_ |
| 34 | `packages/cli/src/__tests__/extension-mission-goal-tools.test.ts` | 4 | 1.7s | -7 | 2295 | delete (quarantined) | `93e8bd994` FN-5899: add mission-goal linking commands and t _(added-with-source)_<br>`26bc80a0a` FN-5958: add mission goal linking to create and _(test+source)_<br>`259cdfb53` FN-6734: stabilize CLI extension test cleanup _(churn-keyword,test-only-churn)_ |
| 35 | `packages/engine/src/__tests__/worktree-acquisition-workspace.test.ts` | 8 | 6.6s | 2 | 2200 | review | `12d33c512` feat(workspace): Phase A U2 — per-repo acquisiti _(added-with-source)_<br>`d5fa8654f` fix(review): Phase A workspace hardening — tool _(churn-keyword,fix+source)_ |
| 36 | `packages/dashboard/src/routes/__tests__/workflow-import-export.test.ts` | 10 | 6.5s | 2 | 2166.7 | review | `407ce03bf` feat(dashboard): workflow and template JSON impo _(added-with-source)_ |
| 37 | `packages/cli/src/__tests__/extension-goal-tools.test.ts` | 7 | 1.6s | -7 | 2160 | delete (quarantined) | `b335f3d7c` FN-5652: add goal retrieval tool for agent execu _(added-with-source)_<br>`fab8a62b5` FN-5977: expose goal retrieval tools across engi _(test+source)_<br>`259cdfb53` FN-6734: stabilize CLI extension test cleanup _(churn-keyword,test-only-churn)_ |
| 38 | `packages/dashboard/app/components/__tests__/AgentDetailView.settings.test.tsx` | 25 | 10.7s | 4 | 2140 | keep | `d98c053ed` feat(FN-4088): split AgentDetailView tests into _(added)_<br>`91931bd5d` feat(FN-4394): complete Step 5 — add dashboard s _(test+source)_<br>`6b2607dae` feat(FN-4400): complete Step 7 — add prompt-size _(test+source)_ |
| 39 | `packages/engine/src/cli-agent/__tests__/session-manager.test.ts` | 19 | 4.2s | 1 | 2100 | review | `4fc1a9dd4` feat(engine): add CliAgentAdapter interface and _(added-with-source)_<br>`773ba7620` test(engine): self-skip real-PTY e2e when PTY I/ _(test-only-churn)_ |
| 40 | `packages/engine/src/__tests__/reliability-interactions/ai-merge-worktree-cleanup.test.ts` | 7 | 6.3s | 2 | 2100 | review | `ac342f501` FN-6220: harden AI merge worktree cleanup _(churn-keyword,added-with-source)_<br>`2085610e9` FN-6246: move AI merge clean rooms into repo sto _(test+source)_<br>`dc4c2b220` FN-6453: clean up AI-merge worktrees after setup _(test+source)_ |
### One-line recommendations
1. `packages/engine/src/__tests__/workspace-merger-idempotency.test.ts` — **delete**: no positive (bug-encoding) signal in history; pure churn — strong delete candidate (slow: 12.7s — high CI time payoff)
2. `packages/dashboard/src/__tests__/github-tracking-delete.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 8.7s — high CI time payoff)
3. `packages/engine/src/__tests__/reliability-interactions/merge-runner-spawn-enoent-prevention.test.ts` — **delete**: no positive (bug-encoding) signal in history; pure churn — strong delete candidate (slow: 7.9s — high CI time payoff)
4. `packages/dashboard/src/__tests__/session-reconnect.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 6.1s — high CI time payoff)
5. `packages/engine/src/__tests__/self-healing-already-merged.real-git.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 4.6s — high CI time payoff)
6. `packages/core/src/__tests__/run-audit.integration.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 5.8s — high CI time payoff)
7. `packages/engine/src/__tests__/workspace-merger-lease.test.ts` — **delete**: no positive (bug-encoding) signal in history; pure churn — strong delete candidate (slow: 5.8s — high CI time payoff)
8. `packages/engine/src/__tests__/workspace-merger.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 8.0s — high CI time payoff)
9. `packages/cli/src/__tests__/extension-task-tools.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 3.8s — high CI time payoff)
10. `packages/engine/src/__tests__/worktree-db-hydrate.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 7.1s — high CI time payoff)
11. `packages/core/src/__tests__/activity-analytics.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 4.6s — high CI time payoff)
12. `packages/dashboard/src/routes/__tests__/board-workflows-route.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.6s — high CI time payoff)
13. `packages/dashboard/app/components/__tests__/TaskDetailModal.responsive-and-dependencies.test.tsx` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 6.6s — high CI time payoff)
14. `packages/core/src/__tests__/todo-store.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 3.8s — high CI time payoff)
15. `packages/engine/src/__tests__/reliability-interactions/soft-blocker-auto-finalize-interactions.real-git.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.4s — high CI time payoff)
16. `packages/core/src/__tests__/move-task-characterization.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 8.1s — high CI time payoff)
17. `packages/engine/src/__tests__/pr-response-run.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 11.6s — high CI time payoff)
18. `packages/engine/src/__tests__/merger-autostash-orphan-surface.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.7s — high CI time payoff)
19. `packages/engine/src/__tests__/merger-empty-cherry-pick-fallback.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.0s — high CI time payoff)
20. `packages/engine/src/__tests__/merger-ai-cleanup.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 2.8s — high CI time payoff)
21. `packages/dashboard/src/__tests__/routes-secrets-sync.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.7s — high CI time payoff)
22. `packages/core/src/__tests__/store-concurrent-writes.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 2.8s — high CI time payoff)
23. `packages/core/src/__tests__/store-archive-search.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 3.0s — high CI time payoff)
24. `packages/engine/src/__tests__/reliability-interactions/in-review-branch-rebind.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 7.4s — high CI time payoff)
25. `packages/core/src/__tests__/store-activity.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 5.9s — high CI time payoff)
26. `packages/dashboard/app/components/__tests__/AgentDetailView.mobile-scroll.test.tsx` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.5s — high CI time payoff)
27. `packages/dashboard/src/__tests__/insights-routes.test.ts` — **keep**: carries real bug-fix / source-coupled signal — keep
28. `packages/engine/src/__tests__/branch-conflicts-ghost-references.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.5s — high CI time payoff)
29. `packages/core/src/__tests__/soft-delete-lineage-children.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.6s — high CI time payoff)
30. `packages/core/src/__tests__/store-handoff-to-review.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 2.6s — high CI time payoff)
31. `packages/engine/src/__tests__/branch-conflicts-zero-unique.test.ts` — **delete**: net-negative value (churn outweighs signal) — delete candidate (slow: 2.2s — high CI time payoff)
32. `packages/engine/src/__tests__/workspace-e2e.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 4.8s — high CI time payoff)
33. `packages/cli/src/__tests__/research-extension-tools.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 2.1s — high CI time payoff)
34. `packages/cli/src/__tests__/extension-mission-goal-tools.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 1.7s — high CI time payoff)
35. `packages/engine/src/__tests__/worktree-acquisition-workspace.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 6.6s — high CI time payoff)
36. `packages/dashboard/src/routes/__tests__/workflow-import-export.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 6.5s — high CI time payoff)
37. `packages/cli/src/__tests__/extension-goal-tools.test.ts` — **delete**: in quarantine ledger — delete per ratchet once expiry passes (slow: 1.6s — high CI time payoff)
38. `packages/dashboard/app/components/__tests__/AgentDetailView.settings.test.tsx` — **keep**: carries real bug-fix / source-coupled signal — keep
39. `packages/engine/src/cli-agent/__tests__/session-manager.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 4.2s — high CI time payoff)
40. `packages/engine/src/__tests__/reliability-interactions/ai-merge-worktree-cleanup.test.ts` — **review**: thin positive signal; confirm it asserts a real invariant before trimming (slow: 6.3s — high CI time payoff)

View File

@@ -0,0 +1,171 @@
import test from "node:test";
import assert from "node:assert/strict";
import {
classifyCommit,
scoreFile,
isFixSubject,
WEIGHTS,
NEGATIVE_KEYWORDS,
} from "../lib/test-value-audit-lib.mjs";
/*
FNXC:TestInfrastructure 2026-06-25-00:00:
Unit coverage for the test-value-audit classifier. Feeds synthetic commit records and
asserts the scoring so the heuristic's behavior is pinned independent of git/FS IO.
*/
// --- isFixSubject ---
test("isFixSubject matches conventional fix subjects incl. fix(FN-XXXX)", () => {
assert.ok(isFixSubject("fix(FN-1234): correct merge race"));
assert.ok(isFixSubject("fix: handle null"));
assert.ok(isFixSubject("fix!: breaking repair"));
assert.ok(!isFixSubject("feat(FN-1): add thing"));
assert.ok(!isFixSubject("prefix change"));
});
// --- classifyCommit positive signals ---
test("fix subject + source change is strongest positive", () => {
const c = classifyCommit({ subject: "fix(FN-9): bug", body: "", status: "M", touchedSource: true });
assert.equal(c.category, "positive");
assert.equal(c.weight, WEIGHTS.fixWithSource);
assert.ok(c.signals.includes("fix+source"));
});
test("fix subject without source change is a weaker positive", () => {
const c = classifyCommit({ subject: "fix: bug", body: "", status: "M", touchedSource: false });
assert.equal(c.weight, WEIGHTS.fix);
assert.ok(c.signals.includes("fix"));
});
test("symptom verification marker adds positive weight on top of fix", () => {
const c = classifyCommit({
subject: "fix(FN-5893): invariant",
body: "## Symptom Verification\nOriginal symptom: ...",
status: "M",
touchedSource: true,
});
assert.equal(c.weight, WEIGHTS.fixWithSource + WEIGHTS.symptomVerification);
assert.ok(c.signals.includes("symptom-verification"));
});
test("added with source is positive; added alone is weak positive", () => {
const withSrc = classifyCommit({ subject: "feat: x", status: "A", touchedSource: true });
assert.equal(withSrc.weight, WEIGHTS.addedWithSource);
assert.ok(withSrc.signals.includes("added-with-source"));
const alone = classifyCommit({ subject: "test: x", status: "A", touchedSource: false });
assert.equal(alone.weight, WEIGHTS.added);
assert.ok(alone.signals.includes("added"));
});
test("non-fix test+source modify is a mild positive", () => {
const c = classifyCommit({ subject: "refactor: rework", status: "M", touchedSource: true });
assert.equal(c.weight, WEIGHTS.testPlusSource);
assert.ok(c.signals.includes("test+source"));
});
// --- classifyCommit negative signals ---
test("flake/quarantine keywords produce negative weight", () => {
for (const kw of NEGATIVE_KEYWORDS) {
const c = classifyCommit({ subject: `chore: ${kw} the test`, status: "M", touchedSource: false });
assert.ok(c.weight <= WEIGHTS.churnKeyword, `keyword ${kw} should be negative`);
assert.ok(c.signals.includes("churn-keyword"));
}
});
test("test-only modify (no source, no fix) is churn", () => {
const c = classifyCommit({ subject: "chore: tidy assertions", status: "M", touchedSource: false });
assert.equal(c.category, "negative");
assert.equal(c.weight, WEIGHTS.testOnlyChurn);
assert.ok(c.signals.includes("test-only-churn"));
});
test("a fix that also mentions a churn keyword nets toward neutral (mixed signal)", () => {
const c = classifyCommit({
subject: "fix(FN-1): widen timeout to stabilize merge",
status: "M",
touchedSource: true,
});
// churnKeyword(-3) + fixWithSource(+3) = 0 -> neutral, both signals recorded.
assert.equal(c.weight, 0);
assert.equal(c.category, "neutral");
assert.ok(c.signals.includes("churn-keyword"));
assert.ok(c.signals.includes("fix+source"));
});
// --- scoreFile aggregation ---
test("scoreFile sums weights and recommends keep for net-positive files", () => {
const commits = [
classifyCommit({ subject: "test: add", status: "A", touchedSource: true }), // +2
classifyCommit({ subject: "fix(FN-2): real bug", status: "M", touchedSource: true }), // +3
];
const s = scoreFile({ commits, durationMs: 500, testCount: 4 });
assert.equal(s.valueScore, 5);
assert.equal(s.positiveCount, 2);
assert.equal(s.recommendation, "keep");
assert.equal(s.safeDelete, false);
});
test("scoreFile recommends delete + safeDelete for pure churn with no positives", () => {
const commits = [
classifyCommit({ subject: "chore: tweak", status: "M", touchedSource: false }), // -1
classifyCommit({ subject: "chore: deflake", status: "M", touchedSource: false }), // -3 (and -1 churn? no: keyword path only)
];
const s = scoreFile({ commits, durationMs: 9000, testCount: 3 });
assert.ok(s.valueScore < 0);
assert.equal(s.positiveCount, 0);
assert.equal(s.recommendation, "delete");
assert.equal(s.safeDelete, true);
});
test("quarantine ledger membership forces delete and applies the penalty", () => {
const commits = [classifyCommit({ subject: "fix(FN-3): bug", status: "M", touchedSource: true })]; // +3
const s = scoreFile({ commits, durationMs: 100, testCount: 2, quarantined: true });
// +3 then -5 quarantine = -2 => delete; but positiveCount=1 so NOT safeDelete.
assert.equal(s.valueScore, WEIGHTS.fixWithSource + WEIGHTS.quarantineLedger);
assert.equal(s.recommendation, "delete");
assert.equal(s.quarantined, true);
assert.equal(s.safeDelete, false);
});
test("deletionPriority surfaces slow + low-value above slow + valuable", () => {
const lowValueSlow = scoreFile({
commits: [classifyCommit({ subject: "chore: tweak", status: "M", touchedSource: false })],
durationMs: 10_000,
testCount: 1,
});
const highValueSlow = scoreFile({
commits: [
classifyCommit({ subject: "fix(FN-4): bug", status: "M", touchedSource: true }),
classifyCommit({ subject: "fix(FN-5): bug2", status: "M", touchedSource: true }),
],
durationMs: 10_000,
testCount: 1,
});
assert.ok(
lowValueSlow.deletionPriority > highValueSlow.deletionPriority,
"low-value slow file must outrank high-value slow file",
);
});
test("missing duration yields zero deletionPriority (unknown cost)", () => {
const s = scoreFile({
commits: [classifyCommit({ subject: "chore: tweak", status: "M", touchedSource: false })],
durationMs: null,
testCount: 1,
});
assert.equal(s.deletionPriority, 0);
assert.equal(s.durationMs, null);
});
test("testOnlyRatio reflects churn fraction", () => {
const commits = [
classifyCommit({ subject: "test: add", status: "A", touchedSource: true }), // not churn
classifyCommit({ subject: "chore: tidy", status: "M", touchedSource: false }), // churn
classifyCommit({ subject: "chore: tidy2", status: "M", touchedSource: false }), // churn
];
const s = scoreFile({ commits, durationMs: 1, testCount: 1 });
assert.equal(s.testOnlyCount, 2);
assert.equal(s.testOnlyRatio, Number((2 / 3).toFixed(3)));
});

View File

@@ -0,0 +1,203 @@
/*
FNXC:TestInfrastructure 2026-06-25-00:00:
Heuristic "test value audit" scoring library. Pure functions only — all git/FS IO
lives in scripts/test-value-audit.mjs so the classification logic stays unit-testable
with synthetic commit records.
Why this exists (requirement): the suite has rotted before (FN-5048 slow tests, the
quarantine deletion-ratchet in AGENTS.md). We want a data-driven signal for WHICH test
files have actually encoded real bugs vs. only ever churned/flaked, so a human can drive
aggressive deletion of low-signal-yet-slow tests. This is a HEURISTIC, never ground truth:
git history is lossy (renames, squashes), commit subjects lie, and a quiet test can still
be load-bearing. Treat the output as evidence, not a verdict.
Scoring model (transparent + auditable on purpose):
- POSITIVE signal = the commit encoded/caught a real behavioral bug. Strongest when a
`fix(...)` / `fix:` subject lands alongside a sibling SOURCE change, or carries a
`## Symptom Verification` regression marker (FN-5893), or the test file was first ADDED
together with source.
- NEGATIVE signal = low value / churn. Commits whose subject/body mention flake words
(flake/quarantine/stabiliz/appease/timeout/retry/flaky/deflake) or that repeatedly
touch ONLY the test file with no source change. Appearing in the quarantine ledger
history is the strongest negative.
*/
/** Churn / appeasement keywords. A commit mentioning any of these is a negative signal. */
export const NEGATIVE_KEYWORDS = [
"flake",
"flaky",
"deflake",
"quarantine",
"stabiliz", // stabilize / stabilization / stabilise
"appease",
"timeout",
"retry",
];
/** Per-commit signed weights. Documented so the report can explain every score. */
export const WEIGHTS = {
churnKeyword: -3,
fixWithSource: 3,
fix: 2,
symptomVerification: 3,
addedWithSource: 2,
added: 1,
testPlusSource: 1.5,
testOnlyChurn: -1,
quarantineLedger: -5,
};
/** Recommendation thresholds on the final summed value score. */
export const THRESHOLDS = {
delete: 0, // valueScore <= 0 => delete candidate
review: 3, // 0 < valueScore <= 3 => review
// valueScore > 3 => keep
};
/**
* True when the subject is a conventional `fix(...)` / `fix:` / `fix!:` commit.
* The repo convention is `fix(FN-XXXX):` so this also captures task bug fixes.
* @param {string} subject
*/
export function isFixSubject(subject) {
return /^\s*fix(\(|:|!)/i.test(subject ?? "");
}
/**
* Classify a single commit that touched a given test file.
*
* @param {object} record
* @param {string} [record.subject] commit subject line
* @param {string} [record.body] commit body
* @param {string} [record.status] git name-status of the TEST file in this commit ("A","M","D","R"...)
* @param {boolean} [record.touchedSource] did the commit also change a non-test source file in the same package?
* @returns {{category:"positive"|"negative"|"neutral", weight:number, signals:string[]}}
*/
export function classifyCommit(record) {
const subject = record.subject ?? "";
const body = record.body ?? "";
const status = record.status ?? "M";
const touchedSource = Boolean(record.touchedSource);
const text = `${subject}\n${body}`.toLowerCase();
const isAdd = status.startsWith("A");
const isFix = isFixSubject(subject);
const hasSymptom = /symptom verification/i.test(body) || /symptom verification/i.test(subject);
const churnKeyword = NEGATIVE_KEYWORDS.some((k) => text.includes(k));
let weight = 0;
const signals = [];
if (churnKeyword) {
weight += WEIGHTS.churnKeyword;
signals.push("churn-keyword");
}
if (isFix && touchedSource) {
weight += WEIGHTS.fixWithSource;
signals.push("fix+source");
} else if (isFix) {
weight += WEIGHTS.fix;
signals.push("fix");
}
if (hasSymptom) {
weight += WEIGHTS.symptomVerification;
signals.push("symptom-verification");
}
if (isAdd && touchedSource) {
weight += WEIGHTS.addedWithSource;
signals.push("added-with-source");
} else if (isAdd) {
weight += WEIGHTS.added;
signals.push("added");
}
// Test+source co-change (non-fix, non-add modify) is real behavioral coverage.
if (!isAdd && !isFix && touchedSource) {
weight += WEIGHTS.testPlusSource;
signals.push("test+source");
}
// A modify that touched ONLY the test file (no source, not a fix, not the add) is churn.
if (!isAdd && !isFix && !touchedSource && !hasSymptom) {
weight += WEIGHTS.testOnlyChurn;
signals.push("test-only-churn");
}
let category = "neutral";
if (weight > 0) category = "positive";
else if (weight < 0) category = "negative";
return { category, weight, signals };
}
/**
* Aggregate per-commit classifications into a file-level value score + recommendation.
*
* @param {object} input
* @param {Array<{category:string, weight:number, signals:string[]}>} input.commits classified commits
* @param {number|null} [input.durationMs] per-file test duration from scripts/test-timings.json
* @param {number} [input.testCount] number of it()/test() cases in the file
* @param {boolean} [input.quarantined] does the file appear in the quarantine ledger (current or historical)?
*/
export function scoreFile({ commits = [], durationMs = null, testCount = 0, quarantined = false }) {
let valueScore = 0;
let positiveCount = 0;
let negativeCount = 0;
let neutralCount = 0;
let testOnlyCount = 0;
const signalTally = {};
for (const c of commits) {
valueScore += c.weight;
if (c.category === "positive") positiveCount += 1;
else if (c.category === "negative") negativeCount += 1;
else neutralCount += 1;
for (const s of c.signals ?? []) {
signalTally[s] = (signalTally[s] ?? 0) + 1;
if (s === "test-only-churn") testOnlyCount += 1;
}
}
if (quarantined) {
valueScore += WEIGHTS.quarantineLedger;
signalTally["quarantine-ledger"] = (signalTally["quarantine-ledger"] ?? 0) + 1;
}
const timesTouched = commits.length;
const testOnlyRatio = timesTouched > 0 ? testOnlyCount / timesTouched : 0;
let recommendation = "keep";
if (quarantined || valueScore <= THRESHOLDS.delete) recommendation = "delete";
else if (valueScore <= THRESHOLDS.review) recommendation = "review";
// Deletion priority surfaces SLOW + LOW-VALUE files first (most time saved per unit of
// lost signal). Valuable files divide their cost down; non-positive files keep full cost
// and get a small extra nudge proportional to how negative they are.
const cost = typeof durationMs === "number" && Number.isFinite(durationMs) ? durationMs : 0;
const negativityBoost = 1 + Math.max(0, -valueScore) * 0.05;
const deletionPriority = (cost / (1 + Math.max(0, valueScore))) * negativityBoost;
// Under the deletion-ratchet (AGENTS.md), a file is a "safe delete" candidate when it
// carries zero positive evidence and is either quarantined or pure churn.
const safeDelete = positiveCount === 0 && (quarantined || valueScore <= 0);
return {
valueScore: Number(valueScore.toFixed(2)),
positiveCount,
negativeCount,
neutralCount,
timesTouched,
testOnlyCount,
testOnlyRatio: Number(testOnlyRatio.toFixed(3)),
signalTally,
durationMs: cost || null,
testCount,
quarantined: Boolean(quarantined),
recommendation,
safeDelete,
deletionPriority: Number(deletionPriority.toFixed(1)),
};
}

View File

@@ -0,0 +1,429 @@
#!/usr/bin/env node
/*
FNXC:TestInfrastructure 2026-06-25-00:00:
Test Value Audit. Builds a heuristic "value score" for every test file under
- packages/(asterisk)/src/(asterisk)(asterisk)/__tests__/(asterisk)(asterisk)
- packages/dashboard/app/(asterisk)(asterisk)/__tests__/(asterisk)(asterisk)
from git history, joins it with per-file durations (scripts/test-timings.json), and ranks
the SLOW + LOW-VALUE files first as deletion candidates. Emits docs/test-value-audit.json
(machine artifact) and docs/test-value-audit.md (human report). This is an EVIDENCE BASE
for a human/follow-up deletion decision — the script never deletes tests.
Requirement context: AGENTS.md "Do Not Add Slow Tests" (FN-5048) + the quarantine deletion
ratchet. We need to know which slow tests are also low-signal so they can be cut without
losing real regression coverage.
HEURISTIC, not ground truth. Caveats live in the generated report's Methodology section.
Usage:
node scripts/test-value-audit.mjs # full run, writes artifacts, prints top 15
node scripts/test-value-audit.mjs --top 30 # change how many rows are printed/written to md
node scripts/test-value-audit.mjs --json-only # skip markdown
*/
import { execFileSync } from "node:child_process";
import { readFileSync, writeFileSync, existsSync } from "node:fs";
import { resolve, relative } from "node:path";
import { fileURLToPath, URL } from "node:url";
import { classifyCommit, scoreFile, WEIGHTS, NEGATIVE_KEYWORDS } from "./lib/test-value-audit-lib.mjs";
const repoRoot = fileURLToPath(new URL("..", import.meta.url));
function arg(name, fallback) {
const i = process.argv.indexOf(name);
if (i === -1) return fallback;
const v = process.argv[i + 1];
return v && !v.startsWith("--") ? v : true;
}
const TOP_N = Number(arg("--top", 40)) || 40;
const JSON_ONLY = process.argv.includes("--json-only");
function git(args) {
return execFileSync("git", args, { cwd: repoRoot, encoding: "utf8", maxBuffer: 512 * 1024 * 1024 });
}
// --- Target test files -------------------------------------------------------
// Globs (POSIX-style, repo-relative) for the two audited surfaces.
function isTargetTestFile(path) {
if (!/\.(test|spec)\.(ts|tsx|mts|cts|js|mjs|cjs)$/.test(path)) return false;
if (!path.includes("/__tests__/")) return false;
if (/^packages\/[^/]+\/src\//.test(path)) return true;
if (path.startsWith("packages/dashboard/app/")) return true;
return false;
}
function packageRootOf(path) {
const m = path.match(/^(packages\/[^/]+)\//);
return m ? m[1] : null;
}
// A non-test source file in the same package (used for "test + source changed together").
function isSourceFileInPackage(path, pkgRoot) {
if (!pkgRoot || !path.startsWith(pkgRoot + "/")) return false;
if (!/\.(ts|tsx|mts|cts|js|mjs|cjs)$/.test(path)) return false;
if (/\.(test|spec)\./.test(path)) return false;
if (path.includes("/__tests__/")) return false;
if (path.includes("/__test-utils__/") || path.includes("/__mocks__/")) return false;
return true;
}
// --- Timings -----------------------------------------------------------------
function loadTimings() {
const p = resolve(repoRoot, "scripts/test-timings.json");
const map = new Map();
if (!existsSync(p)) return map;
const data = JSON.parse(readFileSync(p, "utf8"));
for (const pkg of Object.values(data.packages ?? {})) {
for (const [file, ms] of Object.entries(pkg.files ?? {})) {
// Keep the max if a path appears twice across packages.
map.set(file, Math.max(map.get(file) ?? 0, Number(ms) || 0));
}
}
return map;
}
// --- Quarantine ledger (current + historical) --------------------------------
function loadQuarantinePaths() {
const set = new Set();
const ledger = "scripts/lib/test-quarantine.json";
try {
const current = JSON.parse(readFileSync(resolve(repoRoot, ledger), "utf8"));
for (const e of current.entries ?? []) if (e.file) set.add(e.file);
} catch {
/* ledger may not exist */
}
// Historical entries: scan every past revision of the ledger for "file": "<path>".
try {
const blob = git(["log", "-p", "--format=", "--", ledger]);
for (const m of blob.matchAll(/"file"\s*:\s*"([^"]+)"/g)) set.add(m[1]);
} catch {
/* no history */
}
return set;
}
// --- Whole-history parse -----------------------------------------------------
// One git invocation yields every commit's subject/body/time + name-status file list.
// Field sep \x1f, body terminator \x02, commit start \x01.
function loadHistory() {
const SEP = "\x1f";
const BODY_END = "\x02";
const COMMIT_START = "\x01";
const raw = git([
"log",
"--no-color",
"--name-status",
`--format=${COMMIT_START}%H${SEP}%ct${SEP}%s${SEP}%b${BODY_END}`,
]);
const commits = [];
// newPath -> oldPath rename map (last writer wins; good enough for linear chains).
const renameMap = new Map();
// path -> array of commit indices that touched it (as add/modify/rename target).
const byPath = new Map();
const chunks = raw.split(COMMIT_START);
for (const chunk of chunks) {
if (!chunk.trim()) continue;
const bodyEnd = chunk.indexOf(BODY_END);
if (bodyEnd === -1) continue;
const head = chunk.slice(0, bodyEnd);
const [sha, ct, subject, ...bodyParts] = head.split(SEP);
const body = bodyParts.join(SEP);
const nameStatusBlock = chunk.slice(bodyEnd + 1);
const files = [];
for (const line of nameStatusBlock.split("\n")) {
const t = line.trim();
if (!t) continue;
const cols = t.split("\t");
const statusRaw = cols[0];
if (!statusRaw) continue;
const status = statusRaw[0]; // A/M/D/R/C/T
if ((status === "R" || status === "C") && cols.length >= 3) {
const oldPath = cols[1];
const newPath = cols[2];
files.push({ status, path: newPath, oldPath });
renameMap.set(newPath, oldPath);
} else if (cols.length >= 2) {
files.push({ status, path: cols[1] });
}
}
const idx = commits.length;
commits.push({
sha,
time: Number(ct) * 1000,
subject: subject ?? "",
body: body ?? "",
files,
});
for (const f of files) {
if (f.status === "D") continue;
if (!byPath.has(f.path)) byPath.set(f.path, []);
byPath.get(f.path).push(idx);
}
}
return { commits, renameMap, byPath };
}
// Follow renames backward to collect every historical path this file lived at.
function historicalPaths(currentPath, renameMap) {
const paths = new Set([currentPath]);
let p = currentPath;
let guard = 0;
while (renameMap.has(p) && guard < 100) {
p = renameMap.get(p);
if (paths.has(p)) break;
paths.add(p);
guard += 1;
}
return paths;
}
function countTestCases(absPath) {
try {
const src = readFileSync(absPath, "utf8");
let n = 0;
for (const line of src.split("\n")) {
if (/^\s*(it|test)\s*(\.\w+)?\s*\(/.test(line)) n += 1;
}
return n;
} catch {
return 0;
}
}
function fmtMs(ms) {
if (ms == null) return "n/a";
if (ms >= 1000) return `${(ms / 1000).toFixed(1)}s`;
return `${ms}ms`;
}
// --- Main --------------------------------------------------------------------
function main() {
const started = Date.now();
const timings = loadTimings();
const quarantinePaths = loadQuarantinePaths();
const { commits, renameMap, byPath } = loadHistory();
const tracked = git(["ls-files"]).split("\n").filter(Boolean);
const testFiles = tracked.filter(isTargetTestFile).sort();
const rows = [];
for (const file of testFiles) {
const pkgRoot = packageRootOf(file) ?? "packages/dashboard";
const paths = historicalPaths(file, renameMap);
// Gather candidate commit indices across all historical paths, dedup.
const idxSet = new Set();
for (const p of paths) for (const i of byPath.get(p) ?? []) idxSet.add(i);
const classified = [];
const evidence = [];
for (const i of [...idxSet].sort((a, b) => b - a)) {
const c = commits[i];
const fileEntry = c.files.find((f) => paths.has(f.path) && f.status !== "D");
if (!fileEntry) continue;
const status = fileEntry.status;
const touchedSource = c.files.some(
(f) => f.status !== "D" && isSourceFileInPackage(f.path, pkgRoot),
);
const cls = classifyCommit({ subject: c.subject, body: c.body, status, touchedSource });
classified.push(cls);
evidence.push({
sha: c.sha.slice(0, 9),
time: c.time,
subject: c.subject,
status,
touchedSource,
category: cls.category,
weight: cls.weight,
signals: cls.signals,
});
}
const durationMs = timings.has(file) ? timings.get(file) : null;
const testCount = countTestCases(resolve(repoRoot, file));
const quarantined = [...paths].some((p) => quarantinePaths.has(p));
const score = scoreFile({ commits: classified, durationMs, testCount, quarantined });
const times = evidence.map((e) => e.time).filter(Boolean);
const firstSeen = times.length ? Math.min(...times) : null;
const lastSeen = times.length ? Math.max(...times) : null;
const ageDays = firstSeen ? Math.round((Date.now() - firstSeen) / 86_400_000) : null;
rows.push({
file,
package: pkgRoot,
...score,
ageDays,
lastTouched: lastSeen ? new Date(lastSeen).toISOString().slice(0, 10) : null,
evidence: evidence.slice(0, 6), // most-recent commits as the "why"
});
}
// Rank: slow + low-value first.
rows.sort((a, b) => b.deletionPriority - a.deletionPriority || a.valueScore - b.valueScore);
const summary = {
total: rows.length,
deleteCandidates: rows.filter((r) => r.recommendation === "delete").length,
reviewCandidates: rows.filter((r) => r.recommendation === "review").length,
keep: rows.filter((r) => r.recommendation === "keep").length,
safeDeletes: rows.filter((r) => r.safeDelete).length,
withTiming: rows.filter((r) => r.durationMs != null).length,
quarantined: rows.filter((r) => r.quarantined).length,
};
const artifact = {
generatedAt: new Date().toISOString(),
heuristic: true,
note: "HEURISTIC, not ground truth. Do not auto-delete from this file. See docs/test-value-audit.md methodology + caveats.",
weights: WEIGHTS,
negativeKeywords: NEGATIVE_KEYWORDS,
timingsCapturedFrom: "scripts/test-timings.json",
summary,
rows,
};
const jsonPath = resolve(repoRoot, "docs/test-value-audit.json");
writeFileSync(jsonPath, JSON.stringify(artifact, null, 2) + "\n");
if (!JSON_ONLY) writeMarkdown(rows, summary);
// Console: top 15 rows.
const top = rows.slice(0, 15);
const tbl = top.map((r, i) => ({
"#": i + 1,
file: r.file.replace(/^packages\//, "").length > 58 ? "…" + r.file.slice(-57) : r.file.replace(/^packages\//, ""),
tests: r.testCount,
dur: fmtMs(r.durationMs),
score: r.valueScore,
rec: r.recommendation,
}));
console.log(`\nTest Value Audit — ${rows.length} files analyzed in ${((Date.now() - started) / 1000).toFixed(1)}s`);
console.log(
`delete=${summary.deleteCandidates} review=${summary.reviewCandidates} keep=${summary.keep} ` +
`safeDelete=${summary.safeDeletes} withTiming=${summary.withTiming}\n`,
);
console.log("Top 15 deletion candidates (slow + low value first):");
console.table(tbl);
console.log(`\nArtifacts:\n ${relative(repoRoot, jsonPath)}`);
if (!JSON_ONLY) console.log(` docs/test-value-audit.md`);
}
function recEmoji(rec) {
return rec === "delete" ? "delete" : rec === "review" ? "review" : "keep";
}
function writeMarkdown(rows, summary) {
const top = rows.slice(0, TOP_N);
const L = [];
L.push("# Test Value Audit");
L.push("");
L.push(`> Generated by \`scripts/test-value-audit.mjs\` on ${new Date().toISOString()}.`);
L.push(">");
L.push("> **This is a HEURISTIC, not ground truth.** It is an evidence base for a human");
L.push("> deletion decision. The script does **not** delete any tests. See Methodology + Caveats.");
L.push("");
L.push("## Summary");
L.push("");
L.push(`- Files analyzed: **${summary.total}**`);
L.push(`- With timing data: **${summary.withTiming}**`);
L.push(`- \`delete\` candidates (valueScore ≤ 0 or quarantined): **${summary.deleteCandidates}**`);
L.push(`- \`review\` candidates (0 < valueScore ≤ 3): **${summary.reviewCandidates}**`);
L.push(`- \`keep\` (valueScore > 3): **${summary.keep}**`);
L.push(`- "Safe delete" under the deletion-ratchet (zero positive evidence + churn/quarantine): **${summary.safeDeletes}**`);
L.push(`- In quarantine ledger (current or historical): **${summary.quarantined}**`);
L.push("");
L.push("## Methodology");
L.push("");
L.push("For every audited test file we run a single whole-history `git log --name-status`");
L.push("pass and classify each commit that touched the file (renames followed backward):");
L.push("");
L.push("**Positive signal** (encoded/caught a real bug):");
L.push("- `fix(...)`/`fix:` subject **with** a sibling source change in the same package (+3)");
L.push("- `fix(...)`/`fix:` subject alone (+2)");
L.push("- `## Symptom Verification` regression marker in the commit body (+3, FN-5893)");
L.push("- test file first **added** together with source (+2), or added alone (+1)");
L.push("- a plain test+source co-change modify (+1.5)");
L.push("");
L.push("**Negative signal** (low value / churn):");
L.push("- subject/body mentions " + NEGATIVE_KEYWORDS.map((k) => `\`${k}\``).join(", ") + " (−3)");
L.push("- a modify that touched **only** the test file, no source (−1)");
L.push("- appears in `scripts/lib/test-quarantine.json` history (−5)");
L.push("");
L.push("The per-file **valueScore** is the sum of commit weights (plus the quarantine penalty).");
L.push("**deletionPriority** = `durationMs / (1 + max(0, valueScore))` with a small boost for");
L.push("net-negative files — so the ranking surfaces **slow AND low-value** files first");
L.push("(most CI time saved per unit of lost signal). Recommendation: `delete` (≤0), `review`");
L.push("(≤3), else `keep`.");
L.push("");
L.push("## Caveats (read before deleting anything)");
L.push("");
L.push("- **Heuristic, not truth.** A quiet, never-modified test can still be load-bearing;");
L.push(" a high-churn test can still be valuable. Use this to *prioritize human review*.");
L.push("- **`git log --follow` / rename limits.** Renames are followed only through linear");
L.push(" `R`/`C` name-status chains; squash-merges collapse multi-commit history into one");
L.push(" subject, so per-commit signal is lost for squashed work (this repo defaults to");
L.push(" squash merges — a major reason to treat scores as lower bounds on value).");
L.push("- **Subjects lie.** `fix(...)` is trusted as a positive even if the test was unrelated;");
L.push(" conversely a real bug fixed under a `feat(...)`/`FN-` subject without source co-change");
L.push(" may be undercounted.");
L.push("- **Timing is a snapshot** from `scripts/test-timings.json`; files with no entry show");
L.push(" `n/a` duration and get deletionPriority 0 (cost unknown, not necessarily cheap).");
L.push("- **Not a green light.** Deleting a gate test still requires the gate-eviction process");
L.push(" (AGENTS.md). \"Safe delete\" only flags files that meet the ratchet's churn/quarantine bar.");
L.push("");
L.push(`## Top ${top.length} deletion candidates`);
L.push("");
L.push("| # | File | Tests | Duration | Value | Priority | Rec | Why (recent commits) |");
L.push("|---|------|------:|---------:|------:|---------:|-----|----------------------|");
top.forEach((r, i) => {
const why = r.evidence
.slice(0, 3)
.map((e) => {
const subj = (e.subject || "").replace(/\|/g, "\\|").slice(0, 48);
return `\`${e.sha}\` ${subj} _(${e.signals.join(",") || "neutral"})_`;
})
.join("<br>");
const flags = [];
if (r.quarantined) flags.push("quarantined");
if (r.safeDelete) flags.push("safe-delete");
const rec = recEmoji(r.recommendation) + (flags.length ? ` (${flags.join(", ")})` : "");
L.push(
`| ${i + 1} | \`${r.file}\` | ${r.testCount} | ${fmtMs(r.durationMs)} | ${r.valueScore} | ${r.deletionPriority} | ${rec} | ${why || "_no git evidence_"} |`,
);
});
L.push("");
L.push("### One-line recommendations");
L.push("");
top.forEach((r, i) => {
let rationale;
if (r.recommendation === "delete") {
rationale = r.quarantined
? "in quarantine ledger — delete per ratchet once expiry passes"
: r.positiveCount === 0
? "no positive (bug-encoding) signal in history; pure churn — strong delete candidate"
: "net-negative value (churn outweighs signal) — delete candidate";
} else if (r.recommendation === "review") {
rationale = "thin positive signal; confirm it asserts a real invariant before trimming";
} else {
rationale = "carries real bug-fix / source-coupled signal — keep";
}
if (r.durationMs != null && r.durationMs >= 1000 && r.recommendation !== "keep") {
rationale += ` (slow: ${fmtMs(r.durationMs)} — high CI time payoff)`;
}
L.push(`${i + 1}. \`${r.file}\` — **${r.recommendation}**: ${rationale}`);
});
L.push("");
writeFileSync(resolve(repoRoot, "docs/test-value-audit.md"), L.join("\n") + "\n");
}
main();