fusion/docs/test-audit-report.md

FN-1204 Test Effectiveness Audit

Date: 2026-04-08

1) Executive Summary

FN-3293 stabilization update (2026-05-04)

• Replaced ad-hoc frame sleeps in packages/cli/src/commands/dashboard-tui/__tests__/app.test.tsx with deterministic vi.waitFor-based frame assertions and microtask flush helpers.
• Updated settings remote-action handling so C/V/X/P/L/U/K/R shortcuts are exercised deterministically in tests without timing races on pane focus transitions.
• Removed the blanket { timeout: 90_000 } suite override in packages/droid-cli/src/__tests__/provider.test.ts; lifecycle coverage now relies on fake timers and event-driven completion.
• Tightened dashboard GitHub route tests (packages/dashboard/src/__tests__/routes-github.test.ts) by reducing synthetic retry delays and removing explicit long per-test timeouts.
• Confirmed packages/core/src/__tests__/memory-backend.test.ts remains fast while still asserting installQmd()/ensureQmdInstalled() forward the intended timeout: 120_000 to execFileAsync.
• Restored deterministic CLI bundle-output verification by resolving the droid-runtime probe export path to source entries in workspace tests, eliminating build-order flake from missing dist/probe.js.

Overall test health: Good (with targeted high-risk gaps)

• Total executed tests across audited packages (core, engine, cli, dashboard): 8,188 passing
• Total assertion calls (expect(...)): 18,070
• Overall pattern: strong coverage depth in @fusion/core, strong scenario breadth in @fusion/engine and @fusion/dashboard, but uneven effectiveness in a few critical modules (especially untested orchestration/runtime files and large dashboard surfaces).

Key risks

1. Untested critical runtime path: packages/engine/src/runtimes/child-process-worker.ts (0% coverage, no dedicated test).
2. Large untested dashboard modules: packages/dashboard/src/mission-routes.ts (1,862 LOC), MissionInterviewModal.tsx (1,090 LOC), AgentGenerationModal.tsx, AgentImportModal.tsx.
3. Partially-tested high-surface modules: packages/dashboard/src/routes.ts (55.51% lines despite very large route test suite), packages/cli/src/project-resolver.ts (30.71% lines), packages/engine/src/reviewer.ts (51.28% branch coverage).
4. Flaky/time-sensitive pressure points: dashboard retry-path timeout test in packages/dashboard/src/routes.test.ts:3992-4016 (explicit 30s timeout), heavy timer-driven suites in engine stuck-detector/heartbeat tests.

---

2) Quantitative Metrics

Per-package metrics

Package	Test Files	Source Files	Test:Source	Tests Passed	Coverage (Lines / Branch / Func)	it()	expect()	vi.mock()
@fusion/core	34	33	1.03	1,427	88.98 / 87.05 / 92.80	1,427	3,249	1
@fusion/engine	41	43	0.95	1,464	79.46 / 82.29 / 84.38	1,415	2,873	55
@runfusion/fusion (CLI)	28	24	1.17	522	68.48 / 71.70 / 74.90	512	1,106	73
@fusion/dashboard	163	156	1.04	4,775 (+1 skipped)	73.70 / 84.87 / 65.87	4,743	10,842	174

Test runtime snapshots (latest per-package runs)

• core: Duration 7.32s (packages/core)
• engine: Duration 4.53s (packages/engine, direct vitest run)
• cli: Duration 14.82s (packages/cli)
• dashboard: Duration 31.63s (packages/dashboard)

Per-file coverage hotspots (lowest-impact areas)

Full per-file line/branch/function percentages were collected for all audited files from coverage-final.json; table below highlights the weakest/high-risk files per package.

@fusion/core

File	Lines	Branch	Func
packages/core/src/gh-cli.ts	26.51	100.00	20.00
packages/core/src/ai-summarize.ts	59.53	81.81	90.90

@fusion/engine

File	Lines	Branch	Func
packages/engine/src/runtimes/child-process-worker.ts	0.00	100.00	100.00
packages/engine/src/github.ts	12.50	100.00	0.00
packages/engine/src/runtimes/child-process-runtime.ts	32.36	60.00	34.61
packages/engine/src/pr-monitor.ts	48.04	66.66	84.61
packages/engine/src/pi.ts	56.61	79.16	50.00
packages/engine/src/reviewer.ts	82.48	51.28	66.66

@runfusion/fusion (CLI)

File	Lines	Branch	Func
packages/cli/src/project-resolver.ts	30.71	82.14	33.33
packages/cli/src/commands/settings-export.ts	0.00	0.00	0.00
packages/cli/src/commands/settings-import.ts	0.00	0.00	0.00
packages/cli/src/runtime/native-patch.ts	0.00	100.00	100.00
packages/cli/src/bin.ts	55.67	49.35	100.00
packages/cli/src/commands/git.ts	54.85	41.79	75.00

@fusion/dashboard (selected high-risk files)

File	Lines	Branch	Func
packages/dashboard/src/mission-routes.ts	53.43	95.65	77.77
packages/dashboard/src/subtask-breakdown.ts	51.08	50.00	52.17
packages/dashboard/src/script-store.ts	0.00	0.00	0.00
packages/dashboard/app/components/MissionInterviewModal.tsx	23.82	73.33	16.66
packages/dashboard/app/components/AgentImportModal.tsx	13.97	10.00	12.50
packages/dashboard/app/components/AgentGenerationModal.tsx	23.35	26.66	12.50
packages/dashboard/app/hooks/useProjectHealth.ts	37.23	62.50	100.00
packages/dashboard/app/hooks/useFileBrowser.ts	0.00	0.00	0.00
packages/dashboard/app/hooks/useFileEditor.ts	0.00	0.00	0.00
packages/dashboard/app/hooks/useBackgroundSessions.ts	59.70	100.00	33.33

---

3) Test Quality Assessment

3.1 Mock density analysis

mockDensity = vi.mock() / expect()

Package	vi.mock()	expect()	Mock Density
core	1	3,249	0.000
engine	55	2,873	0.019
cli	73	1,106	0.066
dashboard	174	10,842	0.016

Files where mocks exceed assertions

• packages/cli/src/bin.test.ts — 10 mocks vs 6 assertions (vi.mock(...) concentrated at :55-107; assertions begin around :163+).

3.2 Assertion quality audit (10-file sample)

Sampled files (2-3 per package):

• Core: store.test.ts, db-migrate.test.ts, central-core.test.ts
• Engine: executor-core.test.ts, executor-step-session.test.ts, stuck-task-detector.test.ts
• CLI: commands/task.test.ts, commands/dashboard.test.ts
• Dashboard: routes.test.ts, SettingsModal.test.tsx, components/__tests__/Column.test.tsx

Sample assertion distribution (7,009 sampled assertions):

• State verification: 4,898 (69.9%)
• Interaction verification: 599 (8.5%)
• Rendering checks: 42 (0.6%)
• Error-path assertions: 1,470 (21.0%)

Interpretation:

• Back-end-heavy suites are strong on state and error-path assertions.
• UI rendering assertions exist but are concentrated in component-focused tests (example: packages/dashboard/app/components/__tests__/Column.test.tsx).

3.3 Edge-case vs happy-path

(Using describe / it keyword heuristics for error|fail|invalid|edge|boundary|empty|null|undefined)

Package	Total it()	Edge-Focused it()	Happy-Path it()	Edge Ratio
core	1,427	267	1,160	18.7%
engine	1,415	325	1,090	23.0%
cli	512	89	423	17.4%
dashboard	4,743	771	3,972	16.3%
Total	8,097	1,452	6,645	17.9%

3.4 Test isolation quality (real DB/runtime vs mocked)

Heuristic classification by file content:

Package	Real	Hybrid	Mock-only	Neither
core	13	1	2	18
engine	1	0	32	8
cli	0	2	19	7
dashboard	0	0	115	48

Interpretation:

• core is strongest on realistic persistence-level testing.
• engine, cli, and especially dashboard rely heavily on mocked environments.

3.5 Flaky test indicators

Notable timer/retry/race hotspots:

• packages/engine/src/stuck-task-detector.test.ts (heavy timer simulation)
• packages/engine/src/agent-heartbeat.test.ts (timer-heavy heartbeat behavior)
• packages/cli/src/commands/dashboard.test.ts (many timeout-driven behavioral checks)
• Resolved in FN-3293: packages/dashboard/src/__tests__/routes-github.test.ts batch-import 429/diff-path coverage now runs with deterministic mocked throttling and no explicit per-test 10s/30s timeout overrides.

Observed during test runs:

• recurring MaxListenersExceededWarning in CLI dashboard test runs
• occasional act warnings in websocket hook tests (useBadgeWebSocket) indicating async test fragility risk

---

4) Coverage Gap Analysis

4.1 Untested source files (refined, non-config)

Core (1)

• packages/core/src/automation.ts (144 LOC)

Engine (3)

• packages/engine/src/runtimes/child-process-worker.ts (175 LOC)
• packages/engine/src/agent-tools.ts (88 LOC)
• packages/engine/src/github.ts (38 LOC)

Dashboard (33, top/high-impact shown)

• packages/dashboard/src/mission-routes.ts (1,862 LOC)
• packages/dashboard/app/components/MissionInterviewModal.tsx (1,090 LOC)
• packages/dashboard/app/components/AgentGenerationModal.tsx (434 LOC)
• packages/dashboard/app/components/AgentImportModal.tsx (402 LOC)
• packages/dashboard/src/subtask-breakdown.ts (379 LOC)
• packages/dashboard/src/script-store.ts (75 LOC)
• packages/dashboard/app/hooks/useAgents.ts (60 LOC)
• packages/dashboard/app/hooks/useFileBrowser.ts (85 LOC)
• packages/dashboard/app/hooks/useFileEditor.ts (121 LOC)
• packages/dashboard/app/hooks/useProjectHealth.ts (144 LOC)
• packages/dashboard/app/hooks/useLiveTranscript.ts (44 LOC)
• packages/dashboard/app/hooks/useBackgroundSessions.ts (94 LOC)

CLI (3)

• packages/cli/src/runtime/native-patch.ts (224 LOC)
• packages/cli/src/commands/settings-import.ts (121 LOC)
• packages/cli/src/commands/settings-export.ts (72 LOC)

4.2 Correction to the prelisted “known untested files”

One prelisted item is no longer untested:

• packages/dashboard/src/ai-session-store.ts has packages/dashboard/src/ai-session-store.test.ts (confirmed by file inventory + test content).

4.3 Partially-tested files (tests exist, primary surface under-covered)

• packages/dashboard/src/routes.ts — 55.51% lines despite very large route test suite (routes.test.ts); indicates many route branches still uncovered.
• packages/engine/src/reviewer.ts — 51.28% branch coverage with existing reviewer tests; branching paths under-exercised.
• packages/cli/src/project-resolver.ts — 30.71% lines / 33.33% functions with tests present.
• packages/core/src/gh-cli.ts — 26.51% lines / 20% functions despite gh-cli.test.ts; tests mostly cover helper/logic paths rather than process-exec behavior.

4.4 Critical path integration coverage status

• SQLite migration paths: covered (packages/core/src/migration.test.ts:329-363, db-migrate.test.ts)
• Agent lifecycle/restart recovery: covered (packages/engine/src/restart.integration.test.ts:717+, resumeOrphaned scenarios)
• Task state transitions: covered primarily via packages/core/src/store.test.ts
• Merge conflict resolution: covered in merger tests (packages/engine/src/merger.test.ts:975+, conflict categorization/retry logic)
• WebSocket reconnection: hook-level reconnection covered (packages/dashboard/app/hooks/__tests__/useBadgeWebSocket.test.ts:154), but no full end-to-end websocket reconnection integration test combining server + client state freshness logic.

4.5 Memory pitfall cross-reference

From .fusion/memory/MEMORY.md testing pitfalls:

• Engine pool setting must remain threads — config currently reflects this (packages/engine/vitest.config.ts:10), but there is no dedicated regression test guarding accidental config drift.
• FN-3293 update: dashboard GitHub batch-import retry/diff-path tests no longer rely on explicit long timeout gates; deterministic mocked throttling + reduced delay parameters keep default-lane coverage fast.
• Build-before-test/workspace hydration pitfalls — operationally reproducible; not represented by explicit dedicated regression tests as standalone safeguards.

---

5) Strengths

1. Core package depth is excellent: strong DB/store/migration coverage and many realistic persistence-level tests.
2. Engine scenario breadth is strong: restart, scheduler, merge, semaphore, stuck-task logic, and runtime abstractions are broadly exercised.
3. Dashboard test volume is high: route coverage breadth and many UI/unit tests protect core workflows.
4. Edge-path testing exists in every package: 17.9% edge-focused it() across audited suites.
5. Mock density is generally controlled outside CLI bootstrapping tests.

---

6) Recommendations (ranked)

1. Add dedicated tests for child-process-worker.ts and child-process runtime lifecycle (highest engine isolation risk).
2. Split and test mission-routes.ts by feature area; add integration coverage for mission autopilot + route error handling.
3. Add tests for untested dashboard hooks/components in task/agent workflows (useAgents, useProjectHealth, useFileBrowser, useFileEditor, MissionInterviewModal, AgentImportModal, AgentGenerationModal).
4. Add focused tests for core/src/automation.ts to complement automation-store coverage.
5. Increase branch coverage in engine/reviewer.ts and cli/project-resolver.ts by testing fallback/error branches explicitly.
6. Add end-to-end websocket reconnection + freshness tests (server event + client merge behavior), not just hook-unit reconnection.
7. Stabilize timer-heavy suites by reducing reliance on real delays where feasible and consolidating fake-timer patterns.
8. Add regression guard for engine vitest pool configuration (pool: "threads") to prevent accidental VM pool regressions.

---

7) Follow-up Tasks

Created from this audit:

• FN-1279 — Add engine child-process runtime regression coverage (child-process-worker.ts, runtime lifecycle + IPC behavior).
• FN-1280 — Add focused tests for packages/core/src/automation.ts schedule/validation/error paths.
• FN-1281 — Close major dashboard route/service coverage gaps (mission-routes.ts, subtask-breakdown.ts, script-store.ts).
• FN-1282 — Add tests for untested dashboard agent workflow hooks/components (useAgents, useProjectHealth, file hooks, agent modals, mission interview modal).
• FN-1283 — Improve CLI test effectiveness (settings-import.ts, settings-export.ts, project-resolver.ts, bin.ts fallback paths).

---

FN-1675 Roadmap Regression Test Coverage

Date: 2026-04-16 (updated 2026-05-08)

Roadmap ownership has moved out of @fusion/core into plugins/fusion-plugin-roadmap (@fusion-plugin-examples/roadmap).

• Core roadmap tests/fixtures referenced in earlier drafts (packages/core/src/roadmap-*.ts and packages/core/src/__tests__/roadmap-*.test.ts) were removed during plugin extraction.
• Dashboard roadmap adapter coverage remains at packages/dashboard/src/__tests__/roadmap-routes.routes.test.ts.
• Roadmap domain/ordering/handoff/store coverage now belongs with plugin package tests.

Use plugin/dashboard scoped commands instead of core roadmap commands:

pnpm --filter @fusion-plugin-examples/roadmap test
pnpm --filter @fusion/dashboard exec vitest run src/__tests__/roadmap-routes.routes.test.ts
pnpm --filter @fusion/dashboard exec vitest run app/__tests__/api-settings.test.ts -t roadmap