Merge pull request #51 from Runfusion/codex/testing-suite-quality-prd

test: harden and slim local test workflow
This commit is contained in:
gsxdsm
2026-05-07 10:18:35 -07:00
committed by GitHub
37 changed files with 2074 additions and 408 deletions

View File

@@ -0,0 +1,31 @@
# Testing Suite Hardening Bugs
_Started: 2026-05-05_
This is the living bug log for the testing-suite hardening work described in
`docs/testing-suite-quality-prd.md`. Each entry should stay factual: what broke,
why it broke, how it was fixed, and what command verified the fix.
## Open
| ID | Area | Symptom | Root cause | Planned fix | Status |
|---|---|---|---|---|---|
| _None_ | | | | | |
## Fixed
| ID | Area | Symptom | Root cause | Fix | Verification |
|---|---|---|---|---|---|
| TSH-001 | Local test selection | `pnpm test` could under-test shared package edits. | Changed-package resolution selected direct changed workspaces but not dependent workspaces that import them. | `scripts/test-changed.mjs` now expands affected packages through the workspace reverse-dependency graph. | `node --test scripts/__tests__/test-changed.test.mjs` and `pnpm test:full` |
| TSH-002 | Local test cache | Cached package passes could hide dirty worktree edits. | Cache keys were based on tracked blob SHAs and did not account for modified or untracked working-tree content. | Dirty affected files now bypass package cache reuse so local edits are exercised before PR. | `node --test scripts/__tests__/test-changed.test.mjs` and `pnpm test:full` |
| TSH-003 | PR CI coverage | PR sharding omitted plugin packages and `@fusion/pi-llama-cpp`. | The CI shard package list was hard-coded instead of derived from the workspace graph. | `scripts/ci-test-shard.mjs` now derives shard candidates from workspace packages with test scripts. | `node --test scripts/__tests__/test-changed.test.mjs scripts/__tests__/test-governance.test.mjs` and `pnpm test:full` |
| TSH-004 | Test governance | Runtime package Vitest configs bypassed shared worker budgeting/isolation conventions. | Configs had drifted independently across Droid, Pi Claude, and Pi Llama packages. | The configs now share the same thread pool sizing and isolation defaults, with a governance test to keep them aligned. | `node --test scripts/__tests__/test-governance.test.mjs` and `pnpm test:full` |
| TSH-005 | Plugin changed-test targeting | Plugin edits still forced the entire suite after package-aware targeting was added. | `shouldForceFullSuite()` treated every `plugins/**` path as broad repo surface area. | Plugin workspace paths are now resolved like package paths, so plugin edits run the relevant plugin tests instead of automatically falling back to all tests. | `node --test scripts/__tests__/test-changed.test.mjs` |
| TSH-006 | Skipped-test inventory | The skipped-test inventory documented the wrong gate ownership for the new and legacy extension suites. | The maintained extension integration lane and legacy exhaustive gate were renamed during implementation, but the inventory text still reflected the earlier split. | `docs/skipped-test-inventory.md` now lists `FUSION_TEST_EXTENSION_INTEGRATION` for the maintained built-extension test and `FUSION_TEST_LEGACY_EXTENSION_INTEGRATION` for the legacy suite. | `git diff --check`, `node --test scripts/__tests__/test-governance.test.mjs`, and `pnpm test:full` |
| TSH-008 | Test isolation | `custom-providers.test.ts` wrote to the real `~/.fusion/settings.json` path during the first local full-suite run. | The test used `os.homedir()` directly instead of a fixture home directory. | `readCustomProviders()` now accepts an injectable home directory, and the test uses a temp HOME fixture that is removed after each case. | `pnpm --filter @fusion/engine test -- src/__tests__/custom-providers.test.ts` (ran the full engine lane: 108 files, 3339 tests) |
| TSH-007 | Dashboard runtime/noise | `pnpm test:full` passed but dashboard tests took about 10 minutes locally and emitted repeated SQLite experimental warnings plus git default-branch hints. | Dashboard tests mix real SQLite-backed route coverage, git-backed route fixtures, and jsdom UI suites in one package lane. Node emits `node:sqlite` experimental warnings globally, some git fixtures used plain `git init`, and dashboard tests used deprecated `environmentMatchGlobs`. | Dashboard tests now use explicit Vitest projects for `dashboard-app` and `dashboard-api`, package scripts expose `test:app`/`test:api`/`test:browser-smoke` lanes, the shared Vitest setup filters known SQLite experimental warnings, fixture repos initialize with `main`, and low-value agent CSS class-presence assertions were cut in favor of focused layout/token/no-inline-style contracts. | `pnpm --filter @fusion/dashboard test:app` (287 files, 7254 tests), `pnpm --filter @fusion/dashboard test:api` (132 files, 3493 tests), `pnpm --filter @fusion/core exec vitest run src/__tests__/central-db.test.ts --silent=passed-only --reporter=dot`, and `pnpm test:full` |
| TSH-009 | PR action minutes | PR test shards built the whole workspace before running tests, duplicating the separate build gate. | `pr-checks.yml` had a `pnpm build` step inside every test-shard job. | Removed the pre-test build from PR test shards; package tests that require artifacts build their own fixtures, while `build` remains an explicit PR workflow job. | `pnpm --filter @runfusion/fusion exec vitest run src/__tests__/ci-workflow.test.ts --silent=passed-only --reporter=dot` and `pnpm build` |
| TSH-010 | Dashboard lane scripts | `pnpm --filter @fusion/dashboard test:api` failed with "No test files found" after converting dashboard to Vitest projects. | Positional glob filters were evaluated against all projects instead of selecting the intended dashboard project. | Dashboard lane scripts now use `--project dashboard-app` and `--project dashboard-api`. | `pnpm --filter @fusion/dashboard test:api` and `pnpm --filter @fusion/dashboard exec vitest run --project dashboard-app app/__tests__/agent-css-classes.test.ts --silent=passed-only --reporter=dot` |
| TSH-011 | PR build coverage | While reducing PR shard minutes, the workflow temporarily had no standalone `pnpm build` job. | The old workflow only had lint, typecheck, and test shards, so removing the shard-local build also removed PR build coverage. | Added an explicit `build` job to `pr-checks.yml` and updated the workflow contract test to require it. | `pnpm --filter @runfusion/fusion exec vitest run src/__tests__/ci-workflow.test.ts --silent=passed-only --reporter=dot` and `pnpm build` |
| TSH-012 | Warning filtering | The first SQLite warning filter also installed a `process.on("warning")` listener that could duplicate unrelated warnings. | Node still prints non-filtered warnings by default, so manually writing them again made future warning output noisier. | The shared setup now only wraps `process.emitWarning` for the known SQLite experimental warning, and the dashboard noisy-output marker list no longer suppresses generic trace-warning guidance. | `pnpm --filter @fusion/core typecheck`, `pnpm --filter @fusion/core exec vitest run src/__tests__/central-db.test.ts --silent=passed-only --reporter=dot`, and `pnpm test:full` |
| TSH-013 | Dashboard default runtime | The default dashboard package test still took about 9 minutes after the noise cleanup. | `pnpm --filter @fusion/dashboard test` continued to run every app/jsdom and API/node file, including exhaustive modal/view permutations and broad route matrices intended for deeper sweeps. | Added curated `dashboard-app-quality` and `dashboard-api-quality` Vitest projects for the default package gate, kept exhaustive coverage behind `test:deep`, `test:app`, and `test:api`, and documented when to run each lane. | `/usr/bin/time -p pnpm --filter @fusion/dashboard test` (148 files, 3642 tests, `real 91.73`), `/usr/bin/time -p pnpm --filter @fusion/dashboard test:deep` (419 files, 10747 tests, `real 335.47`), and `/usr/bin/time -p pnpm test:full` (`real 308.34`) |

View File

@@ -45,7 +45,7 @@ pnpm dev # build + run CLI entrypoint in dev mode
pnpm dev:ui # dashboard dev server only
pnpm lint # lint all packages
pnpm test # changed-only workspace tests (falls back to full suite in safety contexts)
pnpm test:full # full workspace test suite (clean-worktree compatible)
pnpm test:full # full workspace quality gate (clean-worktree compatible)
pnpm build # workspace builds (excludes desktop/mobile)
pnpm build:all # full workspace build (includes desktop/mobile)
pnpm verify:workspace # canonical lint -> test -> build verification gate
@@ -68,10 +68,10 @@ Fusion codifies workspace verification as a deterministic contract:
GitHub Actions now runs deterministic test sharding via `pnpm test:ci:shard --shard <index> --total <count>` in both PR checks and manual CI, while keeping local semantics unchanged:
- `pnpm test` remains changed-only local iteration.
- `pnpm test:full` remains the canonical full local suite.
- `pnpm test:full` remains the canonical workspace quality gate; dashboard exhaustive coverage is explicit via `pnpm --filter @fusion/dashboard test:deep`.
- `pnpm verify:workspace` remains the canonical local lint -> test -> build gate.
`test:ci:shard` is a CI-focused entrypoint (`scripts/ci-test-shard.mjs`) that partitions a fixed package list by shard index modulo total shard count so coverage is deterministic and reproducible.
`test:ci:shard` is a CI-focused entrypoint (`scripts/ci-test-shard.mjs`) that partitions workspace packages with `test` scripts by shard index modulo total shard count so coverage is deterministic and reproducible.
`pnpm test` now uses a changed-only entrypoint (`scripts/test-changed.mjs`) for faster local iteration. It resolves the comparison base from `.changeset/config.json` (`baseBranch`) and runs only affected workspaces from `pnpm-workspace.yaml` (both `packages/*` and `plugins/**`) using safe package-first filtering (`pnpm --filter <pkg> test`). It automatically falls back to the full suite when the run is forced (CI / `--full`), the git comparison base or diff cannot be resolved, no changes are detected, shared/root test infrastructure changes, or changed workspace paths cannot be resolved to a workspace package (fail-safe coverage behavior).
@@ -159,6 +159,21 @@ pnpm --filter @runfusion/fusion test:extension-integration
`test:extension-integration` enables `FUSION_TEST_EXTENSION_INTEGRATION=1` and runs the full fn pi extension integration suite. It remains an explicit opt-in lane so default workspace verification stays fast, while still providing a discoverable command for full extension-tool integration coverage.
## Dashboard Test Lanes
Dashboard tests are split into explicit local lanes. The default dashboard package gate is a curated quality gate that keeps representative app/API coverage without running every exhaustive modal, view, and route permutation on every local or PR pass:
```bash
pnpm --filter @fusion/dashboard test # curated app/API quality gate
pnpm --filter @fusion/dashboard test:deep # exhaustive app + API suite
pnpm --filter @fusion/dashboard test:app # exhaustive React/jsdom app tests
pnpm --filter @fusion/dashboard test:api # exhaustive Node API/server tests
pnpm --filter @fusion/dashboard test:browser-smoke # local browser layout smoke
pnpm --filter @fusion/dashboard test:build # built client output contract
```
Use the default lane for normal local iteration before PRs. Run `test:deep` when changing broad dashboard architecture, shared modal/view infrastructure, or route registration behavior where the exhaustive permutations are still useful. The built-client contract remains a separate lane because it performs its own production build. `pnpm build` remains an explicit PR gate, and PR test shards avoid a redundant pre-test workspace build to save GitHub Actions minutes.
## Release Process
Fusion uses Changesets + version PR workflow.

View File

@@ -1,6 +1,6 @@
# Skipped Test Inventory
_Last audited: 2026-04-23 (FN-2346)_
_Last audited: 2026-05-05_
This document tracks intentional skip usage in test suites so stale follow-up backlog items can be retired quickly.
@@ -9,18 +9,13 @@ This document tracks intentional skip usage in test suites so stale follow-up ba
Audit commands:
```bash
rg -n "\b(it|test|describe)\.skip\b|\bskipIf\b|createLoopbackIntegrationTest\(" packages --glob "**/*.{test,spec}.{ts,tsx}"
rg -n "\?\s*it\s*:\s*it\.skip|detectLoopbackBinding" packages/dashboard/src --glob "**/*.{test,spec}.{ts,tsx}"
rg -n "\b(it|test|describe)\.skip\b|\bskipIf\b|createLoopbackIntegrationTest\(" packages plugins scripts --glob "**/*.{test,spec}.{ts,tsx,mjs}"
rg -n "\?\s*it\s*:\s*it\.skip|detectLoopbackBinding|const itPosix" packages plugins scripts --glob "**/*.{test,spec}.{ts,tsx,mjs}"
```
Current results:
1. **Intentional cross-coverage alias**
- `packages/engine/src/executor.test.ts`
- `it.skip("step-session skill selection covered in step-session-executor.test.ts", ...)`
- Rationale: dedicated coverage exists in `step-session-executor.test.ts`; this marker documents ownership.
2. **Environment-gated integration aliases (loopback gated)**
1. **Environment-gated integration aliases (loopback gated)**
- Canonical helper: `packages/dashboard/src/__tests__/loopback-integration-test.ts`
- Helper consumers:
- `packages/dashboard/src/server-static-assets.test.ts`
@@ -30,18 +25,60 @@ Current results:
- Rationale: these suites require real loopback binding support (`127.0.0.1`) and are intentionally environment-gated.
- Auditability: when loopback binding is unavailable, skipped test names include a standardized reason and the suite scope label (`...; scope: <suite scope>`), making coverage gaps explicit in CI/test output.
3. **Build-output checks are now deterministic (no skip gate)**
2. **CLI slow-lane agent export gate**
- `packages/cli/src/commands/__tests__/agent-export.test.ts`
- Pattern: `describe.skipIf(!SHOULD_RUN_SLOW_CLI)("agent-export", ...)`
- Gate: `FUSION_TEST_SLOW_CLI=1` or `FUSION_TEST_SLOW_CLI=true`
- Rationale: these tests perform real workspace and `AgentStore` round-trips and are kept out of the default CLI unit lane.
- Replacement owner: the explicit slow lane (`pnpm --filter @runfusion/fusion test:slow-cli`) is responsible for running them when slow CLI coverage is requested.
3. **CLI pi extension integration gate**
- `packages/cli/src/__tests__/extension-integration.test.ts`
- Pattern: `describe.skipIf(!SHOULD_RUN_EXTENSION_INTEGRATION)("built fn pi extension integration", ...)`
- Gate: `FUSION_TEST_EXTENSION_INTEGRATION=1` or `FUSION_TEST_EXTENSION_INTEGRATION=true`
- Rationale: this built-extension suite requires compiled CLI artifacts, so it is run through the explicit local release lane instead of every unit run.
- Replacement owner: `pnpm --filter @runfusion/fusion test:extension-integration`.
4. **CLI legacy pi extension gate**
- `packages/cli/src/__tests__/extension.test.ts`
- Pattern: `describe.skipIf(!SHOULD_RUN_LEGACY_EXTENSION_INTEGRATION)("fn pi extension (legacy exhaustive suite)", ...)`
- Gate: `FUSION_TEST_LEGACY_EXTENSION_INTEGRATION=1` or `FUSION_TEST_LEGACY_EXTENSION_INTEGRATION=true`
- Rationale: this exhaustive suite is intentionally excluded from default and release lanes while it remains useful only for historical debugging.
- Replacement owner: `packages/cli/src/__tests__/extension-integration.test.ts` owns maintained built-extension coverage.
5. **CLI native binary build gate**
- `packages/cli/src/__tests__/build-exe-cross.test.ts`
- Pattern: four `describe.skipIf(!SHOULD_RUN_BUILD_EXE)(...)` suites for single-target, Windows-target, all-target, and default-target binary builds.
- Gate: `FUSION_TEST_BUILD_EXE=1`, `FUSION_TEST_BUILD_EXE=true`, or `CI=true`
- Rationale: cross-compiling native binaries is intentionally expensive and belongs in the binary/pre-release lane, not every local unit run.
- Replacement owner: `packages/cli/src/__tests__/build-exe.test.ts` and bundle-output tests cover the default fast package contract; `build-exe-cross.test.ts` owns full cross-target coverage when the gate is enabled.
6. **POSIX-only shell syntax cases**
- `packages/engine/src/__tests__/run-verification-command.test.ts`
- `packages/engine/src/__tests__/verification-utils.test.ts`
- Pattern: `const itPosix = onPosix ? it : it.skip`
- Gate: skipped only on `process.platform === "win32"`
- Rationale: a subset of tests uses POSIX shell syntax (`printf`, pipes, and shell quoting). The implementation still uses Node's portable `shell: true`; these specific fixtures are not portable to `cmd.exe`.
- Replacement owner: platform-neutral cases in the same suite continue to run on Windows.
7. **Build-output checks are deterministic (no skip gate)**
- `packages/cli/src/__tests__/bundle-output.test.ts`
- `packages/dashboard/app/__tests__/build-output.test.ts`
- Pattern: each suite builds required artifacts in `beforeAll` and then runs chunking/bundle assertions unconditionally.
- Rationale: clean worktrees and CI environments should execute real output-contract assertions instead of silently skipping when `dist/` is absent.
8. **Skip gate string assertions are not skip markers**
- `packages/cli/src/__tests__/ci-workflow.test.ts`
- Pattern: assertions check that CI workflow text contains `describe.skipIf(...)` gate strings.
- Rationale: this file does not skip tests itself; it verifies the workflow keeps the intentionally gated suites wired.
## Older Follow-up Reconciliation
Previously tracked actionable skip follow-ups are now resolved and should not be treated as open backlog:
- **FN-2085**: wildcard proxy POST body forwarding coverage is active.
- **FN-2076 / FN-2106 / FN-2109**: NewAgentDialog and MissionInterviewModal rollback favorite-toggle regressions are active interaction tests.
- The former engine `executor.test.ts` step-session alias skip is no longer present; ownership now lives in active engine tests.
Searches for those IDs in repository test code and docs now return no active TODO/skip markers tied to unresolved work.

View File

@@ -0,0 +1,341 @@
# Testing Suite Quality PRD
_Created: 2026-05-05_
## Summary
Fusion has a large, useful, but uneven test suite. The suite is strongest where it exercises real SQLite state, git operations, published CLI bundle contracts, route auth/lease semantics, runtime orchestration, and user-visible mobile/native flows. It is weakest where it accumulates high-volume mechanical assertions: field-presence checks, CSS source-string checks, parser/regex matrices, type-shape runtime tests, class-name layout checks, and large mock-heavy component shells.
This PRD defines the work needed to make the suite smaller, more trustworthy, and easier to run correctly. The goal is not to chase a smaller test count for its own sake. The goal is to preserve high-signal regression coverage while cutting tests that lock implementation details or duplicate coverage already provided by better tests.
## Current State
Static inventory from this worktree:
| Area | Source files | Test files | Test cases | Assertions | Test LOC |
|---|---:|---:|---:|---:|---:|
| `packages/dashboard` | 377 | 420 | 10,608 | 26,490 | 244,069 |
| `packages/engine` | 89 | 105 | 3,191 | 7,250 | 83,201 |
| `packages/core` | 88 | 96 | 3,470 | 8,474 | 57,270 |
| `packages/cli` | 57 | 59 | 1,031 | 2,427 | 24,510 |
| Other packages/plugins/scripts | 104 | 92 | 1,004 | 2,046 | 18,123 |
The suite has good infrastructure foundations:
- `pnpm test` runs `scripts/test-changed.mjs`, selecting changed packages and using a cache.
- `pnpm test:full` runs workspace tests with capped worker fanout.
- `pnpm test:locked` exists for cross-worktree contention.
- `pnpm test:isolated` checks temp-directory cleanup.
- Core/dashboard/engine/CLI share test isolation helpers and worker budgeting in most places.
- Dashboard already separates Node server tests from jsdom UI tests.
Key gaps:
- Local changed-test selection does not include reverse dependents.
- The changed-test cache can miss dirty working-tree edits.
- PR CI shards only a hard-coded subset of packages and omits plugin tests and `@fusion/pi-llama-cpp`.
- Coverage is available but not used as a meaningful gate for critical code.
- Several configs do not use shared worker budgeting or isolation.
- Skip inventory is stale.
- The largest files are now hard to reason about and invite low-value additions.
## Goals
1. Keep high-signal tests that guard user-visible behavior, data durability, release artifacts, and concurrency safety.
2. Cut or consolidate low-signal tests that only assert implementation shape.
3. Add missing tests around release-critical and runtime-critical gaps.
4. Make local and CI test commands honest about what they cover.
5. Establish governance so future tests improve signal rather than re-growing mechanical coverage.
## Non-Goals
- Do not rewrite the test framework.
- Do not remove high-value regression tests just because they are long.
- Do not require browser E2E coverage for every UI path.
- Do not make full coverage thresholds block all packages immediately.
- Do not move desktop/mobile into the default root build unless a separate decision is made for platform cost.
## Quality Bar
A test is high-value when it catches a regression a user, release manager, or agent operator would notice:
- state persisted incorrectly,
- task/agent workflow moved to the wrong state,
- user WIP was staged or merged incorrectly,
- published CLI bundle cannot run after npm install,
- auth/lease/security behavior changed,
- mobile/native or desktop flow breaks,
- dashboard feature becomes unreachable,
- process/runtime cleanup fails.
A test is low-value when it mostly asserts:
- one field exists after another test already round-trips the payload,
- a TypeScript type shape at runtime,
- exact CSS declaration text,
- a class name unrelated to behavior,
- every regex/parser permutation with no new behavioral boundary,
- mocked shell wiring already covered by command-specific tests.
## What To Keep
Keep and protect these categories.
### Core
- SQLite initialization, WAL, transaction/savepoint, migration, FTS5, and verification-cache tests in `packages/core/src/__tests__/db.test.ts`.
- Store tests that cover write-lock serialization, SQLite-first reads when blobs are corrupt/missing, dependency/status transitions, archive/unarchive recovery fields, and FN-ticket regressions in `packages/core/src/__tests__/store.test.ts`.
- Agent checkout leasing, conflict, claim, runtime state, and budget tests in `packages/core/src/__tests__/agent-store.test.ts`.
- Environment-sensitive FTS5/SQLite guards such as `packages/core/src/__tests__/fts5-guard.test.ts`.
### Engine
- Executor tests covering worktree recovery, stale refs, pause/resume, model hot-swap, workflow rerun, verification gates, and FN-ticket regressions.
- Restart/recovery integration tests in `packages/engine/src/__tests__/restart.integration.test.ts`.
- Real-git merger staging allowlist tests in `packages/engine/src/__tests__/merger-staging-allowlist.test.ts`.
- Heartbeat scheduler/monitor tests for pause, budget, timer triggers, checkout conflicts, and concurrency.
- Runtime adapter tests that exercise real process lifecycle or IPC semantics.
### Dashboard
- Published-bundle regression guard for static `@fusion/engine` imports in `packages/dashboard/src/__tests__/engine-import-regression.test.ts`.
- Auth and daemon-token integration tests using real `createServer`.
- Remote node sync route contracts, including API key authorization.
- Checkout lease route tests returning user-visible `409 Conflict`.
- Real-ish store/SQLite route tests for insights, tasks, file diffs, and project scoping.
- Hook tests for real race conditions such as stale backend hydration overriding local user choice.
- Product-level mobile feature reachability tests.
### CLI, Desktop, Mobile, Plugins
- CLI bundle-output tests that verify no bare `@fusion/*` imports and validate staged assets.
- Binary smoke tests for isolated no-`package.json` execution, `--help`, dashboard startup on port `0`, and PTY session creation.
- CLI package dependency guards that catch clean-install failures hidden by pnpm hoisting.
- Desktop IPC/bootstrap tests for first-run local/remote flows and shell channels.
- Mobile native behavior tests for deep links, share, push notifications, QR/native-shell boundaries.
- Pi adapter protocol tests for stream lifecycle, spawned flags, internal tool filtering, and MCP config.
- Runtime plugin tests that assert subprocess/output semantics.
## What To Cut Or Consolidate
### P0 Cut Candidates
These should be pruned first because they consume high maintenance time with low regression value.
1. CSS source-string tests that assert exact declarations.
- Examples: `core-modals-mobile.test.tsx`, `board-mobile.test.tsx`, `mobile-css.test.tsx`, `TodoView.mobile-css.test.ts`.
- Keep a few token/breakpoint guardrails and replace the rest with browser-level layout smoke tests.
2. Runtime TypeScript shape tests.
- Example: `packages/plugin-sdk/src/__tests__/index.test.ts`.
- Keep `definePlugin` identity and real manifest validation behavior. Move type coverage to typecheck or compile-time examples.
3. Redundant API wrapper URL/header tests.
- Examples: broad wrapper suites in `packages/dashboard/app/__tests__/api-settings.test.ts` and `api-tasks.test.ts`.
- Keep auth/header/error parsing and streaming edge cases. Consolidate to one happy-path per wrapper family plus route contract tests.
4. Mocked command-router smoke coverage.
- Example: `packages/cli/src/__tests__/bin.test.ts`.
- Keep a small representative routing table. Let command-specific tests own behavior.
### P1 Consolidation Candidates
1. Settings field/default assertions across core.
- Consolidate overlap between settings parity, global settings, and store settings blocks.
- Keep precedence, migration, and round-trip behavior.
2. Regex/parser/error detector matrices.
- Examples: context-limit, transient-error, reconcile-step regex, stream parser.
- Keep provider examples, boundaries, and precedence. Convert linear permutations to compact `it.each`.
3. Plugin example manifest/schema tests.
- Keep one manifest smoke and real hook/tool behavior per example.
- Drop repeated static schema assertions already covered by SDK validation.
4. Duplicate desktop integration smoke.
- Fold the smaller `main.integration.test.ts` checks into the richer `main-integration.test.ts`.
5. Huge component suite structural assertions.
- Split and prune `TaskDetailModal.test.tsx`, `App.test.tsx`, `GitManagerModal.test.tsx`, `ModelOnboardingModal.test.tsx`, `AgentDetailView.test.tsx`.
- Keep behavior paths; cut class-name/layout internals unless they guard a documented regression.
## What To Add
### P0 Additions
1. Reverse-dependent changed-test selection.
- If `@fusion/core` changes, run core plus engine, dashboard, CLI, plugin-sdk, and packages importing it.
- If dashboard or engine changes, run CLI because CLI aliases those source entries for bundle tests.
2. Dirty-worktree cache safety.
- Disable the changed-test cache when affected files are dirty, or include working-tree content in package hashes.
- `pnpm test` must not report cached success for files that have never been tested.
3. CI package coverage parity.
- Generate CI shard package lists from workspace packages with `test` scripts.
- Include plugin packages and `@fusion/pi-llama-cpp`, or run a dedicated plugin test lane.
4. Critical release/tarball contract test.
- Add an `npm pack --dry-run` or equivalent assertion for `@runfusion/fusion`.
- Required contents: `dist/bin.js`, `dist/client/**`, `dist/pi-claude-cli/**`, `dist/droid-cli/**`, `dist/plugins/**`, and `skill/**`.
- Excluded contents: standalone binary outputs and unnecessary runtime build artifacts.
5. Browser-level dashboard smoke lane.
- Cover mobile nav, header overflow, one modal, board/list switch, and footer/input behavior.
- This should use a real browser because jsdom cannot prove layout overflow or fixed-position behavior.
### P1 Additions
1. Critical coverage lane.
- Add `coverage:critical` for targeted files from `docs/test-audit-report.md`, not the entire recursive workspace.
- Initial targets: engine runtime/verification utilities, CLI project resolver/settings import-export, dashboard mission/import/generation routes/components, core `gh-cli`/summary utilities.
2. Engine runtime/process tests.
- Add direct tests for `verification-utils.ts`, especially process-group timeout, abort, and buffer overflow behavior.
- Add direct tests for `custom-providers.ts` malformed/missing settings and valid arrays.
- Add a small adapter test for `task-completion.ts` dependency lookup failures.
- Add SQLite adapter constructor selection coverage if release binaries depend on it.
3. Dashboard route-store integration contracts.
- Add real `TaskStore`/SQLite route-store passes for tasks, settings, nodes, and project scoping.
- Reduce fake-store route tests where equivalent real-store tests exist.
4. Frontend/backend API contract tests.
- For important wrappers, call exported frontend API functions against an in-process route server.
- This replaces separate fetch-string and route-mock assertions.
5. Accessibility smoke.
- Add role/focus/escape coverage for major modals and command surfaces.
6. Pi/plugin integration repairs.
- Replace stale gated CLI pi extension integration with a smaller current integration test against the built extension entry.
- Add a bundled plugin loader integration test that verifies dashboard view/runtime metadata registration.
- Add `pi-llama-cpp` tests for unreachable/malformed server responses and model listing.
7. Desktop/mobile build smoke lanes.
- Add non-default CI lanes for desktop build and mobile sync/build smoke.
- Keep them outside fast PR checks unless changed-file detection selects those packages.
## Run Model
### Local Development
Use the narrowest command that honestly covers the changed surface.
| Situation | Command |
|---|---|
| Package-local change | `pnpm --filter <package> test` |
| Dirty worktree using changed-test path | `pnpm test --no-cache` |
| Shared core/engine/dashboard/CLI change | `pnpm test:full` |
| Multiple worktrees or resource contention | `pnpm test:locked` |
| Isolation audit | `pnpm test:isolated` |
| Dashboard bundle/client-dist change | `pnpm --filter @fusion/dashboard test:build` |
| Published CLI/release-sensitive change | `pnpm --filter @runfusion/fusion test && pnpm --filter @runfusion/fusion test:pre-release && pnpm build` |
| Extension behavior changed | `pnpm --filter @runfusion/fusion test:extension-integration` after repairing stale integration |
| Desktop change | `pnpm --filter @fusion/desktop test typecheck build` |
| Mobile change | `pnpm --filter @fusion/mobile test typecheck` plus relevant Capacitor sync/build lane |
### PR CI
Target state:
1. Install and build once per shard where needed.
2. Run lint and typecheck.
3. Run deterministic test shards generated from all workspace packages with `test` scripts.
4. Split dashboard into separate lanes:
- dashboard app/jsdom,
- dashboard server/routes,
- dashboard slow process/WebSocket/browser smoke.
5. Run plugin package tests.
6. Run `@fusion/pi-llama-cpp` tests.
7. Run `coverage:critical` only on critical target files.
### Release CI
Target state:
1. Run `pnpm verify:workspace`.
2. Run CLI pre-release tests and binary build smoke.
3. Run tarball contract test.
4. Run plugin loader integration.
5. Use the repository release script for real releases: `pnpm release --yes`.
## Governance Rules
1. New tests should cover observable behavior, not implementation shape.
2. New CSS tests should prefer rendered/computed behavior over CSS source text.
3. New parser/regex coverage should be table-driven and limited to representative boundaries.
4. New high-volume tests must name the regression or user-visible contract they protect.
5. Tests tied to FN-ticket regressions are kept unless a better test explicitly replaces them.
6. Every package with a `test` script must either use shared worker budgeting and isolation or be explicitly allowlisted with a reason.
7. Skip markers must be documented in `docs/skipped-test-inventory.md` with an environment gate or replacement coverage owner.
8. Large suites should not grow further; new behavior areas get focused files under `__tests__/`.
## Success Metrics
Within one cleanup cycle:
- Reduce dashboard test LOC by at least 15% without removing listed high-value categories.
- Reduce top five largest test files by moving or pruning at least 20% of their structural assertions.
- CI shards include every workspace package with a `test` script.
- `pnpm test` is dirty-worktree safe.
- `docs/skipped-test-inventory.md` matches current skip/gate usage.
- `coverage:critical` exists with thresholds for selected critical files.
- Browser smoke lane covers the primary dashboard layout risks.
Longer term:
- Keep total test LOC growth below source LOC growth.
- Keep new skipped tests at zero unless they are environment-gated and documented.
- Track flake rate for process/WebSocket/browser lanes separately from deterministic unit lanes.
## Implementation Plan
### Phase 1: Make Test Runs Honest
- Fix changed-test reverse dependency selection.
- Make changed-test cache dirty-worktree safe.
- Generate CI test shards from workspace package metadata.
- Add plugin and `pi-llama-cpp` tests to PR coverage.
- Apply shared worker budgeting/isolation to `droid-cli`, `pi-claude-cli`, and `pi-llama-cpp`.
- Refresh skipped-test inventory.
### Phase 2: Add Missing High-Value Coverage
- Add CLI tarball contract test.
- Add critical engine runtime/process tests.
- Add dashboard browser smoke lane.
- Add route-store integration contracts for tasks/settings/nodes/project scoping.
- Repair CLI pi extension integration.
- Add bundled plugin loader integration.
### Phase 3: Prune Low-Value Coverage
- Cut CSS source-string assertions and replace with browser/computed smoke.
- Consolidate dashboard API wrapper tests.
- Consolidate core settings field/default checks.
- Consolidate parser/regex matrices.
- Trim runtime type-shape tests from plugin SDK.
- Merge duplicate desktop integration smoke.
### Phase 4: Enforce Governance
- Add governance test for test package config consistency.
- Add `coverage:critical`.
- Document run lanes in contributor docs.
- Require PRs adding broad test files to identify the behavior contract they protect.
## Risks
- Pruning can accidentally remove real regression coverage if done mechanically. Mitigation: only cut tests after mapping each to a better owner or identifying it as implementation-shape coverage.
- Browser smoke adds CI cost. Mitigation: keep the lane tiny and targeted.
- Reverse-dependent changed-test selection will run more tests locally. Mitigation: make the behavior explicit and preserve `--filter` escape hatches for expert use.
- Coverage thresholds can create churn if applied globally. Mitigation: start with critical files only.
## Open Questions
- Should dashboard browser smoke use Playwright in CI, or the existing in-app/browser tooling only for local verification?
- Should process/WebSocket tests be tagged in filenames or moved into separate `__tests__/slow/` directories?
- What minimum critical coverage thresholds should apply initially per package?
- Should plugin examples be treated as product tests, documentation tests, or both?