## Summary
Starts **U5** of the package code-organization program after wave 18
(executor peels) landed.
Peels pure free-function clusters out of `self-healing.ts` into
`packages/engine/src/self-healing/` without behavior changes. Public
imports from `./self-healing.js` remain stable via re-exports.
### Peels
| Symbol | New home |
|--------|----------|
| `autoRecoverWorktreeSessionStartFailure` |
`self-healing/auto-recover-worktree-session.ts` |
| `archiveAsGhostBug` | `self-healing/archive-ghost-bug.ts` |
| `hasStepProgress` / work-complete helpers |
`self-healing/step-progress.ts` |
### Line count
- `self-healing.ts`: ~15456 → ~15231 (baseline ratcheted to post-peel
live; main had already drifted past the prior grandfathered ceiling via
organic growth)
- New modules each well under 2,000 lines
## Test plan
- [x] `pnpm --filter @fusion/engine exec tsc --noEmit`
- [x] `self-healing-trait-rekey.test.ts` (autoRecover requeue)
- [x] `self-healing-paused-abort-recovery.test.ts`
- [x] `self-healing-model-unavailable-recovery.test.ts`
- [ ] CI gate
## Follow-ups
U5 Slice B: domain method clusters (startup, in-review, merge-status,
workspace, surfacing) into additional `self-healing/*.ts` modules.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved automatic recovery when worktree sessions fail to start,
including stale or incomplete session data.
* Tasks can be safely requeued while preserving progress, or escalated
after retry limits are reached.
* Improved handling of completed work and failures where task completion
was not recorded.
* Preserved valid task branches during recovery and provided more
reliable fallback requeue behavior.
* Ghost bugs are automatically archived with recovery details and
activity history.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Supersedes #3421 — fork head not writable for main-merge. Same change
rebased onto current main so lifecycle-column lint stays green.
## Test plan
- [x] merges cleanly onto main
- [ ] CI green
Co-authored-by: BESA-Franz <49682134+BESA-Franz@users.noreply.github.com>
## Problem
`app/components/__tests__/navigation-history.test.tsx`,
`TaskDetail.swipe-back.test.tsx`, and
`TaskDetail.mobile-transition.board-panel.test.tsx` fail to load.
FN-8796 (`132026545`, "stabilize task-detail lifecycle snapshots") added
`mergeTaskSnapshot` to `hooks/useTasks.ts`, which
`App.tsx`/`MainContent.tsx` now import. The curated test suites that
`vi.mock("../../hooks/useTasks")` still only expose `useTasks`, so the
fixture throws at import time:
```
No "mergeTaskSnapshot" export is defined on the useTasks mock
```
Confirmed on `origin/main`: `mergeTaskSnapshot` is present in
`useTasks.ts` and imported by `App.tsx`, while these tests' `vi.mock`
shims don't surface it.
## Fix
Update the mocks to surface every export the rendered components import,
using a partial `importOriginal` mock so `mergeTaskSnapshot` keeps its
real implementation (a `vi.fn()` returning `undefined` would make
`App.tsx:2179` `liveTask.id` throw):
```ts
vi.mock("../../hooks/useTasks", async (importOriginal) => {
const actual = await importOriginal();
return { ...actual, useTasks: (_options?: any) => mockUseTasks() };
});
```
## Verification
- `FUSION_DASHBOARD_DEEP=1 vitest run navigation-history swipe-back
board-panel` → **14 files, 116 tests passed**
- Pre-existing on `origin/main`, fixes a regression introduced by
FN-8796.
## Scope
Tests + a one-line docs note only. No production behavior change.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Documentation**
- Documented improved worktree recovery behavior, including safer
handling of incomplete, inactive, and unregistered directories.
- Added guidance for recovery across different storage locations and
bounded cleanup of recognized entries.
- Clarified protections that prevent internal recovery containers from
being treated as worktrees.
- **Tests**
- Improved dashboard test coverage and reliability for task detail
rendering, mobile transitions, swipe-back navigation, and navigation
history.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Fusion <noreply@runfusion.ai>
Co-authored-by: gsxdsm <gsxdsm@users.noreply.github.com>
## Summary
- make the already-merged recovery fixture explicit that no maintenance
sweep owns terminal-failure recovery
- preserve the fail-open `task-wedged` notification assertion when no
landed commit is found
## Test plan
- `corepack pnpm --filter @fusion/engine exec vitest run
--silent=passed-only --reporter=dot src/__tests__/self-healing.test.ts`
(460 passed)
- `corepack pnpm --filter @fusion/engine typecheck`
- `node scripts/run-static-gate-checks.mjs` (12 passed)
- `corepack pnpm --filter @fusion/engine test:core` (422 passed)
- `corepack pnpm --filter @fusion/core test:unit-gate` (184 passed)
- `corepack pnpm --filter @runfusion/fusion test:ci-shape` (71 passed)
- `corepack pnpm test:gate` reaches the PostgreSQL harness but cannot
authenticate locally (`empty password returned by client`) before
product assertions
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Tests**
* Updated failure-notification test conditions to cover scenarios
without automatic maintenance processing.
* Added coverage confirming terminal-failure notifications fail open
when no maintenance sweep owns the task.
* Bypassed production debounce timing to make the test behavior
deterministic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
Supersedes #3416 — the fork head is not writable from maintainers, so
this branch carries the same fix merged onto current `main`.
## Changes
- Bind the self-owned worktree liveness probe in task-executor pure
facades (from #3416).
- Merge current main so the lifecycle-column census baseline stays
green.
## Test plan
- [x] `pnpm check:lifecycle-columns` clean after merge
- [ ] CI green
Closes context from #3416.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Fixed cleanup for self-owned worktrees so it no longer crashes before
checking whether an active session is present.
* Improved safety during worktree reconciliation by correctly verifying
that the worktree is still active before removal.
* **Documentation**
* Added a changeset documenting the worktree cleanup fix.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: BESA-Franz <49682134+BESA-Franz@users.noreply.github.com>
## Summary
- Complete the isolated `@fusion/core` mock used by the
experiment-finalize extension suite
- Classify three intentional physical/synthetic lifecycle literals
introduced on current main
- Re-record the strict lifecycle census baseline with zero unexamined
guards
## Test plan
- `pnpm --filter @runfusion/fusion exec vitest run
src/__tests__/extension-experiment-finalize.test.ts --silent=passed-only
--reporter=dot`
- `pnpm --filter @fusion/core exec vitest run
src/__tests__/task-intake-owner-resolver.test.ts --silent=passed-only
--reporter=dot`
- `pnpm --filter @fusion/engine exec vitest run --project engine-default
src/__tests__/mission-feature-sync-lanes.test.ts --silent=passed-only
--reporter=dot`
- `pnpm check:lifecycle-columns`
- `node scripts/check-mock-completeness.mjs`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved mission reconciliation previews for task links, specification
alignment, and lifecycle updates.
- Prevented stale or superseded validation runs from overwriting current
feature status or ownership.
- Improved blocked-feature diagnostics and archived-task handling across
workflow configurations.
- **Documentation**
- Clarified validation, assignment checks, and mission synchronization
behavior.
- **Tests**
- Expanded coverage for reconciliation previews and validator ownership
scenarios.
- **Chores**
- Updated lifecycle baseline data for known archived-task cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The planning-continuation drain skipped every due `kind: "task"` row whose
`waitReason` was not "planning", on the premise that such rows "belong to a
different drain". No such drain exists: `listDueWorkflowWorkItems` has exactly
two callers, this pass and the self-healing reclaim sweep, and the sweep
deliberately leaves `runnable`/`retrying` rows alone as "the dispatcher's own
queue". A capacity-parked continuation was therefore owned by nobody — skipped
here every poll with no state change and no audit row, and passed over there by
design.
Observed on the Fusion board: eight cards sat runnable for up to 8h with the
engine unpaused, 0 tasks in progress, and 4 of 10 worktrees used. Three carried
`waitReason: "capacity"` from the capacity-suspend path; five carried NULL. The
09:04 reclaim sweep had just moved them held -> runnable, handing them to this
drain and simultaneously putting them out of its own reach, so the auto-resume
fix tightened the strand it repaired.
Dispatch stays admission-gated by `admitPlanningContinuation`, so a
capacity-parked card resumes only when a slot is genuinely free.
Also repairs two stale path allowlists in planning-claim-single-writer.ts: the
mission stores and replan-target.ts moved into subdirectories, leaving that
ratchet red on main and accusing the two modules it exists to exclude.
Verified: the patched classifier returns `actionable` for all 8 live stranded
rows; gate + lint green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment cited 89 registered worktrees against 20 on disk as evidence of stale
registrations. That comparison was against .worktrees/ alone; all 89 registrations
exist, spread across kb-worktrees, orca workspaces and .claude/worktrees. Prune
still belongs before removal on ordering grounds -- a dangling registration fails
the removal, and the throw skips the trailing prune so the retry never clears it --
but the false measurement should not stand as the justification.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both suites are mutation-verified: reverting the corresponding fix fails 4 of the
chat tests and exactly the 3 new self-healing park tests.
The self-healing tests carry a positive control asserting the sweep actually
reached the tip-already-merged arm -- without it the park assertions passed
vacuously against a task shape the candidate filter rejected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two unrelated production failures with a shared symptom of an opaque error.
Chat: FN-8869 hoisted the agent-existence check out of its else branch, so it ran
even when the client supplied an explicit model pair. Model-target chats send the
client-only sentinel `__fn_agent__`, which is never an agent row, so every one of
them 404'd behind the generic "Failed to create chat session" toast. The agent is
now required only when it is the source of model resolution.
Self-healing: a failed `tip-already-merged` cleanup was rethrown and classified
`branch-conflict-unrecoverable`, failing and pausing tasks whose branch was already
an ancestor of the integration ref. Every one of the 78 logged parks carried a
`git worktree remove --force` / `ENOTEMPTY rmdir node_modules` message -- a pnpm
race, not a conflict. Cleanup failure now retries on the next sweep, and prune runs
before removal so a stale registration stops causing the failure it would prevent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independent wedges kept cards silently stuck on the board.
1. Workflow principals were capped. `WorkflowAgentCapacity.acquire` enforced
`settings.maxConcurrent` as a project session budget plus an optional
per-agent `maxWorkflowSessions`, and `routeWorkflowPrincipal`'s availability
test applied the same per-agent ceiling. The workflow roles stand in for
STAGES, not workers, and there is typically one agent per role - so the cap
serialized the entire board behind a single Workflow Executor regardless of
maxConcurrent/maxWorktrees. Admission now always succeeds; the lease survives
as bookkeeping (it is what activeSessions counts and what the renewal timer
keeps warm). `maxProjectSessions` is removed from the input rather than
defaulted, so it cannot be reintroduced without deleting the contract, and
the agent-capacity re-route loops in triage and graph admission are deleted
with the refusal they existed to work around.
2. Continuations that stop in `running` or `held` were never re-polled. The
scheduler's due-poll takes only `runnable`/`retrying`; a row claimed through
a path that leaves `leaseExpiresAt` NULL keeps `state: "running"` forever
after its process dies, and `acquireWorkflowWorkItemLease` can only re-take a
`held` row whose blockedReason matches workflow-principal-%. Observed live:
seven cards `running` behind leases from a process that exited ~9h earlier,
two `held` with a NULL blockedReason for 46h, none emitting a single
run-audit row while stranded. A further 33 active-state rows belonged to
archived+soft-deleted tasks (the FK cascade only fires on hard delete).
New sweep `reconcileStrandedWorkflowContinuations` (startup + periodic)
re-queues both stranded shapes and retires dead tasks' rows, gated by the
canonical liveness triple, a 10-minute grace matching the capacity lease
duration, and a compare-and-set on the scanned state so a real claim wins.
The decision is the pure `evaluateStrandedContinuationReclaim`, shared with
its tests so coverage cannot drift from behavior - the drift that let the
FN-8923 sweep ship covering one ninth of this problem.
Verified: pnpm lint, engine typecheck, pnpm test:gate (606 tests), verify:fast,
and the new suite under mutation (removing either guard fails 3 cases). The two
pre-existing failures in self-healing-orphaned-pending-step-results.test.ts
reproduce identically at HEAD without these changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
## What
Adds a typed, queryable catalogue for engine run-audit events as the
first step of the delivery-pipeline reliability & observability effort.
- **New module** `packages/engine/src/run-audit/run-audit-catalogue.ts`
— a typed registry describing run-audit event kinds (scheduler,
self-healing, merger, worktree, symbol-lock, …) so pipeline
observability can ingest and reason about them consistently.
- **Parity test** `run-audit-catalogue.test.ts` — asserts the catalogue
matches the emitted run-audit event space.
- **Docs** `docs/run-audit.md` + index pointer.
## Why
Run-audit events are currently emitted ad-hoc without a typed contract.
A catalogue gives:
- a single source of truth for event kinds/names,
- a parity guard so any new or renamed event is caught,
- a foundation for delivery-pipeline reliability dashboards.
## Verification
- `@fusion/engine` `tsc` build → **PASS**
- `vitest run run-audit-catalogue.test.ts` → **3 tests passed**
- No production behavior change outside the new module.
## Scope
New isolated module + its test + docs. No changesets/release artifacts.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Documentation**
* Added a run-audit catalogue covering delivery-pipeline finalization,
self-healing reconciliation, and durable-agent error events.
* Documented recorded outcomes, emission conditions, audit-store
querying, and event catalogue maintenance.
* Added a documentation index entry linking to the new catalogue.
* **Tests**
* Added validation to ensure documented audit events remain complete,
consistently formatted, and synchronized with the supported event
catalogue.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Fusion <noreply@runfusion.ai>
## Summary
Supersedes #3405 — the fork head is not writable from maintainers, so
this branch carries the same fix rebased/merged onto current `main`.
## Conflict resolution
- Main already landed the equivalent fail-open path as **FN-8919**
(`readLinkedTaskOrUndefined` + per-agent try/catch).
- Kept the additional `recoverAgentsRunningOnInactiveTasks` regression
that covers task-gone races plus transient lookup isolation.
- Dropped the duplicate changeset (main already has
`fn-8919-agent-link-sweep-fail-open`).
## Test plan
- [x] `git merge-tree` clean against `main`
- [ ] CI green
Closes context from #3405.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved recovery handling when task lookups fail.
* Agents linked to deleted or missing tasks are now unlinked, while
agents affected by temporary errors remain preserved.
* Recovery continues for other eligible agents instead of stopping after
an individual lookup failure.
* **Tests**
* Added regression coverage for deleted, missing, and temporarily
unavailable tasks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Codex <codex@openai.com>
Convert the SettingsModal.general 'saves X via settings payload' tests from
real-timer waitFor (each burning a real ~500ms auto-save debounce) to the
FN-7506 fake-timer pattern already used by the auto-save tests in this file.
Payload-routing/coalescing assertions are unchanged; only the artificial
wall-clock wait is removed (Standing Rule: prefer fake timers over real time
waits). Isolated test-execution time drops ~21.7s -> ~15.0s (82 tests, all
green across repeated runs).
## Problem
Dashboard typecheck fails with **TS2416** in `@fusion/core`'s
`TaskStore`:
```
Property 'emit' in type 'TaskStore' is not assignable to the same property in base type 'EventEmitter<TaskStoreEvents>'.
```
The `override emit<E extends string | symbol>(event, ...args)` generic
conflicts with the base class's generic `emit<K>(eventName: keyof
TaskStoreEvents | K, ...)`. This breaks the dashboard typecheck / CI
merge gate.
## Fix
Change the override to:
```ts
override emit(event: unknown, ...args: any[]): boolean {
return EventEmitter.prototype.emit.call(this, event as string, ...args);
}
```
`event: unknown` remains assignable to the base's generic signature
while still forwarding non-typed runtime keys (`agent:log`,
`settings:updated`, …). Internal `EventEmitter.prototype.emit` calls
cast `event as string`. Behavior-preserving.
## Verification
- `@fusion/dashboard` `tsc --noEmit` → **PASS** (previously failed with
TS2416)
- `eslint` on touched file → clean
- Single-file change (`packages/core/src/store.ts`, +6/−3)
## Scope
No behavior change, no changesets required.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved task event handling to support a broader range of event
identifiers.
* Preserved cached-lane information for single-argument task update
events.
* Maintained support for custom and arbitrary event names without
disrupting existing behavior.
* Improved classification of workflow roles, session purposes, and
outcome-related status checks in lifecycle analysis, producing more
accurate findings and reducing misleading results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fusion wedged on "starting" and never brought the engine up. The dashboard bound
the migration holding server on 4040, then every query behind it failed with
"sorry, too many clients already", so the card never progressed and the
supervisor crash-looped.
Root cause: each spec-drift reconcile costs a DEDICATED PostgreSQL connection.
persist -> appendSpecDriftReport -> withPlanningLifecycleLock opens its own
postgres(directUrl, { max: 1 }) session, because the planning advisory lock is
session-scoped and deliberately fences a stale report against a newer plan.
enqueue() released every id straight into its own microtask, and project-engine
enqueues every task at runtime-boundary setup (listTasks includeArchived). On a
1,082-task project that opened ~1,082 lock sessions simultaneously against
max_connections = 500. The cluster saturated ~25s into boot and stayed saturated.
The flat 1s retry then made it self-sustaining rather than transient: once
saturated, every task failed for the same shared reason and re-armed in lockstep
once per second, re-opening the whole fleet of sessions and pinning the very
resource it was waiting on. Measured 4,777 lock sessions in 17 seconds.
Fix, contained to the reconciler — the advisory lock and its fencing semantics
are load-bearing and unchanged:
- concurrency bound (maxConcurrent, default 4) drained by a fair
insertion-ordered pump, so fan-out can no longer exceed a known connection cost
- per-task in-flight dedupe; two passes on one task would contend on that task's
own advisory lock while holding two connections
- exponential backoff with jitter capped at 60s, and retries re-enter through
enqueue so a retry storm is bounded by the same limit as a first pass
Verified against the real 1,082-task project: connections stay flat at 3-10
across a 70s boot that previously reached 1,109 and saturated, and the engine
boots through to executing tasks and shuts down cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
