The periodic self-healing sweep at
SelfHealingManager.reconcileDoneTaskIntegrity() emits a single
'Integrity warning: done-task finalize evidence is unproven (<reason>)' log
entry per task when the task is in 'done' but has no provable on-main
evidence. Dedup was via an in-memory finalizeUnprovenWarned Set per manager
instance, so every engine restart resurfaced the same warning on the next
sweep — significant noise on done tasks that legitimately lack evidence,
typically residue of FN-4811 contamination (FN-4771/FN-4778 in production).
Adds an optional MergeDetails.integrityWarning = { warnedAt, reason } field
and persists it on the first warning. Both warning sites in
reconcileDoneTaskIntegrity() (the unproven-and-still-mergeable branch and
the unproven-final branch) now consult the persisted record:
- Same reason as persisted → skip re-emitting, just rehydrate the in-memory
Set for in-process consistency.
- Different reason → re-warn (so a *new* classification problem still
surfaces) and update the persisted record.
Tests added under
packages/engine/src/__tests__/reliability-interactions/integrity-warning-persisted-dedup.test.ts
(real-git, 4 cases):
- First sweep: emits warning + persists record.
- Second sweep, same instance: in-memory Set dedupes (existing contract).
- Fresh manager (simulated engine restart) + pre-persisted record:
persisted dedup suppresses re-emission.
- Fresh manager + persisted record with different reason: must re-warn
and overwrite the persisted reason.
Full engine suite: 308 files, 5041 tests pass, 1 skipped. Lint clean.
Fusion-Task-Id: FN-4811
Two follow-ups stacked on the FN-4811 active-worktree liveness gate:
1. Stale conflict-path recovery (FN-4813 production failure)
When 'git worktree remove --force' fails with 'fatal: validation failed,
cannot remove working tree', the worktree directory is missing on disk
and the git admin entry is stale. Without this recovery, every retry of
tryCreateWorktree on a stale conflict path failed 3 times with
'automatic cleanup failed', leaving tasks unable to create worktrees.
cleanupConflictingWorktree now catches that specific error class, runs
'git worktree prune' to drop the stale admin entry, best-effort deletes
the branch, and returns success so the caller can proceed.
Implementation note: the original attempt used existsSync(worktreePath)
as a pre-check, but vitest's vi.clearAllMocks() can leave the existsSync
mock returning undefined, causing the new branch to fire inside tests
that didn't expect it and leading to worker OOM in
executor-worktree.test.ts. The error-class-based catch is robust against
mock state and matches the real production failure signal exactly.
2. Collapsed broken FN-4806 nested branches
The previous FN-4806 refactor (commit 087b1a766) accidentally nested the
genuine 'agent finished without calling fn_task_done after N retries'
failure path INSIDE the silent-recovery branch, meaning ordinary
failures were being silently requeued (no status=failed, no onError, no
retry-budget burn) instead of being surfaced.
Restored the clean two-branch structure:
} else if (retryAbortedDueToReclaim) {
// silent recovery (FN-4806)
} else {
// genuine no-fn_task_done exhaustion: mark failed, onError, burn budget
}
Also clears baseCommitSha on silent recovery (matches the parallel
session-start-failure path's metadata clearing).
Tests:
- Adds 'FN-4811 follow-up (FN-4813): recovers from validation failed'
case to active-worktree-removal-liveness.test.ts (12 total cases).
- executor-recovery.test.ts no-fn_task_done reclaim coverage now
asserts baseCommitSha is cleared.
- executor-recovery.test.ts 'does not mark task as failed when invalid
transition error occurs on completion' regression fixed by restoring
the failure-path branch.
- executor-core.test.ts 'still enforces fn_task_done requirement in
fast mode' restored.
Full engine suite: 307 files, 5037 tests pass, 1 skipped. Lint clean.
Fusion-Task-Id: FN-4811
The executor's conflict-recovery paths (cleanupConflictingWorktree,
handleBranchConflict, and tryCreateWorktree's live-foreign/stale-resolved
branches) could force-remove a worktree even when it was currently bound
to an active executor session. This caused the FN-4781/FN-4804 cascade:
- 'Execution blocked: assigned worktree path disappeared mid-task' as
git deleted the live agent's filesystem out from under it
- Two parallel runs for the same task alive simultaneously, with the
second run started in a fresh worktree while the first was still
holding the old session
- Cross-task log attribution (an FN-4804 runContext writing to FN-4781)
- Post-merge 'branch tip misbound but content found on main via trailer'
rescues firing on every successful merge as the bookkeeping was
corrupted mid-merge
Adds a hard liveness gate centralized in findActiveWorktreeOwner(), which
checks both the in-memory activeWorktrees map and the DB for non-done,
non-paused, in-progress tasks bound to the worktree. The gate fires at
two points:
1. cleanupConflictingWorktree returns false (refuses removal) when an
active owner is found, logging an FN-4811 refusal entry.
2. handleBranchConflict short-circuits to 'sticky' BEFORE invoking
inspectBranchConflict, because some inspection branches force-remove
unconditionally.
When cleanup is refused, the live-foreign and stale-resolved branches in
tryCreateWorktree now FALL THROUGH to the suffix-rename path (rather
than returning null) so the requesting task can still proceed without
disturbing the live owner.
Tests:
- New reliability-interactions backstop at
src/__tests__/reliability-interactions/active-worktree-removal-liveness.test.ts
covers findActiveWorktreeOwner (5 cases: in-memory match, requesting
task excluded, DB-level match, paused exclusion, terminal-column
exclusion, self-exclusion), cleanupConflictingWorktree gate (3 cases:
in-memory refuse, DB refuse, no-owner proceed), and handleBranchConflict
gate (2 cases: short-circuit + inspection-skipped, no-owner proceeds).
- Updates existing executor-worktree.test.ts assertion that was
documenting the bug behavior (force-removing active worktree) to match
the new contract (refuses + falls through to suffix-rename).
Full engine suite: 307 files, 5035 tests pass.
Fusion-Task-Id: FN-4811
The executor's no-fn_task_done retry loop has two reclaim signals
(retryAbortedDueToReclaim=true): (1) pre-retry liveness recheck where the
task DB shows worktree/branch was cleared, and (2) session-start failure
because the worktree path no longer exists. Both are engine self-heal
situations triggered by FN-4546 stale-active-branch reclaim, FN-4742
self-healing removals, or related housekeeping paths — the agent never
got a fair retry attempt.
Previously these surfaced as task status=failed with error
'Worktree/branch reclaimed during no-fn_task_done retry — requeueing',
fired onError, and burned the taskDoneRetryCount budget. Three legitimate
problems followed: tasks accumulated spurious failures in the UI, the
exhausted-budget branch escalated reclaimed tasks to in-review instead of
retrying, and the noise masked the underlying worktree-removal regression
(FN-4811).
Now the reclaim branch silently:
- clears stale worktree/branch metadata so the next pickup creates a fresh worktree
- requeues to todo with preserveProgress
- logs an informational 'engine self-heal, no failure' line
- does NOT set status=failed, does NOT bump taskDoneRetryCount, does NOT call onError
The genuine 'agent finished without calling fn_task_done after N retries'
exhaustion path (retryAbortedDueToReclaim=false) is unchanged.
Tests updated in
packages/engine/src/__tests__/reliability-interactions/executor-no-task-done-vs-worktree-reclaim.test.ts
to assert the new silent-recovery contract on all three reclaim paths.
Fusion-Task-Id: FN-4806
Adds test coverage for multi-node routing in the hybrid executor, specifically covering routing behavior across nodes (FN-4775 Step 3).
Fusion-Task-Id: FN-4775
Fusion-Task-Lineage: d75a91c9-0c6c-421c-8bc9-11586590d6fe
Adds parameterized test cases to the worktree-liveness test to exercise different configured `worktreesDir` paths, improving coverage of the worktree escape/isolation path.
Fusion-Task-Id: FN-4737
Fusion-Task-Lineage: 40e262ea-2507-461e-b9f5-940b3b4490f9
