Commit Graph

244 Commits

Author SHA1 Message Date
Fusion
f4aa6d7b8b fix(FN-4811): persist done-task integrity warnings across engine restarts
The periodic self-healing sweep at
SelfHealingManager.reconcileDoneTaskIntegrity() emits a single
'Integrity warning: done-task finalize evidence is unproven (<reason>)' log
entry per task when the task is in 'done' but has no provable on-main
evidence. Dedup was via an in-memory finalizeUnprovenWarned Set per manager
instance, so every engine restart resurfaced the same warning on the next
sweep — significant noise on done tasks that legitimately lack evidence,
typically residue of FN-4811 contamination (FN-4771/FN-4778 in production).

Adds an optional MergeDetails.integrityWarning = { warnedAt, reason } field
and persists it on the first warning. Both warning sites in
reconcileDoneTaskIntegrity() (the unproven-and-still-mergeable branch and
the unproven-final branch) now consult the persisted record:

  - Same reason as persisted → skip re-emitting, just rehydrate the in-memory
    Set for in-process consistency.
  - Different reason → re-warn (so a *new* classification problem still
    surfaces) and update the persisted record.

Tests added under
packages/engine/src/__tests__/reliability-interactions/integrity-warning-persisted-dedup.test.ts
(real-git, 4 cases):

  - First sweep: emits warning + persists record.
  - Second sweep, same instance: in-memory Set dedupes (existing contract).
  - Fresh manager (simulated engine restart) + pre-persisted record:
    persisted dedup suppresses re-emission.
  - Fresh manager + persisted record with different reason: must re-warn
    and overwrite the persisted reason.

Full engine suite: 308 files, 5041 tests pass, 1 skipped. Lint clean.

Fusion-Task-Id: FN-4811
2026-05-16 16:51:43 -07:00
Fusion
dcd6bf60f3 fix(FN-4811): recover from validation-failed remove + collapse broken FN-4806 nested branches
Two follow-ups stacked on the FN-4811 active-worktree liveness gate:

1. Stale conflict-path recovery (FN-4813 production failure)

   When 'git worktree remove --force' fails with 'fatal: validation failed,
   cannot remove working tree', the worktree directory is missing on disk
   and the git admin entry is stale. Without this recovery, every retry of
   tryCreateWorktree on a stale conflict path failed 3 times with
   'automatic cleanup failed', leaving tasks unable to create worktrees.

   cleanupConflictingWorktree now catches that specific error class, runs
   'git worktree prune' to drop the stale admin entry, best-effort deletes
   the branch, and returns success so the caller can proceed.

   Implementation note: the original attempt used existsSync(worktreePath)
   as a pre-check, but vitest's vi.clearAllMocks() can leave the existsSync
   mock returning undefined, causing the new branch to fire inside tests
   that didn't expect it and leading to worker OOM in
   executor-worktree.test.ts. The error-class-based catch is robust against
   mock state and matches the real production failure signal exactly.

2. Collapsed broken FN-4806 nested branches

   The previous FN-4806 refactor (commit 087b1a766) accidentally nested the
   genuine 'agent finished without calling fn_task_done after N retries'
   failure path INSIDE the silent-recovery branch, meaning ordinary
   failures were being silently requeued (no status=failed, no onError, no
   retry-budget burn) instead of being surfaced.

   Restored the clean two-branch structure:

     } else if (retryAbortedDueToReclaim) {
       // silent recovery (FN-4806)
     } else {
       // genuine no-fn_task_done exhaustion: mark failed, onError, burn budget
     }

   Also clears baseCommitSha on silent recovery (matches the parallel
   session-start-failure path's metadata clearing).

Tests:

  - Adds 'FN-4811 follow-up (FN-4813): recovers from validation failed'
    case to active-worktree-removal-liveness.test.ts (12 total cases).
  - executor-recovery.test.ts no-fn_task_done reclaim coverage now
    asserts baseCommitSha is cleared.
  - executor-recovery.test.ts 'does not mark task as failed when invalid
    transition error occurs on completion' regression fixed by restoring
    the failure-path branch.
  - executor-core.test.ts 'still enforces fn_task_done requirement in
    fast mode' restored.

Full engine suite: 307 files, 5037 tests pass, 1 skipped. Lint clean.

Fusion-Task-Id: FN-4811
2026-05-16 16:25:28 -07:00
Fusion
4c26aa6e91 fix(FN-4811): refuse to force-remove worktrees actively bound to live sessions
The executor's conflict-recovery paths (cleanupConflictingWorktree,
handleBranchConflict, and tryCreateWorktree's live-foreign/stale-resolved
branches) could force-remove a worktree even when it was currently bound
to an active executor session. This caused the FN-4781/FN-4804 cascade:

  - 'Execution blocked: assigned worktree path disappeared mid-task' as
    git deleted the live agent's filesystem out from under it
  - Two parallel runs for the same task alive simultaneously, with the
    second run started in a fresh worktree while the first was still
    holding the old session
  - Cross-task log attribution (an FN-4804 runContext writing to FN-4781)
  - Post-merge 'branch tip misbound but content found on main via trailer'
    rescues firing on every successful merge as the bookkeeping was
    corrupted mid-merge

Adds a hard liveness gate centralized in findActiveWorktreeOwner(), which
checks both the in-memory activeWorktrees map and the DB for non-done,
non-paused, in-progress tasks bound to the worktree. The gate fires at
two points:

  1. cleanupConflictingWorktree returns false (refuses removal) when an
     active owner is found, logging an FN-4811 refusal entry.
  2. handleBranchConflict short-circuits to 'sticky' BEFORE invoking
     inspectBranchConflict, because some inspection branches force-remove
     unconditionally.

When cleanup is refused, the live-foreign and stale-resolved branches in
tryCreateWorktree now FALL THROUGH to the suffix-rename path (rather
than returning null) so the requesting task can still proceed without
disturbing the live owner.

Tests:

  - New reliability-interactions backstop at
    src/__tests__/reliability-interactions/active-worktree-removal-liveness.test.ts
    covers findActiveWorktreeOwner (5 cases: in-memory match, requesting
    task excluded, DB-level match, paused exclusion, terminal-column
    exclusion, self-exclusion), cleanupConflictingWorktree gate (3 cases:
    in-memory refuse, DB refuse, no-owner proceed), and handleBranchConflict
    gate (2 cases: short-circuit + inspection-skipped, no-owner proceeds).
  - Updates existing executor-worktree.test.ts assertion that was
    documenting the bug behavior (force-removing active worktree) to match
    the new contract (refuses + falls through to suffix-rename).

Full engine suite: 307 files, 5035 tests pass.

Fusion-Task-Id: FN-4811
2026-05-16 15:36:07 -07:00
Fusion
3f8a5e6839 fix(FN-4806): silently recover when worktree/branch reclaimed mid-retry
The executor's no-fn_task_done retry loop has two reclaim signals
(retryAbortedDueToReclaim=true): (1) pre-retry liveness recheck where the
task DB shows worktree/branch was cleared, and (2) session-start failure
because the worktree path no longer exists. Both are engine self-heal
situations triggered by FN-4546 stale-active-branch reclaim, FN-4742
self-healing removals, or related housekeeping paths — the agent never
got a fair retry attempt.

Previously these surfaced as task status=failed with error
'Worktree/branch reclaimed during no-fn_task_done retry — requeueing',
fired onError, and burned the taskDoneRetryCount budget. Three legitimate
problems followed: tasks accumulated spurious failures in the UI, the
exhausted-budget branch escalated reclaimed tasks to in-review instead of
retrying, and the noise masked the underlying worktree-removal regression
(FN-4811).

Now the reclaim branch silently:
- clears stale worktree/branch metadata so the next pickup creates a fresh worktree
- requeues to todo with preserveProgress
- logs an informational 'engine self-heal, no failure' line
- does NOT set status=failed, does NOT bump taskDoneRetryCount, does NOT call onError

The genuine 'agent finished without calling fn_task_done after N retries'
exhaustion path (retryAbortedDueToReclaim=false) is unchanged.

Tests updated in
packages/engine/src/__tests__/reliability-interactions/executor-no-task-done-vs-worktree-reclaim.test.ts
to assert the new silent-recovery contract on all three reclaim paths.

Fusion-Task-Id: FN-4806
2026-05-16 15:16:05 -07:00
Fusion (runfusion.ai)
415cd6a111 test(FN-4783): stabilize workspace test lanes for verification
Fusion-Task-Id: FN-4783
Fusion-Task-Lineage: 7a02897b-e303-4947-86b2-3dd7a343f653
2026-05-16 13:46:43 -07:00
Fusion (runfusion.ai)
05bd4fcdf9 test(FN-4754): add engine dashboard diff import boundary guard
Fusion-Task-Id: FN-4754
Fusion-Task-Lineage: 69980e60-c1ad-4e19-bee0-970312669d70
2026-05-16 10:50:06 -07:00
Fusion (runfusion.ai)
33755d44fb fix(FN-4742): stabilize worktrunk removal interaction assertions
Fusion-Task-Id: FN-4742
Fusion-Task-Lineage: 59bca56e-c9d9-4066-9f5a-6d8ee252a100
2026-05-16 10:08:02 -07:00
Fusion (runfusion.ai)
378762c4ab fix(FN-4742): address review for worktrunk removal backstop
Fusion-Task-Id: FN-4742
Fusion-Task-Lineage: 59bca56e-c9d9-4066-9f5a-6d8ee252a100
2026-05-16 10:08:02 -07:00
Fusion (runfusion.ai)
251a472458 test(FN-4742): complete Step 5 — add worktrunk removal interaction backstop
Fusion-Task-Id: FN-4742
Fusion-Task-Lineage: 59bca56e-c9d9-4066-9f5a-6d8ee252a100
2026-05-16 10:08:02 -07:00
Fusion (runfusion.ai)
07baf9526e test(FN-4628): address review feedback for interaction backstop
Fusion-Task-Id: FN-4628
Fusion-Task-Lineage: ebd36565-946b-4b4f-b8df-4d99d7a5f074
2026-05-15 23:50:12 -07:00
Fusion (runfusion.ai)
ad99f54a9d test(FN-4628): complete Step 4 — add worktrunk self-healing interaction backstop
Fusion-Task-Id: FN-4628
Fusion-Task-Lineage: ebd36565-946b-4b4f-b8df-4d99d7a5f074
2026-05-15 23:50:12 -07:00
Fusion (runfusion.ai)
178bad0d08 test(FN-4626): complete Step 6 — add worktrunk audit coverage
Fusion-Task-Id: FN-4626
Fusion-Task-Lineage: 04b06bf6-b08c-4ff1-b69a-ab93f6e508b3
2026-05-15 22:18:09 -07:00
Fusion (runfusion.ai)
0f9c07454e fix(FN-4662): address test typing and scenario coverage
Fusion-Task-Id: FN-4662
Fusion-Task-Lineage: f3de365a-ccb5-404b-9d0c-f9bc70d2678a
2026-05-15 21:57:57 -07:00
Fusion (runfusion.ai)
cdd85ba235 test(FN-4662): complete Step 3 — add starvation sweep coverage
Fusion-Task-Id: FN-4662
Fusion-Task-Lineage: f3de365a-ccb5-404b-9d0c-f9bc70d2678a
2026-05-15 21:57:57 -07:00
Fusion (runfusion.ai)
3339d94d42 feat(FN-4625): complete Step 4 — add self-healing worktrunk pause guardrail
Fusion-Task-Id: FN-4625
Fusion-Task-Lineage: c1d9b414-b5dd-4786-ba51-0b8adec4acf5
2026-05-15 20:53:54 -07:00
Fusion (runfusion.ai)
296203443b feat(FN-4701): add no-changes-finalized reliability interaction test
Adds reliability interaction test coverage for the `no-changes-finalized` outcome (FN-4701), covering the benign case where verification-only done tasks complete with no owned commits or surviving branch.

Fusion-Task-Id: FN-4701
2026-05-15 20:38:04 -07:00
Fusion (runfusion.ai)
b144ff7636 feat(FN-4700): complete Step 3 — reconcile benign verification finalizes
Fusion-Task-Id: FN-4700
Fusion-Task-Lineage: 87097576-c2be-4be4-8e10-9af85831fe5d
2026-05-15 20:18:55 -07:00
Fusion (runfusion.ai)
ad73d76586 feat(FN-4640): complete Step 6 — dashboard sandbox audit surface
Fusion-Task-Id: FN-4640
Fusion-Task-Lineage: 4a91265f-1714-4854-b08d-7ddb06074253
2026-05-15 16:19:42 -07:00
Fusion
bd3809ed81 docs(FN-4656): complete Step 6 — docs, changeset, and integrity sweep notes
Fusion-Task-Id: FN-4656
Fusion-Task-Lineage: 81e8bd91-bc02-4bb7-a590-0e6e84e3535f
2026-05-15 14:06:28 -07:00
Fusion
4ea1fa3e23 test(FN-4656): complete Step 4 — add reliability interaction coverage
Fusion-Task-Id: FN-4656
Fusion-Task-Lineage: 81e8bd91-bc02-4bb7-a590-0e6e84e3535f
2026-05-15 14:06:28 -07:00
Fusion
ebd0be13d9 feat(FN-4651): complete Step 3 — recover no-progress worktree failures
Fusion-Task-Id: FN-4651
Fusion-Task-Lineage: 14d15555-468d-4690-8efd-a09b31b5d407
2026-05-15 12:51:41 -07:00
Fusion
8866934b7d test(FN-4653): complete Step 6 — finalize governance and typing-safe interactions
Fusion-Task-Id: FN-4653
Fusion-Task-Lineage: b1bc9bd0-0270-412b-bb63-fb32ea5f44d0
2026-05-15 12:40:10 -07:00
Fusion
2aeb2cf180 test(FN-4653): complete Step 5 — add governance pause-gating interactions
Fusion-Task-Id: FN-4653
Fusion-Task-Lineage: b1bc9bd0-0270-412b-bb63-fb32ea5f44d0
2026-05-15 12:40:10 -07:00
Fusion
fcbddfe9e2 test(FN-4653): complete Step 4 — cover restart sweep landed-content behavior
Fusion-Task-Id: FN-4653
Fusion-Task-Lineage: b1bc9bd0-0270-412b-bb63-fb32ea5f44d0
2026-05-15 12:40:10 -07:00
Fusion
5548b6f4a6 test(FN-4653): complete Step 3 — add scheduler scope and stale-blockedBy interaction
Fusion-Task-Id: FN-4653
Fusion-Task-Lineage: b1bc9bd0-0270-412b-bb63-fb32ea5f44d0
2026-05-15 12:40:10 -07:00
Fusion
875b322664 test(FN-4653): complete Step 2 — cover mixed blocker sweep handoff
Fusion-Task-Id: FN-4653
Fusion-Task-Lineage: b1bc9bd0-0270-412b-bb63-fb32ea5f44d0
2026-05-15 12:40:10 -07:00
Fusion
208753fb88 test(FN-4653): complete Step 1 — add merge-path soft-blocker auto-finalize interaction
Fusion-Task-Id: FN-4653
Fusion-Task-Lineage: b1bc9bd0-0270-412b-bb63-fb32ea5f44d0
2026-05-15 12:40:10 -07:00
Fusion
d53110cd04 test(FN-4648): complete Step 4 — regression coverage updates
Fusion-Task-Id: FN-4648
Fusion-Task-Lineage: 82289e69-7e72-40e4-acaf-b98ad82ad354
2026-05-15 11:41:35 -07:00
Fusion
5bef3daf47 test(FN-4601): add retry race reliability interaction coverage
Fusion-Task-Id: FN-4601
Fusion-Task-Lineage: 136888a7-aee7-4eda-a165-16448a98e626
2026-05-15 07:27:23 -07:00
Fusion
38267f479f feat(FN-4559): complete remaining recovery, tests, and docs
Fusion-Task-Id: FN-4559
Fusion-Task-Lineage: 9276f683-657c-4604-aaef-143da488b287
2026-05-15 00:50:11 -07:00
Fusion
d88e1bd143 feat(FN-4565): align self-healing assertion with FN-4563 unusable-worktree
Test-only update aligning a self-healing assertion's wording with the FN-4563 unusable-worktree terminology, touching only the reliability-interaction test file.

Fusion-Task-Id: FN-4565
2026-05-14 23:41:25 -07:00
Fusion
c11389a75e test(FN-4564): complete Step 3 — add self-healing rebound coverage
Fusion-Task-Id: FN-4564
Fusion-Task-Lineage: 1a0c186a-1eb1-496a-aa8d-1bdf2254bb20
2026-05-14 23:18:47 -07:00
Fusion
05ecf1fb41 feat(FN-4545): complete Step 10 — documentation and delivery artifacts
Fusion-Task-Id: FN-4545
Fusion-Task-Lineage: 95108429-618e-4a6d-9fa9-7ac2596665a2
2026-05-14 22:16:44 -07:00
Fusion
650e6443fe test(FN-4545): add reliability interaction coverage
Fusion-Task-Id: FN-4545
Fusion-Task-Lineage: 95108429-618e-4a6d-9fa9-7ac2596665a2
2026-05-14 22:16:44 -07:00
Fusion
c8d9492b61 feat(FN-4550): remove duplicate Case 6 skip coverage
Removes a duplicate "Case 6" skip from the merge-strategy overlap test file, cleaning up the test coverage without changing behavior.

Fusion-Task-Id: FN-4550

Fusion-Task-Lineage: 3b254a66-bd47-4699-a788-3a57178a2884
2026-05-14 20:21:09 -07:00
Fusion
5e1a5a55bb feat(FN-4536): complete auto-recovery branch/worktree handler
Fusion-Task-Id: FN-4536
Fusion-Task-Lineage: 8d5615e4-e75d-413d-b3ab-0c4874b21fc5
2026-05-14 19:06:25 -07:00
Fusion
79eb74ca7e feat(FN-4534): add auto-recovery dispatcher and pause-site wiring
Fusion-Task-Id: FN-4534
Fusion-Task-Lineage: 2b46a5df-efc6-4305-a0f2-69991e58512c
2026-05-14 17:59:57 -07:00
Fusion
72a71e9db6 test(FN-4508): complete Step 5 — add reliability interaction backstops
Fusion-Task-Id: FN-4508
Fusion-Task-Lineage: 948a9cae-3975-4f15-bd47-2f88b379171d
2026-05-14 16:33:27 -07:00
Fusion
cc18c4f780 test(FN-4523): complete Step 4 — add completion fan-out coverage
Fusion-Task-Id: FN-4523
Fusion-Task-Lineage: c7a02071-f911-4c42-a174-cdcf2453fd23
2026-05-14 14:09:57 -07:00
Fusion
62743e2bac test(FN-4500): complete Step 4 — add reliability interaction coverage
Fusion-Task-Id: FN-4500
Fusion-Task-Lineage: 81d33759-70f6-4958-abf8-3db20f918c01
2026-05-14 12:44:11 -07:00
Fusion
2d9badbc1f test(FN-4500): cover fast-path guards and align legacy reclaim assertions
Fusion-Task-Id: FN-4500
Fusion-Task-Lineage: 81d33759-70f6-4958-abf8-3db20f918c01
2026-05-14 12:44:10 -07:00
Fusion
b3590d9220 test(FN-4485): add reliability interaction race coverage
Fusion-Task-Id: FN-4485
Fusion-Task-Lineage: 034088dc-ebc4-4e12-8314-39419d41b23f
2026-05-14 11:23:45 -07:00
Fusion
b72bd0575d feat(FN-4485): complete Step 4-5 orphan rescue and audit paths
Fusion-Task-Id: FN-4485
Fusion-Task-Lineage: 034088dc-ebc4-4e12-8314-39419d41b23f
2026-05-14 11:23:44 -07:00
Fusion
c7f8d38210 test(FN-4361): add reliability interaction backstop suite
Fusion-Task-Id: FN-4361
Fusion-Task-Lineage: bdf2de3b-c9ce-4a29-9e60-9a029f913ae9
2026-05-13 22:08:41 -07:00