Production failure shape:
Auto-recovery failed: branch conflict unrecoverable \u2014
Branch fusion/fn-4847 is already checked out at /.../deft-crane
(tip a881ccc86660, 24 stranded commits since 0b28388876).
Run branch recovery and explicitly choose whether to reclaim or
discard prior work.
The 24 stranded commits are cross-task contamination residue from the
FN-4781/FN-4804/FN-4814 worktree-race era \u2014 they are NOT FN-4847's work.
Previously this paused the task with pausedReason='branch-conflict-
unrecoverable' and the task got stuck forever waiting for human
adjudication.
User intent (FN-4847): 'just create a new branch and keep going and
discard the old one'. Implementation:
1. auto-recovery.ts:actionForMode \u2014 in 'deterministic-only' mode (the
default), branch-conflict-unrecoverable now returns 'retry' (was
'pause'). This routes the failure to the handler instead of pausing.
2. auto-recovery-handlers/branch-worktree.ts \u2014 'live-foreign' inspection
no longer emits irreducible-pause. Instead:
- Check FN-4811 active-session registry. If the foreign worktree is
bound to a live executor/merger session, do NOT force-remove it
(would yank the live agent's filesystem). Just requeue and let
downstream conflict-recovery handle it.
- Otherwise: force-delete the foreign worktree (--force) + prune git
worktree admin entries + force-delete the branch. Errors at each
step are best-effort and logged.
- Emit new audit event 'branch-worktree:foreign-branch-discarded'
with stranded-commit count, live-ownership flag, success flags.
- Requeue task to 'todo' with preserveProgress, clearing
branch+baseCommitSha.
3. run-audit.ts \u2014 register new DatabaseMutationType.
4. executor-worktree.test.ts \u2014 update the 'records recovery context'
test to assert the new retry+requeue contract (was asserting the old
pause-with-status-failed contract).
Verification:
- Targeted suite (4 files, 343 tests): pass.
- pnpm --filter @fusion/engine build: clean.
- pnpm lint: clean.
Fusion-Task-Id: FN-4847
Adds a new secrets store module to `@fusion/core` (277 lines in `secrets-store.ts`) and exports it from the package index.
Fusion-Task-Id: FN-4791
Fusion-Task-Lineage: 7ff20b8a-37e1-46c6-8003-b542df9f98b1
Refactors the settings store tests in `packages/core` by consolidating round-trip and default value coverage into structured `it.each` matrices, trimming the test file by roughly 60 lines while maintaining equivalent coverage.
Fusion-Task-Id: FN-4836
Hardens test timeouts across 12 reliability and integration test files (branch-conflict recovery, merger diff/overlap guards, self-healing, worktree hydration, workflow/file-scope interactions), covering both real-git and mock-based test lanes with consistent timeout adjustments to reduce flakiness.
Fusion-Task-Id: FN-4839
The [scope-leak] reviewLevel=N enforcement=warn warning was firing on
many in-progress tasks for off-scope .changeset/FN-XXXX-*.md files (the
production signature on FN-4789, FN-4801, FN-4818 \u2014 their branches all
contained .changeset/FN-4811-*.md files from the in-progress fix stack).
By convention every task may add its own changeset entry under
.changeset/ per AGENTS.md 'Finalizing Changes' section, so .changeset/
files are now treated as always-allowed by the scope-leak guard
regardless of the task's declared file scope.
Cross-task changeset leakage is still caught by stronger downstream
guards (file-scope invariant at squash, post-merge audit) at much
higher signal-to-noise. This change only suppresses the noisy
per-execution warning that was flooding logs without adding any
defensive value.
Adds a new exported helper isAlwaysAllowedScopeLeakPath() so the
allowlist surface is easy to extend. Test coverage in
scope-leak-changeset-allowlist.test.ts.
Also (incorporated from interrupted merge state): loosens the
executing-task-lock.test.ts assertion that one losing-instance store
sees zero work-log entries rather than the brittle exact-count of
mockedCreateFnAgent invocations (the no-fn_task_done retry path can
fire on the winning instance, so the count varies).
Fusion-Task-Id: FN-4811
Investigating FN-4814 + FN-4811 re-failures after commit 8bef30655 (which
added per-instance synchronous this.executing.add) revealed the per-instance
guard was insufficient. FN-4809 log at 02:48:17-18 UTC:
02:48:17 [-] Resuming execution after unpause
02:48:17 [-] Step 4 (Testing & Verification) -> pending
02:48:17 [-] Step 4 (Testing & Verification) -> pending
02:48:17 [6097725-y2nb] Executor detected stale merge state ...
02:48:18 [6097816-9gde] Executor detected stale merge state ...
Both runs y2nb and 9gde reached executor.ts:2661 (which is INSIDE execute(),
past the synchronous this.executing.add claim). The only viable explanation
is that there is more than one TaskExecutor instance in the same Node
process (engine restart race, multi-project hybrid runtime, or similar code
path). Each instance has its own executing Set, so the per-instance guard
doesn't help.
Fix: module-level singleton executingTaskLock in active-session-registry.ts,
shared across all TaskExecutor instances. execute() synchronously tryClaim()s
the lock; if false, bails. Every existing this.executing.delete() site also
calls executingTaskLock.release(). Per-instance this.executing kept because
many other call sites use it (this.executing.has at handler gates,
stuck-detector, resumeTaskForAgent, etc.).
Test setup (resetExecutorMocks in executor-test-helpers.ts) clears the lock
between tests so process-wide state doesn't leak (executor-pause and
executor-prompt tests would otherwise show 'expected 2 createFnAgent calls
but got 0' / 'expected not called but called 3 times' flakes).
Tests:
- executing-task-lock.test.ts: 2 cases. Key case creates TWO TaskExecutor
instances and races them on the same task ID, asserts only ONE actually
runs. Verified FAILS on prior code (8bef30655) and PASSES on fix.
Verification:
- Targeted suite (4 files, 170 tests): pass.
- pnpm --filter @fusion/engine build: clean.
- pnpm lint: clean.
Fusion-Task-Id: FN-4811
TaskExecutor.execute() had a classic JS async race window. Original:
async execute(task) {
if (this.executing.has(task.id)) return; // check
const assignedAgentId = task.assignedAgentId;
if (assignedAgentId && await this.shouldDeferForHeartbeat(...)) // AWAIT yields
return;
this.executing.add(task.id); // add (too late)
...
}
Two concurrent execute(task) calls (scheduler dispatch + task:moved event
handler + restart-recovery) both:
1. Pass the synchronous has() check (Set is empty).
2. Enter the awaited shouldDeferForHeartbeat call (yields the event loop).
3. Resume and both call this.executing.add(task.id).
4. Both proceed to create the same worktree path.
Production failure shape (FN-4814 + FN-4811, observed within minutes):
01:30:56 [runA-caoe] Worktree created at /...worktrees/bright-mesa
01:30:56 [runB-w23q] Worktree created at /...worktrees/bright-mesa
01:30:58 worktree liveness assertion failed: not_usable_task_worktree
01:31:48 [thirdRun] also fires liveness assertion fail
01:37:48 In-review stall surfaced [no-worktree-no-merge-confirmed]
This is the root cause of the entire FN-4781/FN-4804/FN-4814/FN-4811
cascade. Every other guard added today (FN-4811 active-session gate,
self-healing reclaim defer, validation-failed recovery, silent reclaim
recovery, integrity-warning dedup) was patching SYMPTOMS of the
duplicate-run race. With this fix, the symptoms stop appearing.
Fix: claim the slot synchronously immediately after the has() check,
release it on the heartbeat-defer early-return path. No await happens
between check and claim, so the race window is closed.
Test added under
packages/engine/src/__tests__/reliability-interactions/concurrent-execute-race.test.ts
verified to fail on the prior (a1b1f9aa0) executor.ts and pass on the
fixed version:
- Two concurrent execute() calls produce the SAME number of
createFnAgent invocations as one execute() call (no amplification).
- A second sequential execute() after the first completes IS allowed
(slot was released).
The task must have assignedAgentId set to exercise the race \u2014 without
it, the short-circuit `assignedAgentId && ...` evaluates the left side
to false synchronously, and no await happens.
Full engine suite: 5048+ tests pass. The 7 transient test-file failures
in the broad parallel run are pre-existing flaky real-git tests
(branch-conflicts-zero-unique, branch-conflicts-recovery,
merger-overlap-guard subprocess-guard contention) \u2014 all of them pass
when run alone or as a smaller group, none touch the executor.execute()
path.
Fusion-Task-Id: FN-4811
The reclaimSelfOwnedBranchConflicts sweep was force-pausing actively-running
tasks. Production failure shape on FN-4819:
1. Self-healing sweep runs every cycle and inspects branch conflicts.
2. For FN-4819, inspection classified the conflict as 'tip-already-merged'
(the task's branch tip was already on main).
3. Sweep called removeWorktree({ reason: SelfHealingBranchConflict }).
4. The FN-4811 active-session gate correctly refused: the worktree was
bound to FN-4819/executor (a live agent session was using it).
5. The thrown ActiveSessionWorktreeRemovalError was caught by the outer
reclaim catch block.
6. The catch escalated to AutoRecoveryDispatcher with class
'branch-conflict-unrecoverable'.
7. decision.action === 'pause' marked the task failed + paused +
pausedReason='branch-conflict-unrecoverable' + moved to in-review.
Net effect: the FN-4811 gate (which is correct \u2014 you can't yank a live
worktree) became a regression source because the self-healing sweep
interpreted the refusal as fatal. Tasks that were actively making progress
got paused with a misleading 'branch conflict unrecoverable' error.
Fix: at the top of the per-task reclaim loop in
reclaimSelfOwnedBranchConflicts, check
activeSessionRegistry.isPathActive(task.worktree) and continue for any
task whose worktree is currently bound to a live session. The reclaim
will retry on the next sweep (sweeps run every cycle) once the session
has finished using the worktree. No data is lost, no decision is forced.
Test added under
packages/engine/src/__tests__/reliability-interactions/reclaim-defers-on-active-session.test.ts
covering:
- The skip path: when activeSessionRegistry has a registration for
task.worktree, the sweep MUST NOT call inspectBranchConflict,
removeWorktree, or isUsableTaskWorktree. The task MUST stay in
in-progress, not be marked failed/paused, not be moved to in-review.
- Control: with no registration, the sweep DOES proceed and reaches
inspectBranchConflict (preserving existing behavior).
Full engine suite: 314 files, 5061 tests pass, 1 skipped. Lint clean.
Build clean.
Fusion-Task-Id: FN-4811
