Files
fusion/docs/self-healing-backward-move-audit.md
gsxdsm 936dd1ea96 FN-8924: retain safe age gate for merge recovery
Document why interrupted merge recovery retains its age-based admission gate.

- Record why merger logs cannot establish task ownership or orphan status.
- Pin non-owner recovery behavior when a merging task was recently updated.

Files changed:
 docs/self-healing-backward-move-audit.md           | 8 +++++++-
 packages/engine/src/__tests__/self-healing.test.ts | 9 ++++++++-
 2 files changed, 15 insertions(+), 2 deletions(-)

Fusion-Task-Id: FN-8924

Fusion-Task-Lineage: 19e7f803-21fe-43fb-8ca6-eecf8ff61df4

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
2026-08-10 09:09:04 -07:00

17 KiB
Raw Blame History

Self-Healing Backward-Move Audit (FN-5335)

Invariant

Backward lifecycle moves (in-progress → todo, in-review → in-progress/todo, in-review → paused-stuck/failed) from a maintenance sweep require triple proof:

  1. Verified dead session — no active session registry entry for any pathsForTask(taskId), no executingTaskLock.has(taskId), and this.options.isTaskActive?.(taskId) !== true.
  2. Unusable worktree — classifyTaskWorktree(this.options.rootDir, task.worktree) returns { ok: false }, OR task.worktree is null and there is no registered fusion/<id> worktree.
  3. No recent executor activity — stage staleness anchor exceeds stage grace window and there is no recent worktree:incomplete-detected event within the same trailing window.

Stages that cannot satisfy all three must either (a) tighten predicate to require triple proof, or (b) downgrade to annotation-only (task:<stage>-no-action).

Catalog

Method Source line Trigger condition Grace source Evidence required today Action today Direction Disposition New evidence required New action
recoverCompletedTasks 1455 in-progress + all steps terminal n/a step completion move to in-review FORWARD keep n/a n/a
recoverStrandedCompletedTodoTasks 1507 todo + all steps terminal n/a step completion move to in-progress (resume) FORWARD keep n/a n/a
recoverStaleMergingStatus 3521 in-review stale merging status staleMergingStatusMinAgeMs stale status + no active merge clear status; re-enqueue eligible unpaused non-workspace task RECONCILE + FORWARD RE-ENQUEUE keep paused cards never enqueue; only merge-deadlock-detected may clear its orphaned stamp n/a
reclaimPrConflicts 1606 PR mergeable=conflicting path-dependent conflict inspect result delegates to reclaimPrConflictForTask RECONCILE/BACKWARD mix keep n/a n/a
reclaimPrConflictForTask 1622 reclaimable conflicting PR branch inspectConflict + usable worktree checks active-session/usable-worktree checks only may move in-review→todo BACKWARD tighten triple proof on in-review→todo path gate move; emit task:reclaim-pr-conflict-no-action
reclaimSelfOwnedBranchConflicts 1772 branch conflict self-owned conflict inspector local checks conflict classifier only may move in-review→todo BACKWARD tighten triple proof before backward move gate move; emit task:reclaim-self-owned-branch-conflict-no-action
reclaimStaleActiveBranches 2174 stale active branch metadata STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS age + no active session + no local changes metadata clear/prune RECONCILE-ONLY keep n/a n/a
reconcileCompletedTask 2438 done fanout cleanup n/a dependency unblock checks blocker/status metadata updates RECONCILE-ONLY keep n/a n/a
reconcileInReviewBranchRebind 2646 in-review missing branch metadata n/a unique candidate branch proof branch metadata rebind/clear RECONCILE-ONLY keep n/a n/a
reconcileTaskWorktreeMetadata 2815 stale task.worktree metadata mixed (includes stale merge thresholds) registered-worktree reconciliation metadata fixes (+ targeted no-op move hygiene) RECONCILE-ONLY keep n/a n/a
autoReboundPausedScopeDecayDetailed 2922 paused in-progress scope holder stale beyond decay threshold settings.pausedScopeDecayMs age + scoped follower count move to todo (preserve progress/worktree/resume state) BACKWARD tighten triple proof + existing decay predicates gate move; emit task:auto-rebound-scope-decay-no-action
surfaceDbCorruption 3259 db corruption marker notification cooldown db marker presence notify/log ANNOTATION-ONLY keep n/a n/a
reconcileSelfDefeatingDependencies 3499 dependency self-loop n/a graph proof dependency metadata repair RECONCILE-ONLY keep n/a n/a
reconcileDependencyCycles 3557 persisted dependency cycle n/a cycle detector selective dependency edge repair RECONCILE-ONLY keep n/a n/a
reconcileStaleMergerStatus 3702 stale merger status staleMergingStatusMinAgeMs age/no active merge clear status RECONCILE-ONLY keep n/a n/a
finalizeNoOpReviewTasks 3750 in-review already-on-main no-op git landed proof landed/no-op proof mostly in-review→done; one in-review→todo remediation path FORWARD + BACKWARD edge tighten (backward edge only) triple proof for fallback in-review→todo path gate fallback; emit task:finalize-no-op-review-no-action
reconcileDoneTaskIntegrity 3870 done task integrity mismatch n/a integrity check metadata/status hygiene RECONCILE-ONLY keep n/a n/a
recoverMergeableReviewTasks 3996 mergeable in-review task merge blocker checks merge readiness queue/merge status writes INTERNAL-RETRY keep n/a n/a
recoverReviewTasksWithFailedPreMergeSteps 4112 failed pre-merge workflow step maxFix budget failed step + budget revive via executor INTERNAL-RETRY keep n/a n/a
recoverStaleIncompleteReviewTasks 4205 in-review with incomplete steps stale settings.taskStuckTimeoutMs age + incomplete steps move to todo preserveProgress BACKWARD tighten triple proof + existing stale/incomplete predicates gate move; emit task:stale-incomplete-review-no-action
surfaceInReviewStalls 4281 in-review stall reasons settings.taskStuckTimeoutMs, inReviewStallDeadlockThreshold stall signal + repetition log; deadlock branch pauses/failed BACKWARD branch already-tightened (threshold + FN-5147) keep existing no change
surfaceDependencyBlockedTodos 4390 blocked todo report reporter internals blocked dependency snapshot report/log ANNOTATION-ONLY keep n/a n/a
surfaceInReviewStalled 4416 quiet in-review backlog signal settings.inReviewStalledThresholdMs quiet-window signal log only ANNOTATION-ONLY keep n/a n/a
surfaceStalePausedReviews 4478 stale paused in-review settings.stalePausedReviewThresholdMs age signal log only ANNOTATION-ONLY keep n/a n/a
surfaceStalePausedTodos 4529 stale paused todo settings.stalePausedTodoThresholdMs age signal log only ANNOTATION-ONLY keep n/a n/a
recoverGhostReviewTasks 4583 idle in-review ghost settings.taskStuckTimeoutMs idle + status filters move to todo preserveProgress BACKWARD tighten triple proof + existing ghost predicates gate move; emit task:ghost-review-no-action
recoverInterruptedMergingTasks 4650 stale transient merging status updatedAt age ≥ settings.taskStuckTimeoutMs stale status + landed-commit detection done finalize or status clear/requeue FORWARD/INTERNAL-RETRY keep FN-8924 retained the age gate: merger-agent logs are not an orphan-proof task-scoped clock n/a
recoverDoneTaskMergeMetadata 4777 done merge metadata drift n/a landed evidence + metadata gap metadata update RECONCILE-ONLY keep n/a n/a
recoverMergedReviewTasks 4937 in-review already merged landed commit proof landed proof move to done FORWARD keep n/a n/a
recoverStuckMergeDeadlocks (landed) 5036 retry-exhausted merge failed but landed cooldown + retries landed proof move to done + unblock deps FORWARD keep n/a n/a
recoverStuckMergeDeadlocks (no-landed) 5036 retry-exhausted merge failed and not landed DEADLOCK_RECOVERY_COOLDOWN_MS no landed proof only updateTask({ paused: true }) + log BACKWARD tighten triple proof before pause/fail disposition gate pause; emit task:stuck-merge-deadlock-no-action
recoverOrphanOnlyScopeViolations 5192 failed scope-only orphan but landed landed proof parsed scope payload + landed proof move to done or failed-blocked FORWARD/RECONCILE keep n/a n/a
recoverAlreadyMergedReviewTasks 5350 retry-exhausted failed review but landed retries threshold landed proof move to done FORWARD keep n/a n/a
recoverCompletionHandoffLimbo 5473 in-review no handoff state after task_done COMPLETION_HANDOFF_LIMBO_GRACE_MS done-marker age + no active task + merge blocker absent + retry cap requeue merge or fail-exhausted already proof-gated already-tightened (FN-4999) keep no change
recoverBranchMisboundInReviewTasks 5581 in-review failed merge lineage misbound but landed retries threshold classifyOwnedLandedEvidence proof move to done FORWARD keep n/a n/a
recoverForeignOnlyContaminatedInReviewTasks 5692 foreign-only contamination retries/config gates contamination attribution proof reanchor/requeue or annotate INTERNAL-RETRY keep n/a n/a
recoverMisclassifiedFailures 5789 in-review failed but no merge blocker and all steps done n/a completed steps + no blocker clear status/error RECONCILE-ONLY keep n/a n/a
recoverInProgressLimbo 5877 in-progress no branch + missing/none worktree + pending steps ORPHANED_EXECUTION_RECOVERY_GRACE_MS dead/no-progress proof set move to todo preserveProgress BACKWARD already-tightened (FN-5219) keep no change
recoverOrphanedExecutions 5991 stale in-progress no active execution ORPHANED_EXECUTION_RECOVERY_GRACE_MS / ORPHANED_WITH_WORKTREE_GRACE_MS stale orphan candidate only annotation event only ANNOTATION-ONLY already-tightened (FN-5337) keep no change
recoverAgentsRunningOnInactiveTasks 6089 agent runtime drift agent heartbeat ages agent-store runtime checks end/restart agents RECONCILE-ONLY keep n/a n/a
recoverDriftedAgentTaskLinks 6126 agent assigned to terminal/missing task n/a task-link mismatch clear assignment RECONCILE-ONLY keep n/a n/a
recoverOrphanedAgents 6180 dead parent/direct-report linkage n/a org topology checks pause/delete/reparent decisions RECONCILE-ONLY keep n/a n/a
recoverStaleHeartbeatRuns 6372 stale heartbeat run records run age thresholds pid/age mismatch terminate run records RECONCILE-ONLY keep n/a n/a
recoverNoProgressNoTaskDoneFailures 6451 in-progress failed no-task-done no progress implicit (no explicit grace) no-step-progress + no git work + not executing clear metadata + move to todo BACKWARD tighten triple proof + no-progress checks + recent liveness-audit absence gate move; emit task:no-progress-no-task-done-no-action
recoverMissingWorktreeReviewFailures 6516 in-review failed OR merge-active (merging/merging-pr/merging-fix) session-start missing/unusable worktree classifier-based error classifier proof only autoRecover requeue to todo BACKWARD tightened triple proof + classifier proof + allowsAutoMergeProcessing + workspace-task exclusion gate requeue; emit task:missing-worktree-review-no-action or task:reconcile-missing-worktree-merge-active-no-action; successful merge-active recovery clears worktree/branch/sessionFile, resets the worktree-session retry budget, increments recoveryRetryCount as the bounded stale-metadata clear counter, emits task:reconcile-missing-worktree-merge-active, and requeues to todo preserving progress
recoverPartialProgressNoTaskDoneFailures 6586 in-review failed no-task-done with partial progress bounded by MAX_TASK_DONE_RETRIES no-task-done + partial progress + retry budget clear error + move to todo preserveProgress BACKWARD tighten triple proof + retry-budget predicates gate move; emit task:partial-progress-no-task-done-no-action
recoverApprovedTriageTasks 6706 triage planning specified stale APPROVED_TRIAGE_RECOVERY_GRACE_MS planning idle + valid PROMPT.md recoverApprovedTriageTask callback FORWARD keep n/a n/a
recoverStarvedRefinementTriageTasks 6827 refinement planning stale STARVED_REFINEMENT_RECOVERY_GRACE_MS no progress idle requeue/annotation in triage RECONCILE-ONLY keep n/a n/a
recoverOrphanedPlanningTasks 6940 planning-status tasks drifted APPROVED_TRIAGE_RECOVERY_GRACE_MS planning drift + inactive clear planning status RECONCILE-ONLY keep n/a n/a

Per-stage rationale

FN-8924 — retain age-based interrupted-merge recovery

  • Decision: REJECT. recoverInterruptedMergingTasks() retains Date.now() - task.updatedAt >= settings.taskStuckTimeoutMs for its non-owner grace gate.
  • Concrete counter-evidence: the required replacement clock, getLastMergerAgentActivityMs(taskId), reads entries written by the merger itself. The merge body's log() helper in packages/engine/src/merge/merger-ai.ts appends a task logEntry and a "merger" agent log together (lines 1317–1321; duplicated by the workspace helper at 1869–1873). These diagnostic writes are intentionally best-effort/unfenced, so an abandoned or superseded merge body can refresh both updatedAt and the proposed attributed clock. Merger-recency is therefore not an orphan-proof, task-scoped clock and cannot satisfy FN-8924's adoption bias.
  • The process-global active-merge-start clock is not an alternative: measureActiveMergeSilenceMs() falls back to getActiveMergeStartedAtMs() only for the owner path, where isActiveMergeWedged() first proves task identity. It has no task parameter and can belong to another merge. No durable cross-process merge-ownership claim or lease exists; the remaining pending/task-active/session/lock signals are in-process only, leaving the same cross-process blind spot that the retained age gate already has.
  • Consequently, the owner arm and all mutation-time recovery behavior remain unchanged. The proposed two-phase re-proof and routing-input divergence design is not adopted without a sound admission clock; changing sibling stale-status, deadlock, or dashboard Retry definitions would create unsupported divergent semantics. The regression test pins this contract: a fresh-updatedAt, unowned merge-active task is not recovered even if an older merger log would satisfy the rejected quiet-window proposal.

recoverStaleIncompleteReviewTasks

  • Current evidence is age + incomplete steps only; this can race live executor/session churn.
  • Tighten chosen: preserve recovery semantics but require dead-session + unusable-worktree + stale activity proof.
  • Composes with FN-5147 first (autoMerge false short-circuit), then FN-5335 proof gate.

recoverGhostReviewTasks

  • Current evidence is idle age + status exclusion; no hard proof task is dead.
  • Tighten chosen to avoid speculative in-review→todo regressions.

reclaimPrConflictForTask / reclaimSelfOwnedBranchConflicts

  • These can push in-review backward to todo on conflict reclaim paths.
  • Tighten chosen for backward path only; forward/metadata-only branches unchanged.

finalizeNoOpReviewTasks (fallback backward edge)

  • Method is mostly forward finalize; fallback requeue path can move backward.
  • Tighten only that branch.

recoverStuckMergeDeadlocks (no-landed branch)

  • Existing no-landed path pauses immediately after failed landed-proof.
  • Tighten chosen: pause/fail mutation requires triple proof; otherwise annotate and leave terminal state untouched.

recoverNoProgressNoTaskDoneFailures

  • Current predicate checks progress/git cleanliness but not session/worktree/liveness triple-proof.
  • Tighten to prevent reopening actively recovering tasks and to compose with FN-4935 recent liveness events.

recoverMissingWorktreeReviewFailures

  • Existing classifier proof is necessary but not sufficient for backward movement.
  • Tightened: require full triple-proof before auto requeue, and keep allowsAutoMergeProcessing as the first lifecycle gate so autoMerge:false remains terminal-until-merged.
  • Merge-active rows (merging, merging-pr, merging-fix) with the same unusable-worktree session-start signature are now owned here before interrupted/deadlocked merge sweeps can re-drive the phantom path. Workspace tasks are annotation-only in this path; per-repo land recovery remains workspace-owned.

recoverPartialProgressNoTaskDoneFailures

  • Retry-budget alone can still reopen tasks under active/residual execution races.
  • Tighten: add triple-proof gate before each requeue.

surfaceInReviewStalls deadlock branch

  • Classified BACKWARD but retained as already-tightened due explicit deadlock threshold + FN-5147 terminal-until-merged guard contract.

Stages explicitly left alone

  • Forward-only: recoverCompletedTasks, recoverStrandedCompletedTodoTasks, recoverMergedReviewTasks, recoverAlreadyMergedReviewTasks, recoverBranchMisboundInReviewTasks, recoverOrphanOnlyScopeViolations.
  • Already-tightened proof owners: recoverInProgressLimbo (FN-5219), recoverCompletionHandoffLimbo (FN-4999), recoverOrphanedExecutions (FN-5337 task:orphan-detected-no-action).
  • RECONCILE/annotation-only stages remain unchanged.

Step 1 self-check coverage

Executed mutation-scan grep -nE 'moveTask\(.*"todo"|moveTask\(.*"in-progress"|updateTask\(.*paused:\s*true|status:\s*"failed"' packages/engine/src/self-healing.ts and confirmed every mutation site is represented above with explicit direction + disposition (including autoReboundPausedScopeDecayDetailed, surfaceInReviewStalls deadlock branch, and recoverStuckMergeDeadlocks no-landed pause branch).

Open questions / out-of-scope

  • None discovered in Step 1 requiring separate task creation.