root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(FN-5335): enforce triple-proof gating for backward recovery

Browse Source 
- Gate self-healing backward moves behind audited triple-proof predicates across reclaim paths
- Skip reclaim-pr-conflict mutations when proof checks fail and preserve no-action behavior
- Add broad unit and reliability-interaction coverage for triple-proof and cross-layer scenarios
- Document backward-move stage invariants, diagnostics, and add delivery changeset for @runfusion/fusion
This commit is contained in:
Fusion (runfusion.ai)
2026-05-21 19:15:27 -07:00
committed by gsxdsm
parent a2a5db8151
commit a8715ed963
16 changed files with 914 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/architecture.md
View File

@@ -1061,6 +1061,7 @@ The run-audit system records every mutation performed by the engine across four
- **Git / `merge:no-op-attribution-mismatch-skipped`** — emitted when the FN-5304 source-tip guard cannot run because the source branch ref is unavailable (for example already pruned). `target` is the task ID; metadata includes `reason` (`"source-ref-unavailable"`).
- **Database / `task:auto-recover-misrouted-foreign-commit`** — emitted per dropped misrouted commit during FN-4948 contamination recovery. `target` is the recovering task; metadata carries `{ droppedSha, foreignTaskId, paths }`.
- **Database / `task:orphan-detected-no-action`** — emitted by `recoverOrphanedExecutions` (FN-5337) when row metadata looks orphaned after grace windows; annotation-only event with no lifecycle mutation (`in-progress` task stays put).
- **Database / `task:*-no-action` backward-move family (FN-5335)** — backward self-healing sweeps now emit annotation-only events when triple proof fails instead of mutating lifecycle state. New mutation types: `task:reclaim-pr-conflict-no-action`, `task:reclaim-self-owned-branch-conflict-no-action`, `task:auto-rebound-scope-decay-no-action`, `task:finalize-no-op-review-no-action`, `task:stale-incomplete-review-no-action`, `task:ghost-review-no-action`, `task:stuck-merge-deadlock-no-action`, `task:no-progress-no-task-done-no-action`, `task:missing-worktree-review-no-action`, `task:partial-progress-no-task-done-no-action`. See `docs/self-healing-backward-move-audit.md` for per-stage disposition.
- **Filesystem** — file:write, prompt:write, attachment:create, etc.
- **Sandbox** — backend lifecycle events from `SandboxBackend` wiring in executor/merger/routine-runner (`sandbox:prepare`, `sandbox:run`, `sandbox:failure`, `sandbox:fallback`) introduced after FN-4636.

3
docs/diagnostics.md
View File

@@ -44,6 +44,9 @@ The process supervisor logs when it registers a supervised child, starts teardow
- `surface-stale-paused-reviews`
- Log prefix: `Stale paused review surfaced [stale-paused-review]: paused ...`
- Purpose: paused in-review backlog-health detector gated by `stalePausedReviewThresholdMs`.
- FN-5335 backward-move annotations
- Log prefix shape: `[<stage-name>] <taskId>: triple-proof not satisfied — no action (operator-decides)`
- Representative stage names: `no-progress-no-task-done`, `partial-progress-no-task-done`, `stale-incomplete-review`, `ghost-review`, `missing-worktree-review`, `stuck-merge-deadlock`, `finalize-no-op-review`, `reclaim-pr-conflict`, `reclaim-self-owned-branch-conflict`, `auto-rebound-paused-scope-decay`.
## No-progress churn stuck-task escalation (`[executor]`, `[stuck-detector]`, `[self-healing]`)

110
docs/self-healing-backward-move-audit.md Normal file
View File

@@ -0,0 +1,110 @@
# Self-Healing Backward-Move Audit (FN-5335)
## Invariant
Backward lifecycle moves (in-progress → todo, in-review → in-progress/todo, in-review → paused-stuck/failed) from a maintenance sweep require **triple proof**:
1. Verified dead session — no active session registry entry for any `pathsForTask(taskId)`, no `executingTaskLock.has(taskId)`, and `this.options.isTaskActive?.(taskId) !== true`.
2. Unusable worktree — `classifyTaskWorktree(this.options.rootDir, task.worktree)` returns `{ ok: false }`, OR `task.worktree` is null and there is no registered `fusion/<id>` worktree.
3. No recent executor activity — stage staleness anchor exceeds stage grace window and there is no recent `worktree:incomplete-detected` event within the same trailing window.
Stages that cannot satisfy all three must either (a) tighten predicate to require triple proof, or (b) downgrade to annotation-only (`task:<stage>-no-action`).
## Catalog
| Method | Source line | Trigger condition | Grace source | Evidence required today | Action today | Direction | Disposition | New evidence required | New action |
|---|---:|---|---|---|---|---|---|---|---|
| recoverCompletedTasks | 1455 | in-progress + all steps terminal | n/a | step completion | move to in-review | FORWARD | keep | n/a | n/a |
| recoverStrandedCompletedTodoTasks | 1507 | todo + all steps terminal | n/a | step completion | move to in-progress (resume) | FORWARD | keep | n/a | n/a |
| recoverStaleMergingStatus | 1557 | in-review stale merging status | `staleMergingStatusMinAgeMs` | stale status + no active merge | clear status | RECONCILE-ONLY | keep | n/a | n/a |
| reclaimPrConflicts | 1606 | PR mergeable=conflicting | path-dependent | conflict inspect result | delegates to reclaimPrConflictForTask | RECONCILE/BACKWARD mix | keep | n/a | n/a |
| reclaimPrConflictForTask | 1622 | reclaimable conflicting PR branch | inspectConflict + usable worktree checks | active-session/usable-worktree checks only | may move in-review→todo | BACKWARD | tighten | triple proof on in-review→todo path | gate move; emit `task:reclaim-pr-conflict-no-action` |
| reclaimSelfOwnedBranchConflicts | 1772 | branch conflict self-owned | conflict inspector local checks | conflict classifier only | may move in-review→todo | BACKWARD | tighten | triple proof before backward move | gate move; emit `task:reclaim-self-owned-branch-conflict-no-action` |
| reclaimStaleActiveBranches | 2174 | stale active branch metadata | `STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS` | age + no active session + no local changes | metadata clear/prune | RECONCILE-ONLY | keep | n/a | n/a |
| reconcileCompletedTask | 2438 | done fanout cleanup | n/a | dependency unblock checks | blocker/status metadata updates | RECONCILE-ONLY | keep | n/a | n/a |
| reconcileInReviewBranchRebind | 2646 | in-review missing branch metadata | n/a | unique candidate branch proof | branch metadata rebind/clear | RECONCILE-ONLY | keep | n/a | n/a |
| reconcileTaskWorktreeMetadata | 2815 | stale task.worktree metadata | mixed (includes stale merge thresholds) | registered-worktree reconciliation | metadata fixes (+ targeted no-op move hygiene) | RECONCILE-ONLY | keep | n/a | n/a |
| autoReboundPausedScopeDecayDetailed | 2922 | paused in-progress scope holder stale beyond decay threshold | `settings.pausedScopeDecayMs` | age + scoped follower count | move to todo (preserve progress/worktree/resume state) | BACKWARD | tighten | triple proof + existing decay predicates | gate move; emit `task:auto-rebound-scope-decay-no-action` |
| surfaceDbCorruption | 3259 | db corruption marker | notification cooldown | db marker presence | notify/log | ANNOTATION-ONLY | keep | n/a | n/a |
| reconcileSelfDefeatingDependencies | 3499 | dependency self-loop | n/a | graph proof | dependency metadata repair | RECONCILE-ONLY | keep | n/a | n/a |
| reconcileDependencyCycles | 3557 | persisted dependency cycle | n/a | cycle detector | selective dependency edge repair | RECONCILE-ONLY | keep | n/a | n/a |
| reconcileStaleMergerStatus | 3702 | stale merger status | `staleMergingStatusMinAgeMs` | age/no active merge | clear status | RECONCILE-ONLY | keep | n/a | n/a |
| finalizeNoOpReviewTasks | 3750 | in-review already-on-main no-op | git landed proof | landed/no-op proof | mostly in-review→done; one in-review→todo remediation path | FORWARD + BACKWARD edge | tighten (backward edge only) | triple proof for fallback in-review→todo path | gate fallback; emit `task:finalize-no-op-review-no-action` |
| reconcileDoneTaskIntegrity | 3870 | done task integrity mismatch | n/a | integrity check | metadata/status hygiene | RECONCILE-ONLY | keep | n/a | n/a |
| recoverMergeableReviewTasks | 3996 | mergeable in-review task | merge blocker checks | merge readiness | queue/merge status writes | INTERNAL-RETRY | keep | n/a | n/a |
| recoverReviewTasksWithFailedPreMergeSteps | 4112 | failed pre-merge workflow step | maxFix budget | failed step + budget | revive via executor | INTERNAL-RETRY | keep | n/a | n/a |
| recoverStaleIncompleteReviewTasks | 4205 | in-review with incomplete steps stale | `settings.taskStuckTimeoutMs` | age + incomplete steps | move to todo preserveProgress | BACKWARD | tighten | triple proof + existing stale/incomplete predicates | gate move; emit `task:stale-incomplete-review-no-action` |
| surfaceInReviewStalls | 4281 | in-review stall reasons | `settings.taskStuckTimeoutMs`, `inReviewStallDeadlockThreshold` | stall signal + repetition | log; deadlock branch pauses/failed | BACKWARD branch | already-tightened (threshold + FN-5147) | keep existing | no change |
| surfaceDependencyBlockedTodos | 4390 | blocked todo report | reporter internals | blocked dependency snapshot | report/log | ANNOTATION-ONLY | keep | n/a | n/a |
| surfaceInReviewStalled | 4416 | quiet in-review backlog signal | `settings.inReviewStalledThresholdMs` | quiet-window signal | log only | ANNOTATION-ONLY | keep | n/a | n/a |
| surfaceStalePausedReviews | 4478 | stale paused in-review | `settings.stalePausedReviewThresholdMs` | age signal | log only | ANNOTATION-ONLY | keep | n/a | n/a |
| surfaceStalePausedTodos | 4529 | stale paused todo | `settings.stalePausedTodoThresholdMs` | age signal | log only | ANNOTATION-ONLY | keep | n/a | n/a |
| recoverGhostReviewTasks | 4583 | idle in-review ghost | `settings.taskStuckTimeoutMs` | idle + status filters | move to todo preserveProgress | BACKWARD | tighten | triple proof + existing ghost predicates | gate move; emit `task:ghost-review-no-action` |
| recoverInterruptedMergingTasks | 4650 | stale transient merging status | `settings.taskStuckTimeoutMs` | stale status + landed-commit detection | done finalize or status clear/requeue | FORWARD/INTERNAL-RETRY | keep | n/a | n/a |
| recoverDoneTaskMergeMetadata | 4777 | done merge metadata drift | n/a | landed evidence + metadata gap | metadata update | RECONCILE-ONLY | keep | n/a | n/a |
| recoverMergedReviewTasks | 4937 | in-review already merged | landed commit proof | landed proof | move to done | FORWARD | keep | n/a | n/a |
| recoverStuckMergeDeadlocks (landed) | 5036 | retry-exhausted merge failed but landed | cooldown + retries | landed proof | move to done + unblock deps | FORWARD | keep | n/a | n/a |
| recoverStuckMergeDeadlocks (no-landed) | 5036 | retry-exhausted merge failed and not landed | `DEADLOCK_RECOVERY_COOLDOWN_MS` | no landed proof only | `updateTask({ paused: true })` + log | BACKWARD | tighten | triple proof before pause/fail disposition | gate pause; emit `task:stuck-merge-deadlock-no-action` |
| recoverOrphanOnlyScopeViolations | 5192 | failed scope-only orphan but landed | landed proof | parsed scope payload + landed proof | move to done or failed-blocked | FORWARD/RECONCILE | keep | n/a | n/a |
| recoverAlreadyMergedReviewTasks | 5350 | retry-exhausted failed review but landed | retries threshold | landed proof | move to done | FORWARD | keep | n/a | n/a |
| recoverCompletionHandoffLimbo | 5473 | in-review no handoff state after task_done | `COMPLETION_HANDOFF_LIMBO_GRACE_MS` | done-marker age + no active task + merge blocker absent + retry cap | requeue merge or fail-exhausted | already proof-gated | already-tightened (FN-4999) | keep | no change |
| recoverBranchMisboundInReviewTasks | 5581 | in-review failed merge lineage misbound but landed | retries threshold | classifyOwnedLandedEvidence proof | move to done | FORWARD | keep | n/a | n/a |
| recoverForeignOnlyContaminatedInReviewTasks | 5692 | foreign-only contamination | retries/config gates | contamination attribution proof | reanchor/requeue or annotate | INTERNAL-RETRY | keep | n/a | n/a |
| recoverMisclassifiedFailures | 5789 | in-review failed but no merge blocker and all steps done | n/a | completed steps + no blocker | clear status/error | RECONCILE-ONLY | keep | n/a | n/a |
| recoverInProgressLimbo | 5877 | in-progress no branch + missing/none worktree + pending steps | `ORPHANED_EXECUTION_RECOVERY_GRACE_MS` | dead/no-progress proof set | move to todo preserveProgress | BACKWARD | already-tightened (FN-5219) | keep | no change |
| recoverOrphanedExecutions | 5991 | stale in-progress no active execution | `ORPHANED_EXECUTION_RECOVERY_GRACE_MS` / `ORPHANED_WITH_WORKTREE_GRACE_MS` | stale orphan candidate only | annotation event only | ANNOTATION-ONLY | already-tightened (FN-5337) | keep | no change |
| recoverAgentsRunningOnInactiveTasks | 6089 | agent runtime drift | agent heartbeat ages | agent-store runtime checks | end/restart agents | RECONCILE-ONLY | keep | n/a | n/a |
| recoverDriftedAgentTaskLinks | 6126 | agent assigned to terminal/missing task | n/a | task-link mismatch | clear assignment | RECONCILE-ONLY | keep | n/a | n/a |
| recoverOrphanedAgents | 6180 | dead parent/direct-report linkage | n/a | org topology checks | pause/delete/reparent decisions | RECONCILE-ONLY | keep | n/a | n/a |
| recoverStaleHeartbeatRuns | 6372 | stale heartbeat run records | run age thresholds | pid/age mismatch | terminate run records | RECONCILE-ONLY | keep | n/a | n/a |
| recoverNoProgressNoTaskDoneFailures | 6451 | in-progress failed no-task-done no progress | implicit (no explicit grace) | no-step-progress + no git work + not executing | clear metadata + move to todo | BACKWARD | tighten | triple proof + no-progress checks + recent liveness-audit absence | gate move; emit `task:no-progress-no-task-done-no-action` |
| recoverMissingWorktreeReviewFailures | 6516 | in-review failed session-start missing/unusable worktree | classifier-based | error classifier proof only | autoRecover requeue to todo | BACKWARD | tighten | triple proof + classifier proof | gate requeue; emit `task:missing-worktree-review-no-action` |
| recoverPartialProgressNoTaskDoneFailures | 6586 | in-review failed no-task-done with partial progress | bounded by `MAX_TASK_DONE_RETRIES` | no-task-done + partial progress + retry budget | clear error + move to todo preserveProgress | BACKWARD | tighten | triple proof + retry-budget predicates | gate move; emit `task:partial-progress-no-task-done-no-action` |
| recoverApprovedTriageTasks | 6706 | triage planning approved stale | `APPROVED_TRIAGE_RECOVERY_GRACE_MS` | planning idle + approved spec | recoverApprovedTriageTask callback | FORWARD | keep | n/a | n/a |
| recoverStarvedRefinementTriageTasks | 6827 | refinement planning stale | `STARVED_REFINEMENT_RECOVERY_GRACE_MS` | no progress idle | requeue/annotation in triage | RECONCILE-ONLY | keep | n/a | n/a |
| recoverOrphanedPlanningTasks | 6940 | planning-status tasks drifted | `APPROVED_TRIAGE_RECOVERY_GRACE_MS` | planning drift + inactive | clear planning status | RECONCILE-ONLY | keep | n/a | n/a |
## Per-stage rationale
### recoverStaleIncompleteReviewTasks
- Current evidence is age + incomplete steps only; this can race live executor/session churn.
- Tighten chosen: preserve recovery semantics but require dead-session + unusable-worktree + stale activity proof.
- Composes with FN-5147 first (autoMerge false short-circuit), then FN-5335 proof gate.
### recoverGhostReviewTasks
- Current evidence is idle age + status exclusion; no hard proof task is dead.
- Tighten chosen to avoid speculative in-review→todo regressions.
### reclaimPrConflictForTask / reclaimSelfOwnedBranchConflicts
- These can push in-review backward to todo on conflict reclaim paths.
- Tighten chosen for backward path only; forward/metadata-only branches unchanged.
### finalizeNoOpReviewTasks (fallback backward edge)
- Method is mostly forward finalize; fallback requeue path can move backward.
- Tighten only that branch.
### recoverStuckMergeDeadlocks (no-landed branch)
- Existing no-landed path pauses immediately after failed landed-proof.
- Tighten chosen: pause/fail mutation requires triple proof; otherwise annotate and leave terminal state untouched.
### recoverNoProgressNoTaskDoneFailures
- Current predicate checks progress/git cleanliness but not session/worktree/liveness triple-proof.
- Tighten to prevent reopening actively recovering tasks and to compose with FN-4935 recent liveness events.
### recoverMissingWorktreeReviewFailures
- Existing classifier proof is necessary but not sufficient for backward movement.
- Tighten: require full triple-proof before auto requeue.
### recoverPartialProgressNoTaskDoneFailures
- Retry-budget alone can still reopen tasks under active/residual execution races.
- Tighten: add triple-proof gate before each requeue.
### surfaceInReviewStalls deadlock branch
- Classified BACKWARD but retained as already-tightened due explicit deadlock threshold + FN-5147 terminal-until-merged guard contract.
## Stages explicitly left alone
- Forward-only: `recoverCompletedTasks`, `recoverStrandedCompletedTodoTasks`, `recoverMergedReviewTasks`, `recoverAlreadyMergedReviewTasks`, `recoverBranchMisboundInReviewTasks`, `recoverOrphanOnlyScopeViolations`.
- Already-tightened proof owners: `recoverInProgressLimbo` (FN-5219), `recoverCompletionHandoffLimbo` (FN-4999), `recoverOrphanedExecutions` (FN-5337 `task:orphan-detected-no-action`).
- RECONCILE/annotation-only stages remain unchanged.
## Step 1 self-check coverage
Executed mutation-scan `grep -nE 'moveTask\(.*"todo"|moveTask\(.*"in-progress"|updateTask\(.*paused:\s*true|status:\s*"failed"' packages/engine/src/self-healing.ts` and confirmed every mutation site is represented above with explicit direction + disposition (including `autoReboundPausedScopeDecayDetailed`, `surfaceInReviewStalls` deadlock branch, and `recoverStuckMergeDeadlocks` no-landed pause branch).
## Open questions / out-of-scope
- None discovered in Step 1 requiring separate task creation.