FN-5704: escalate self-owned execution limbo instead of endless resume

Escalate reclaimable in-progress resume limbo into an explicit failure path to stop infinite self-healing loops.

- add resume-limbo escalation handling in self-healing/executor flow so self-owned stuck execution is failed and surfaced
- extend core task/run-audit types and retry-reset/store behavior to persist and expose the new escalation state
- add reliability interaction coverage for reclaim self-owned resume limbo escalation and update related schema/store/CLI/plugin tests and docs

Files changed:
 AGENTS.md                                          |   1 +
 docs/architecture.md                               |   2 +
 packages/cli/src/commands/__tests__/task.test.ts   |   2 +
 packages/core/src/__tests__/db-migrate.test.ts     |  12 +-
 packages/core/src/__tests__/db.test.ts             |  34 ++---
 packages/core/src/__tests__/goals-schema.test.ts   |   2 +-
 packages/core/src/__tests__/insight-store.test.ts  |  10 +-
 packages/core/src/__tests__/mission-store.test.ts  |   2 +-
 packages/core/src/__tests__/run-audit.test.ts      |   2 +-
 packages/core/src/__tests__/secrets-schema.test.ts |   6 +-
 .../core/src/__tests__/store-merge-queue.test.ts   |   2 +-
 packages/core/src/__tests__/task-documents.test.ts |   2 +-
 packages/core/src/db.ts                            |  13 +-
 packages/core/src/manual-retry-reset.ts            |   1 +
 packages/core/src/store.ts                         |  37 ++++-
 packages/core/src/types.ts                         |  11 ++
 ...laim-self-owned-resume-limbo-escalation.test.ts | 168 +++++++++++++++++++++
 packages/engine/src/executor.ts                    |   5 +
 packages/engine/src/run-audit.ts                   |   1 +
 packages/engine/src/self-healing.ts                |  72 +++++++++
 .../src/store/__tests__/roadmap-store.test.ts      |   4 +-
 21 files changed, 345 insertions(+), 44 deletions(-)

Fusion-Task-Id: FN-5704

Fusion-Task-Lineage: c8f73d16-d95b-450f-a514-e7d3c2f7aebe
This commit is contained in:
gsxdsm
2026-05-29 19:37:54 -07:00
parent 129e17d16c
commit 0dbb1cd6f9
21 changed files with 345 additions and 44 deletions

View File

@@ -158,6 +158,7 @@ When `settings.autoMerge: false`, `in-review` is terminal-until-merged by a huma
- FN-5432 backstop: `packages/engine/src/__tests__/reliability-interactions/dependency-cycle-reconcile.test.ts` extends FN-5256 coverage with long-cycle ambiguous sweep, write-boundary/sweep race, self-defeating+cycle non-contradiction across one maintenance flow, and audit-event shape regression; core regression cases (long cycle, self-loop via update, incremental-update closes a loop, moveTask seam invariant, DependencyCycleError shape) live in `packages/core/src/__tests__/store-dependency-cycle.test.ts`. User-facing pull/stash audit event behavior (`pull:fast-forward`, `stash:pop-conflict`) is documented in `docs/dashboard-guide.md` under Merge Advance Notice / Smart Pull. - FN-5432 backstop: `packages/engine/src/__tests__/reliability-interactions/dependency-cycle-reconcile.test.ts` extends FN-5256 coverage with long-cycle ambiguous sweep, write-boundary/sweep race, self-defeating+cycle non-contradiction across one maintenance flow, and audit-event shape regression; core regression cases (long cycle, self-loop via update, incremental-update closes a loop, moveTask seam invariant, DependencyCycleError shape) live in `packages/core/src/__tests__/store-dependency-cycle.test.ts`. User-facing pull/stash audit event behavior (`pull:fast-forward`, `stash:pop-conflict`) is documented in `docs/dashboard-guide.md` under Merge Advance Notice / Smart Pull.
- FN-5403 backstop: `packages/engine/src/__tests__/reliability-interactions/engine-stop-aborts-execution.test.ts` locks stop-ordering behavior so engine shutdown aborts executor AI sessions before drain wait and preserves task-row lifecycle semantics. - FN-5403 backstop: `packages/engine/src/__tests__/reliability-interactions/engine-stop-aborts-execution.test.ts` locks stop-ordering behavior so engine shutdown aborts executor AI sessions before drain wait and preserves task-row lifecycle semantics.
- FN-5704 backstop: `packages/engine/src/__tests__/reliability-interactions/reclaim-self-owned-resume-limbo-escalation.test.ts` guards reclaim/unpause no-progress oscillation recovery by capping repeated no-progress resumes, escalating to preserve-work `todo` rebound, and emitting `task:resume-limbo-escalated` audit metadata while exempting progress/user-paused/autoMerge-off cases.
--- ---

View File

@@ -1738,6 +1738,7 @@ This section preserves the detailed lifecycle/self-healing contracts that were f
- **In-review is terminal-until-merged under `autoMerge: false` (FN-5147)**: when a project sets `settings.autoMerge: false`, `in-review` is the intended resting state until a human merges the PR. No lifecycle-mutating self-healing sweep (`reclaimSelfOwnedBranchConflicts`, `recoverGhostReviewTasks`, `recoverStaleIncompleteReviewTasks`, `recoverInterruptedMergingTasks`, `recoverStuckMergeDeadlocks`, `recoverMissingWorktreeReviewFailures`, `recoverPartialProgressNoTaskDoneFailures`, `recoverCompletionHandoffLimbo`, `recoverMergeableReviewTasks`, `recoverMergedReviewTasks`, `recoverAlreadyMergedReviewTasks`, `recoverOrphanOnlyScopeViolations`, `recoverForeignOnlyContaminatedInReviewTasks`, `recoverReviewTasksWithFailedPreMergeSteps`, `finalizeNoOpReviewTasks`, `surfaceInReviewStalls`, `surfaceInReviewStalled`) may move the task out of `in-review`, mark it `paused`/`failed`, or re-enqueue it for execution. RECONCILE-ONLY sweeps (branch rebind, blocker fan-out, stale-status clears, contamination metadata cleanup, attribution restore, PR refresh, misclassified-failure error clearing) continue to run. - **In-review is terminal-until-merged under `autoMerge: false` (FN-5147)**: when a project sets `settings.autoMerge: false`, `in-review` is the intended resting state until a human merges the PR. No lifecycle-mutating self-healing sweep (`reclaimSelfOwnedBranchConflicts`, `recoverGhostReviewTasks`, `recoverStaleIncompleteReviewTasks`, `recoverInterruptedMergingTasks`, `recoverStuckMergeDeadlocks`, `recoverMissingWorktreeReviewFailures`, `recoverPartialProgressNoTaskDoneFailures`, `recoverCompletionHandoffLimbo`, `recoverMergeableReviewTasks`, `recoverMergedReviewTasks`, `recoverAlreadyMergedReviewTasks`, `recoverOrphanOnlyScopeViolations`, `recoverForeignOnlyContaminatedInReviewTasks`, `recoverReviewTasksWithFailedPreMergeSteps`, `finalizeNoOpReviewTasks`, `surfaceInReviewStalls`, `surfaceInReviewStalled`) may move the task out of `in-review`, mark it `paused`/`failed`, or re-enqueue it for execution. RECONCILE-ONLY sweeps (branch rebind, blocker fan-out, stale-status clears, contamination metadata cleanup, attribution restore, PR refresh, misclassified-failure error clearing) continue to run.
- **Auto-merge integration-root default (FN-5279)**: direct auto-merge now defaults `mergeIntegrationWorktree` to `reuse-task-worktree`; merger must pass the reuse handoff gates or emit `merge:reuse-handoff-refused` and leave the task in `in-review` without silently falling back to `cwd-integration-branch` (`cwd-main` remains a deprecated alias normalized to that mode). - **Auto-merge integration-root default (FN-5279)**: direct auto-merge now defaults `mergeIntegrationWorktree` to `reuse-task-worktree`; merger must pass the reuse handoff gates or emit `merge:reuse-handoff-refused` and leave the task in `in-review` without silently falling back to `cwd-integration-branch` (`cwd-main` remains a deprecated alias normalized to that mode).
- **Orphaned execution sweep is observation-only (FN-5337)**: `recoverOrphanedExecutions` only annotates stale in-progress candidates with `task:orphan-detected-no-action` and `[orphan-detected] ... no action (operator-decides)` logs. It must never move `in-progress`/`in-review` backward to `todo` or mutate lease/worktree metadata. Proof-based backward recovery remains exclusively in `recoverInProgressLimbo` (FN-5219), `RestartRecoveryCoordinator`, `recoverMissingWorktreeReviewFailures`, and explicit executor/merger failure paths. Reintroducing lifecycle mutation here requires hard git/session proof gating plus CEO+CTO+PM sign-off. - **Orphaned execution sweep is observation-only (FN-5337)**: `recoverOrphanedExecutions` only annotates stale in-progress candidates with `task:orphan-detected-no-action` and `[orphan-detected] ... no action (operator-decides)` logs. It must never move `in-progress`/`in-review` backward to `todo` or mutate lease/worktree metadata. Proof-based backward recovery remains exclusively in `recoverInProgressLimbo` (FN-5219), `RestartRecoveryCoordinator`, `recoverMissingWorktreeReviewFailures`, and explicit executor/merger failure paths. Reintroducing lifecycle mutation here requires hard git/session proof gating plus CEO+CTO+PM sign-off.
- **Self-owned reclaim resume-limbo escalation (FN-5704)**: `reclaimSelfOwnedBranchConflicts` tracks `resumeLimboCount`, `resumeLimboTipSha`, and `resumeLimboStepSignature` for in-progress reclaim/unpause loops. If reclaim finds no progress (same tip, same step-status signature, and no active-session signal) for `MAX_NO_PROGRESS_RESUME_ATTEMPTS` consecutive sweeps, self-healing escalates by moving the task to `todo` with `preserveWorktree: true`, `preserveProgress: true`, and `preserveResumeState: true` instead of endlessly re-arming resume. Escalation emits `task:resume-limbo-escalated` run-audit metadata (`frozenTipSha`, `idleMs`, `resumeAttemptCount`, `currentStep`) and resets the limbo counter.
- **No-progress churn terminalization (FN-5168)**: `StuckTaskDetector` now tracks ignored `fn_task_update` rebuffs via `recordIgnoredStepUpdate(taskId)` and, after one loop/compact-and-resume recovery has already fired in the same `execute()` lifecycle, escalates `ignoredStepUpdateCount >= 25` to the terminal reason `no-progress-churn`. `SelfHealingManager.checkStuckBudget()` maps that reason directly to `STUCK_NO_PROGRESS_CHURN`, emits `task:stuck-no-progress-churn-terminalized` with `{ taskId, ignoredStepUpdateCount, stuckKillStreak, lastReason }`, and parks the task in `in-review` without consuming the normal stuck-kill budget. Under FN-5147 `autoMerge: false`, that failed in-review task remains terminal-until-merged just like `STUCK_LOOP_EXHAUSTED`; the new class adds an earlier bounded exit, not a re-execution path. - **No-progress churn terminalization (FN-5168)**: `StuckTaskDetector` now tracks ignored `fn_task_update` rebuffs via `recordIgnoredStepUpdate(taskId)` and, after one loop/compact-and-resume recovery has already fired in the same `execute()` lifecycle, escalates `ignoredStepUpdateCount >= 25` to the terminal reason `no-progress-churn`. `SelfHealingManager.checkStuckBudget()` maps that reason directly to `STUCK_NO_PROGRESS_CHURN`, emits `task:stuck-no-progress-churn-terminalized` with `{ taskId, ignoredStepUpdateCount, stuckKillStreak, lastReason }`, and parks the task in `in-review` without consuming the normal stuck-kill budget. Under FN-5147 `autoMerge: false`, that failed in-review task remains terminal-until-merged just like `STUCK_LOOP_EXHAUSTED`; the new class adds an earlier bounded exit, not a re-execution path.
- **Landed-files attribution (FN-5103)**: Rebase-strategy `mergeDetails.landedFiles` / `filesChanged` / `insertions` / `deletions` are captured from task-attributable commits only via `filterFilesToOwnTaskCommits` (subject-prefix + trailer + bracket-prefix evidence), tagged `landedFilesAttributionRestricted: true`. Zero own commits → `landedFiles: []` and `noOpVerifiedShortCircuit: true`. FN-5304 guard: when `<rebaseBaseSha>..HEAD` reports zero own commits, merger must also validate the source `fusion/<id>` tip; if that source tip still has attributable own commits relative to `rebaseBaseSha`, throw `SilentNoOpAttributionMismatchError`, refuse writing `mergeConfirmed: true`, park the task in `in-review` with `status: "failed"`, and emit `merge:no-op-attribution-mismatch`. If source ref is unavailable, skip with diagnostic + `merge:no-op-attribution-mismatch-skipped` (`reason: "source-ref-unavailable"`). Attribution-helper failures fall back to the unrestricted `rebaseBaseSha..sha` walk and set `landedFilesCaptureFallback: 'attribution-failed'`. Self-healing `recoverDoneTaskMergeMetadata` skips reconcile when `landedFilesAttributionRestricted` or `noOpVerifiedShortCircuit` is set so the narrower set is not overwritten with the full range. Squash-strategy capture is unchanged. - **Landed-files attribution (FN-5103)**: Rebase-strategy `mergeDetails.landedFiles` / `filesChanged` / `insertions` / `deletions` are captured from task-attributable commits only via `filterFilesToOwnTaskCommits` (subject-prefix + trailer + bracket-prefix evidence), tagged `landedFilesAttributionRestricted: true`. Zero own commits → `landedFiles: []` and `noOpVerifiedShortCircuit: true`. FN-5304 guard: when `<rebaseBaseSha>..HEAD` reports zero own commits, merger must also validate the source `fusion/<id>` tip; if that source tip still has attributable own commits relative to `rebaseBaseSha`, throw `SilentNoOpAttributionMismatchError`, refuse writing `mergeConfirmed: true`, park the task in `in-review` with `status: "failed"`, and emit `merge:no-op-attribution-mismatch`. If source ref is unavailable, skip with diagnostic + `merge:no-op-attribution-mismatch-skipped` (`reason: "source-ref-unavailable"`). Attribution-helper failures fall back to the unrestricted `rebaseBaseSha..sha` walk and set `landedFilesCaptureFallback: 'attribution-failed'`. Self-healing `recoverDoneTaskMergeMetadata` skips reconcile when `landedFilesAttributionRestricted` or `noOpVerifiedShortCircuit` is set so the narrower set is not overwritten with the full range. Squash-strategy capture is unchanged.
- **Soft-delete scheduler invalidation (FN-5137)**: `task:deleted` events must invalidate `AutoClaimSnapshotManager` and clear scheduler bookkeeping (`pausedTaskIds`, `failedTaskIds`, `wasNodeDispatchValidationBlocked`, `wasNodeBlocked`); `executor.execute()` / `resumeOrphaned()` / `resumeTaskForAgent()` refuse any task with `deletedAt` set. - **Soft-delete scheduler invalidation (FN-5137)**: `task:deleted` events must invalidate `AutoClaimSnapshotManager` and clear scheduler bookkeeping (`pausedTaskIds`, `failedTaskIds`, `wasNodeDispatchValidationBlocked`, `wasNodeBlocked`); `executor.execute()` / `resumeOrphaned()` / `resumeTaskForAgent()` refuse any task with `deletedAt` set.
@@ -1764,6 +1765,7 @@ Reliability-layer changes are in scope. Interaction regression backstops live in
- FN-5147 backstop: `packages/engine/src/__tests__/reliability-interactions/in-review-automerge-off.test.ts` covers `autoMerge: false` + long-quiet in-review + maintenance/startup sweep cycles, asserting no column move / no paused / no status mutation / no requeue, plus explicit regression guards for `surfaceInReviewStalls` and `surfaceInReviewStalled`. - FN-5147 backstop: `packages/engine/src/__tests__/reliability-interactions/in-review-automerge-off.test.ts` covers `autoMerge: false` + long-quiet in-review + maintenance/startup sweep cycles, asserting no column move / no paused / no status mutation / no requeue, plus explicit regression guards for `surfaceInReviewStalls` and `surfaceInReviewStalled`.
- FN-5168 backstop: `packages/engine/src/__tests__/reliability-interactions/non-progress-churn.test.ts` covers loop→compact recovery followed by ignored-step-update churn escalation, terminal `beforeRequeue(false)` behavior, audit/log payloads, and FN-5147 autoMerge-off composition. - FN-5168 backstop: `packages/engine/src/__tests__/reliability-interactions/non-progress-churn.test.ts` covers loop→compact recovery followed by ignored-step-update churn escalation, terminal `beforeRequeue(false)` behavior, audit/log payloads, and FN-5147 autoMerge-off composition.
- FN-5219 backstop: `packages/engine/src/__tests__/reliability-interactions/in-progress-limbo-recovery.test.ts` covers `recoverInProgressLimbo` composition with `recoverOrphanedExecutions` (no double-recovery), `reconcile-task-worktree-metadata` (live rebindable worktree wins), `recoverMissingWorktreeReviewFailures` (in-review vs in-progress disjoint), and executor task-id claim skip, plus an explicit FN-5149 reproduction case. - FN-5219 backstop: `packages/engine/src/__tests__/reliability-interactions/in-progress-limbo-recovery.test.ts` covers `recoverInProgressLimbo` composition with `recoverOrphanedExecutions` (no double-recovery), `reconcile-task-worktree-metadata` (live rebindable worktree wins), `recoverMissingWorktreeReviewFailures` (in-review vs in-progress disjoint), and executor task-id claim skip, plus an explicit FN-5149 reproduction case.
- FN-5704 backstop: `packages/engine/src/__tests__/reliability-interactions/reclaim-self-owned-resume-limbo-escalation.test.ts` covers bounded no-progress reclaim/resume detection, preserve-work escalation to `todo`, `task:resume-limbo-escalated` audit metadata, progress-signal reset behavior, and user-paused/autoMerge-off non-escalation guards.
- FN-5337 backstop: `packages/engine/src/__tests__/reliability-interactions/orphan-detected-no-requeue.test.ts` locks observation-only orphan detection across FN-5279 repro metadata desync, worktree-present and worktree-missing candidates, FN-5219 ordering, FN-5147 in-review isolation, FN-5083 branch-cleared composition, lease-manager non-invocation, and per-sweep idempotent audit emission. - FN-5337 backstop: `packages/engine/src/__tests__/reliability-interactions/orphan-detected-no-requeue.test.ts` locks observation-only orphan detection across FN-5279 repro metadata desync, worktree-present and worktree-missing candidates, FN-5219 ordering, FN-5147 in-review isolation, FN-5083 branch-cleared composition, lease-manager non-invocation, and per-sweep idempotent audit emission.
- FN-5256 backstop: `packages/engine/src/__tests__/reliability-interactions/dependency-cycle-reconcile.test.ts` covers persisted dependency-cycle detection via `reconcileDependencyCycles`, bounded umbrella-back-edge auto-repair, ambiguous-cycle observe-only behavior, composition ordering with `reconcileSelfDefeatingDependencies`, and the post-sweep write-time guard invariant. Core write-boundary regressions (FN-5240/5241/5242 signature, indirect cycle, umbrella back-edge rejection) live in `packages/core/src/__tests__/store-dependency-cycle.test.ts`. - FN-5256 backstop: `packages/engine/src/__tests__/reliability-interactions/dependency-cycle-reconcile.test.ts` covers persisted dependency-cycle detection via `reconcileDependencyCycles`, bounded umbrella-back-edge auto-repair, ambiguous-cycle observe-only behavior, composition ordering with `reconcileSelfDefeatingDependencies`, and the post-sweep write-time guard invariant. Core write-boundary regressions (FN-5240/5241/5242 signature, indirect cycle, umbrella back-edge rejection) live in `packages/core/src/__tests__/store-dependency-cycle.test.ts`.
- FN-5325 backstop: `packages/engine/src/__tests__/reliability-interactions/scheduler-overlap-priority-inversion.test.ts` covers queued-overlap priority/age deferral, equal-priority age ordering, FN-4969 fanout composition, and one-shot per-pass `scheduler:overlap-priority-inversion` audit surfacing against running lower-priority blockers. - FN-5325 backstop: `packages/engine/src/__tests__/reliability-interactions/scheduler-overlap-priority-inversion.test.ts` covers queued-overlap priority/age deferral, equal-priority age ordering, FN-4969 fanout composition, and one-shot per-pass `scheduler:overlap-priority-inversion` audit surfacing against running lower-priority blockers.

View File

@@ -2428,6 +2428,7 @@ describe("runTaskRetry", () => {
completionHandoffLimboRecoveryCount: 0, completionHandoffLimboRecoveryCount: 0,
mergeAuditBounceCount: 0, mergeAuditBounceCount: 0,
mergeRetries: 0, mergeRetries: 0,
resumeLimboCount: 0,
}); });
expect(mockMoveTask).toHaveBeenCalledWith("FN-001", "todo"); expect(mockMoveTask).toHaveBeenCalledWith("FN-001", "todo");
expect(mockLogEntry).toHaveBeenCalledWith("FN-001", "Retry requested from CLI", "Task reset to todo for retry"); expect(mockLogEntry).toHaveBeenCalledWith("FN-001", "Retry requested from CLI", "Task reset to todo for retry");
@@ -2501,6 +2502,7 @@ describe("runTaskRetry", () => {
completionHandoffLimboRecoveryCount: 0, completionHandoffLimboRecoveryCount: 0,
mergeAuditBounceCount: 0, mergeAuditBounceCount: 0,
mergeRetries: 0, mergeRetries: 0,
resumeLimboCount: 0,
}); });
expect(mockMoveTask).toHaveBeenCalledWith("FN-001", "todo"); expect(mockMoveTask).toHaveBeenCalledWith("FN-001", "todo");
expect(mockLogEntry).toHaveBeenCalledWith("FN-001", "Retry requested from CLI", "Task reset to todo for retry"); expect(mockLogEntry).toHaveBeenCalledWith("FN-001", "Retry requested from CLI", "Task reset to todo for retry");

View File

@@ -715,7 +715,7 @@ describe("schema migration", () => {
const row = db.prepare("SELECT deletedAt FROM tasks WHERE id = 'FN-legacy'").get() as { deletedAt: string | null }; const row = db.prepare("SELECT deletedAt FROM tasks WHERE id = 'FN-legacy'").get() as { deletedAt: string | null };
expect(row.deletedAt).toBeNull(); expect(row.deletedAt).toBeNull();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
db.close(); db.close();
}); });
@@ -748,7 +748,7 @@ describe("schema migration", () => {
{ id: "WS-001", mode: "prompt", gateMode: "advisory" }, { id: "WS-001", mode: "prompt", gateMode: "advisory" },
{ id: "WS-002", mode: "script", gateMode: "advisory" }, { id: "WS-002", mode: "script", gateMode: "advisory" },
]); ]);
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
db.close(); db.close();
}); });
@@ -798,7 +798,7 @@ describe("schema migration", () => {
reviewerContextRetryCount: 0, reviewerContextRetryCount: 0,
reviewerFallbackRetryCount: 0, reviewerFallbackRetryCount: 0,
}); });
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
db.close(); db.close();
}); });
@@ -827,7 +827,7 @@ describe("schema migration", () => {
const columns = db.prepare("PRAGMA table_info(milestones)").all() as Array<{ name: string }>; const columns = db.prepare("PRAGMA table_info(milestones)").all() as Array<{ name: string }>;
expect(columns.map((column) => column.name)).toContain("acceptanceCriteria"); expect(columns.map((column) => column.name)).toContain("acceptanceCriteria");
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
db.close(); db.close();
}); });
@@ -868,7 +868,7 @@ describe("schema migration", () => {
const missionColumns = db.prepare("PRAGMA table_info(missions)").all() as Array<{ name: string }>; const missionColumns = db.prepare("PRAGMA table_info(missions)").all() as Array<{ name: string }>;
expect(missionColumns.map((column) => column.name)).toContain("autoMerge"); expect(missionColumns.map((column) => column.name)).toContain("autoMerge");
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
db.close(); db.close();
}); });
@@ -902,7 +902,7 @@ describe("schema migration", () => {
{ id: "WS-002", mode: "script", enabled: 1, gateMode: "advisory" }, { id: "WS-002", mode: "script", enabled: 1, gateMode: "advisory" },
{ id: "WS-003", mode: "prompt", enabled: 0, gateMode: "advisory" }, { id: "WS-003", mode: "prompt", enabled: 0, gateMode: "advisory" },
]); ]);
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
db.close(); db.close();
}); });

View File

@@ -330,7 +330,7 @@ describe("Database", () => {
}); });
it("seeds schema version", () => { it("seeds schema version", () => {
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
}); });
it("includes tokenUsageCacheWriteTokens on freshly initialized tasks table", () => { it("includes tokenUsageCacheWriteTokens on freshly initialized tasks table", () => {
@@ -389,7 +389,7 @@ describe("Database", () => {
it("is idempotent - calling init() twice does not fail", () => { it("is idempotent - calling init() twice does not fail", () => {
expect(() => db.init()).not.toThrow(); expect(() => db.init()).not.toThrow();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
}); });
it("does not overwrite existing config on re-init", () => { it("does not overwrite existing config on re-init", () => {
// Update the config // Update the config
@@ -1459,7 +1459,7 @@ describe("schema migrations", () => {
db.init(); db.init();
// Verify version bumped to 29 (includes v1→v2 through v26→v29) // Verify version bumped to 29 (includes v1→v2 through v26→v29)
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
// Verify new columns exist and existing data is intact // Verify new columns exist and existing data is intact
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
@@ -1484,11 +1484,11 @@ describe("schema migrations", () => {
const db = new Database(fusionDir); const db = new Database(fusionDir);
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
// Re-init should not fail // Re-init should not fail
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
db.close(); db.close();
}); });
@@ -1523,7 +1523,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
expect(cols.map((col) => col.name)).toContain("priority"); expect(cols.map((col) => col.name)).toContain("priority");
@@ -1564,7 +1564,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const colNames = cols.map((col) => col.name); const colNames = cols.map((col) => col.name);
@@ -1636,7 +1636,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const colNames = cols.map((col) => col.name); const colNames = cols.map((col) => col.name);
@@ -1876,7 +1876,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
const cols = db.prepare("PRAGMA table_info(chat_messages)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(chat_messages)").all() as Array<{ name: string }>;
expect(cols.map((col) => col.name)).toContain("attachments"); expect(cols.map((col) => col.name)).toContain("attachments");
@@ -1950,7 +1950,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'agentRatings'").all() as Array<{ name: string }>; const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'agentRatings'").all() as Array<{ name: string }>;
expect(tables).toEqual([{ name: "agentRatings" }]); expect(tables).toEqual([{ name: "agentRatings" }]);
@@ -1974,7 +1974,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'mission_events'").all() as Array<{ name: string }>; const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'mission_events'").all() as Array<{ name: string }>;
expect(tables).toEqual([{ name: "mission_events" }]); expect(tables).toEqual([{ name: "mission_events" }]);
@@ -2078,7 +2078,7 @@ describe("schema migrations", () => {
db.init(); db.init();
// Verify version bumped to 29 // Verify version bumped to 29
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
// Verify new columns exist and existing data is intact // Verify new columns exist and existing data is intact
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
@@ -2297,7 +2297,7 @@ describe("schema migrations", () => {
localDb.init(); localDb.init();
expect(localDb.getSchemaVersion()).toBe(98); expect(localDb.getSchemaVersion()).toBe(99);
const columns = localDb.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const columns = localDb.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
expect(columns.map((column) => column.name)).toContain("tokenUsageCacheWriteTokens"); expect(columns.map((column) => column.name)).toContain("tokenUsageCacheWriteTokens");
@@ -2608,7 +2608,7 @@ describe("createDatabase factory", () => {
const db = createDatabase(fusionDir); const db = createDatabase(fusionDir);
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
expect(db.getLastModified()).toBeGreaterThan(0); expect(db.getLastModified()).toBeGreaterThan(0);
db.close(); db.close();
@@ -2762,7 +2762,7 @@ describe("migration v77 task token budget columns", () => {
migrated = new Database(fusion); migrated = new Database(fusion);
migrated.init(); migrated.init();
expect(migrated.getSchemaVersion()).toBe(98); expect(migrated.getSchemaVersion()).toBe(99);
const rows = migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const rows = migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const names = new Set(rows.map((row) => row.name)); const names = new Set(rows.map((row) => row.name));
expect(names.has("tokenBudgetSoftAlertedAt")).toBe(true); expect(names.has("tokenBudgetSoftAlertedAt")).toBe(true);
@@ -2808,7 +2808,7 @@ describe("migration v67 drops orphan project auth tables", () => {
migrated = new Database(fusion); migrated = new Database(fusion);
migrated.init(); migrated.init();
expect(migrated.getSchemaVersion()).toBe(98); expect(migrated.getSchemaVersion()).toBe(99);
const tables = migrated const tables = migrated
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'") .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'")
.all() as Array<{ name: string }>; .all() as Array<{ name: string }>;
@@ -2835,7 +2835,7 @@ describe("migration v67 drops orphan project auth tables", () => {
try { try {
fresh.init(); fresh.init();
expect(fresh.getSchemaVersion()).toBe(98); expect(fresh.getSchemaVersion()).toBe(99);
const tables = fresh const tables = fresh
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'") .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'")
.all() as Array<{ name: string }>; .all() as Array<{ name: string }>;

View File

@@ -91,6 +91,6 @@ describe("goals schema", () => {
}); });
it("reports schema version 92", () => { it("reports schema version 92", () => {
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
}); });
}); });

View File

@@ -1000,7 +1000,7 @@ describe("Migration: pre-33 DB upgrade", () => {
// Step 1: Create a fresh database at v33 (runs all migrations up to 33) // Step 1: Create a fresh database at v33 (runs all migrations up to 33)
const db1 = createDatabase(legacyDir); const db1 = createDatabase(legacyDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(98); expect(db1.getSchemaVersion()).toBe(99);
db1.close(); db1.close();
// Step 2: Manually downgrade to version 32 and drop insight tables // Step 2: Manually downgrade to version 32 and drop insight tables
@@ -1035,7 +1035,7 @@ describe("Migration: pre-33 DB upgrade", () => {
expect(tableNamesBefore).not.toContain("project_insight_runs"); expect(tableNamesBefore).not.toContain("project_insight_runs");
// Now run init — this triggers the v32→v33 migration // Now run init — this triggers the v32→v33 migration
db3.init(); db3.init();
expect(db3.getSchemaVersion()).toBe(98); expect(db3.getSchemaVersion()).toBe(99);
// Step 4: Verify insight tables exist after migration // Step 4: Verify insight tables exist after migration
const tablesAfter = db3.prepare( const tablesAfter = db3.prepare(
@@ -1066,12 +1066,12 @@ describe("Migration: pre-33 DB upgrade", () => {
try { try {
const db1 = createDatabase(testDir); const db1 = createDatabase(testDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(98); expect(db1.getSchemaVersion()).toBe(99);
db1.close(); db1.close();
const db2 = createDatabase(testDir); const db2 = createDatabase(testDir);
expect(() => db2.init()).not.toThrow(); expect(() => db2.init()).not.toThrow();
expect(db2.getSchemaVersion()).toBe(98); expect(db2.getSchemaVersion()).toBe(99);
db2.close(); db2.close();
} finally { } finally {
rmSync(testDir, { recursive: true, force: true }); rmSync(testDir, { recursive: true, force: true });
@@ -1085,7 +1085,7 @@ describe("Migration: pre-33 DB upgrade", () => {
// Step 1: Create a fresh DB and run migrations // Step 1: Create a fresh DB and run migrations
const db1 = createDatabase(compatDir); const db1 = createDatabase(compatDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(98); expect(db1.getSchemaVersion()).toBe(99);
// Step 2: Strip lifecycle and cancelledAt columns by recreating the // Step 2: Strip lifecycle and cancelledAt columns by recreating the
// table without them. This simulates a DB that was created before the // table without them. This simulates a DB that was created before the

View File

@@ -3200,7 +3200,7 @@ describe("MissionStore", () => {
describe("Loop State & Validator Run Schema (v31)", () => { describe("Loop State & Validator Run Schema (v31)", () => {
it("schema version is 40 after migration", () => { it("schema version is 40 after migration", () => {
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
}); });
it("mission_features table has loop state columns", () => { it("mission_features table has loop state columns", () => {

View File

@@ -584,7 +584,7 @@ describe("Run Audit", () => {
}); });
it("schema version is bumped to 40", () => { it("schema version is bumped to 40", () => {
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
}); });
}); });
}); });

View File

@@ -42,7 +42,7 @@ describe("secrets schema migrations", () => {
const version = db const version = db
.prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'") .prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'")
.get() as { value: string }; .get() as { value: string };
expect(version.value).toBe("98"); expect(version.value).toBe("99");
} finally { } finally {
db.close(); db.close();
rmSync(dir, { recursive: true, force: true }); rmSync(dir, { recursive: true, force: true });
@@ -105,7 +105,7 @@ describe("secrets schema migrations", () => {
const version = db const version = db
.prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'") .prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'")
.get() as { value: string }; .get() as { value: string };
expect(version.value).toBe("98"); expect(version.value).toBe("99");
} finally { } finally {
db.close(); db.close();
rmSync(dir, { recursive: true, force: true }); rmSync(dir, { recursive: true, force: true });
@@ -155,7 +155,7 @@ describe("secrets schema migrations", () => {
.prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'") .prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'")
.get() as { value: string }; .get() as { value: string };
expect(projectVersion.value).toBe("98"); expect(projectVersion.value).toBe("99");
expect(centralVersion.value).toBe("13"); expect(centralVersion.value).toBe("13");
} finally { } finally {
projectDb.close(); projectDb.close();

View File

@@ -60,7 +60,7 @@ describe("TaskStore merge queue", () => {
expect.arrayContaining(["idx_mergeQueue_lease_ready", "idx_mergeQueue_leaseExpiresAt"]), expect.arrayContaining(["idx_mergeQueue_lease_ready", "idx_mergeQueue_leaseExpiresAt"]),
); );
expect(store.getDatabase().getSchemaVersion()).toBe(98); expect(store.getDatabase().getSchemaVersion()).toBe(99);
}); });
it("migrates a legacy v88 database and preserves task rows", async () => { it("migrates a legacy v88 database and preserves task rows", async () => {

View File

@@ -51,7 +51,7 @@ describe("TaskStore task documents", () => {
expect(tableNames.has("task_documents")).toBe(true); expect(tableNames.has("task_documents")).toBe(true);
expect(tableNames.has("task_document_revisions")).toBe(true); expect(tableNames.has("task_document_revisions")).toBe(true);
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
const index = db const index = db
.prepare( .prepare(

View File

@@ -149,7 +149,7 @@ export function probeFts5(db: DatabaseSync): boolean {
// ── Schema Definition ──────────────────────────────────────────────── // ── Schema Definition ────────────────────────────────────────────────
const SCHEMA_VERSION = 98; const SCHEMA_VERSION = 99;
function normalizeTaskComments( function normalizeTaskComments(
steeringComments: SteeringComment[] | undefined, steeringComments: SteeringComment[] | undefined,
@@ -242,6 +242,9 @@ CREATE TABLE IF NOT EXISTS tasks (
planningModelId TEXT, planningModelId TEXT,
mergeRetries INTEGER, mergeRetries INTEGER,
workflowStepRetries INTEGER, workflowStepRetries INTEGER,
resumeLimboCount INTEGER DEFAULT 0,
resumeLimboTipSha TEXT,
resumeLimboStepSignature TEXT,
recoveryRetryCount INTEGER, recoveryRetryCount INTEGER,
taskDoneRetryCount INTEGER DEFAULT 0, taskDoneRetryCount INTEGER DEFAULT 0,
worktreeSessionRetryCount INTEGER DEFAULT 0, worktreeSessionRetryCount INTEGER DEFAULT 0,
@@ -3726,6 +3729,14 @@ export class Database {
}); });
} }
if (version < 99) {
this.applyMigration(99, () => {
this.addColumnIfMissing("tasks", "resumeLimboCount", "INTEGER DEFAULT 0");
this.addColumnIfMissing("tasks", "resumeLimboTipSha", "TEXT");
this.addColumnIfMissing("tasks", "resumeLimboStepSignature", "TEXT");
});
}
} }
/** /**

View File

@@ -2,6 +2,7 @@ import type { Task } from "./types.js";
export const MANUAL_RETRY_RESET_COUNTER_KEYS = [ export const MANUAL_RETRY_RESET_COUNTER_KEYS = [
"stuckKillCount", "stuckKillCount",
"resumeLimboCount",
"recoveryRetryCount", "recoveryRetryCount",
"taskDoneRetryCount", "taskDoneRetryCount",
"worktreeSessionRetryCount", "worktreeSessionRetryCount",

View File

@@ -95,6 +95,9 @@ interface TaskRow {
mergeRetries: number | null; mergeRetries: number | null;
workflowStepRetries: number | null; workflowStepRetries: number | null;
stuckKillCount: number | null; stuckKillCount: number | null;
resumeLimboCount: number | null;
resumeLimboTipSha: string | null;
resumeLimboStepSignature: string | null;
postReviewFixCount: number | null; postReviewFixCount: number | null;
recoveryRetryCount: number | null; recoveryRetryCount: number | null;
taskDoneRetryCount: number | null; taskDoneRetryCount: number | null;
@@ -1415,6 +1418,9 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
mergeRetries: row.mergeRetries ?? undefined, mergeRetries: row.mergeRetries ?? undefined,
workflowStepRetries: row.workflowStepRetries ?? undefined, workflowStepRetries: row.workflowStepRetries ?? undefined,
stuckKillCount: row.stuckKillCount ?? undefined, stuckKillCount: row.stuckKillCount ?? undefined,
resumeLimboCount: row.resumeLimboCount ?? undefined,
resumeLimboTipSha: row.resumeLimboTipSha || undefined,
resumeLimboStepSignature: row.resumeLimboStepSignature || undefined,
postReviewFixCount: row.postReviewFixCount ?? undefined, postReviewFixCount: row.postReviewFixCount ?? undefined,
recoveryRetryCount: row.recoveryRetryCount ?? undefined, recoveryRetryCount: row.recoveryRetryCount ?? undefined,
taskDoneRetryCount: row.taskDoneRetryCount ?? undefined, taskDoneRetryCount: row.taskDoneRetryCount ?? undefined,
@@ -1924,7 +1930,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
"modelPresetId", "modelProvider", "modelId", "modelPresetId", "modelProvider", "modelId",
"validatorModelProvider", "validatorModelId", "validatorModelProvider", "validatorModelId",
"planningModelProvider", "planningModelId", "planningModelProvider", "planningModelId",
"mergeRetries", "workflowStepRetries", "stuckKillCount", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
"error", "summary", "thinkingLevel", "executionMode", "error", "summary", "thinkingLevel", "executionMode",
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride", "tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt", "createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt",
@@ -1973,7 +1979,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
"modelPresetId", "modelProvider", "modelId", "modelPresetId", "modelProvider", "modelId",
"validatorModelProvider", "validatorModelId", "validatorModelProvider", "validatorModelId",
"planningModelProvider", "planningModelId", "planningModelProvider", "planningModelId",
"mergeRetries", "workflowStepRetries", "stuckKillCount", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
"error", "summary", "thinkingLevel", "executionMode", "error", "summary", "thinkingLevel", "executionMode",
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride", "tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt", "createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt",
@@ -2039,6 +2045,9 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
task.mergeRetries ?? null, task.mergeRetries ?? null,
task.workflowStepRetries ?? null, task.workflowStepRetries ?? null,
task.stuckKillCount ?? 0, task.stuckKillCount ?? 0,
task.resumeLimboCount ?? 0,
task.resumeLimboTipSha ?? null,
task.resumeLimboStepSignature ?? null,
task.postReviewFixCount ?? 0, task.postReviewFixCount ?? 0,
task.recoveryRetryCount ?? null, task.recoveryRetryCount ?? null,
task.taskDoneRetryCount ?? 0, task.taskDoneRetryCount ?? 0,
@@ -2138,7 +2147,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
id, lineageId, title, description, priority, "column", status, size, reviewLevel, currentStep, id, lineageId, title, description, priority, "column", status, size, reviewLevel, currentStep,
worktree, blockedBy, overlapBlockedBy, paused, userPaused, baseBranch, branch, autoMerge, executionStartBranch, baseCommitSha, modelPresetId, modelProvider, worktree, blockedBy, overlapBlockedBy, paused, userPaused, baseBranch, branch, autoMerge, executionStartBranch, baseCommitSha, modelPresetId, modelProvider,
modelId, validatorModelProvider, validatorModelId, planningModelProvider, planningModelId, mergeRetries, modelId, validatorModelProvider, validatorModelId, planningModelProvider, planningModelId, mergeRetries,
workflowStepRetries, stuckKillCount, postReviewFixCount, recoveryRetryCount, taskDoneRetryCount, worktreeSessionRetryCount, completionHandoffLimboRecoveryCount, verificationFailureCount, mergeConflictBounceCount, mergeAuditBounceCount, mergeTransientRetryCount, branchConflictRecoveryCount, reviewerContextRetryCount, reviewerFallbackRetryCount, nextRecoveryAt, error, workflowStepRetries, stuckKillCount, resumeLimboCount, resumeLimboTipSha, resumeLimboStepSignature, postReviewFixCount, recoveryRetryCount, taskDoneRetryCount, worktreeSessionRetryCount, completionHandoffLimboRecoveryCount, verificationFailureCount, mergeConflictBounceCount, mergeAuditBounceCount, mergeTransientRetryCount, branchConflictRecoveryCount, reviewerContextRetryCount, reviewerFallbackRetryCount, nextRecoveryAt, error,
summary, thinkingLevel, executionMode, tokenUsageInputTokens, tokenUsageOutputTokens, tokenUsageCachedTokens, summary, thinkingLevel, executionMode, tokenUsageInputTokens, tokenUsageOutputTokens, tokenUsageCachedTokens,
tokenUsageCacheWriteTokens, tokenUsageTotalTokens, tokenUsageFirstUsedAt, tokenUsageLastUsedAt, tokenBudgetSoftAlertedAt, tokenBudgetHardAlertedAt, tokenBudgetOverride, createdAt, updatedAt, columnMovedAt, tokenUsageCacheWriteTokens, tokenUsageTotalTokens, tokenUsageFirstUsedAt, tokenUsageLastUsedAt, tokenBudgetSoftAlertedAt, tokenBudgetHardAlertedAt, tokenBudgetOverride, createdAt, updatedAt, columnMovedAt,
firstExecutionAt, cumulativeActiveMs, executionStartedAt, executionCompletedAt, firstExecutionAt, cumulativeActiveMs, executionStartedAt, executionCompletedAt,
@@ -2165,7 +2174,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
id, lineageId, title, description, priority, "column", status, size, reviewLevel, currentStep, id, lineageId, title, description, priority, "column", status, size, reviewLevel, currentStep,
worktree, blockedBy, overlapBlockedBy, paused, userPaused, baseBranch, branch, autoMerge, executionStartBranch, baseCommitSha, modelPresetId, modelProvider, worktree, blockedBy, overlapBlockedBy, paused, userPaused, baseBranch, branch, autoMerge, executionStartBranch, baseCommitSha, modelPresetId, modelProvider,
modelId, validatorModelProvider, validatorModelId, planningModelProvider, planningModelId, mergeRetries, modelId, validatorModelProvider, validatorModelId, planningModelProvider, planningModelId, mergeRetries,
workflowStepRetries, stuckKillCount, postReviewFixCount, recoveryRetryCount, taskDoneRetryCount, worktreeSessionRetryCount, completionHandoffLimboRecoveryCount, verificationFailureCount, mergeConflictBounceCount, mergeAuditBounceCount, mergeTransientRetryCount, branchConflictRecoveryCount, reviewerContextRetryCount, reviewerFallbackRetryCount, nextRecoveryAt, error, workflowStepRetries, stuckKillCount, resumeLimboCount, resumeLimboTipSha, resumeLimboStepSignature, postReviewFixCount, recoveryRetryCount, taskDoneRetryCount, worktreeSessionRetryCount, completionHandoffLimboRecoveryCount, verificationFailureCount, mergeConflictBounceCount, mergeAuditBounceCount, mergeTransientRetryCount, branchConflictRecoveryCount, reviewerContextRetryCount, reviewerFallbackRetryCount, nextRecoveryAt, error,
summary, thinkingLevel, executionMode, tokenUsageInputTokens, tokenUsageOutputTokens, tokenUsageCachedTokens, summary, thinkingLevel, executionMode, tokenUsageInputTokens, tokenUsageOutputTokens, tokenUsageCachedTokens,
tokenUsageCacheWriteTokens, tokenUsageTotalTokens, tokenUsageFirstUsedAt, tokenUsageLastUsedAt, tokenBudgetSoftAlertedAt, tokenBudgetHardAlertedAt, tokenBudgetOverride, createdAt, updatedAt, columnMovedAt, tokenUsageCacheWriteTokens, tokenUsageTotalTokens, tokenUsageFirstUsedAt, tokenUsageLastUsedAt, tokenBudgetSoftAlertedAt, tokenBudgetHardAlertedAt, tokenBudgetOverride, createdAt, updatedAt, columnMovedAt,
firstExecutionAt, cumulativeActiveMs, executionStartedAt, executionCompletedAt, firstExecutionAt, cumulativeActiveMs, executionStartedAt, executionCompletedAt,
@@ -2204,6 +2213,9 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
mergeRetries = excluded.mergeRetries, mergeRetries = excluded.mergeRetries,
workflowStepRetries = excluded.workflowStepRetries, workflowStepRetries = excluded.workflowStepRetries,
stuckKillCount = excluded.stuckKillCount, stuckKillCount = excluded.stuckKillCount,
resumeLimboCount = excluded.resumeLimboCount,
resumeLimboTipSha = excluded.resumeLimboTipSha,
resumeLimboStepSignature = excluded.resumeLimboStepSignature,
postReviewFixCount = excluded.postReviewFixCount, postReviewFixCount = excluded.postReviewFixCount,
recoveryRetryCount = excluded.recoveryRetryCount, recoveryRetryCount = excluded.recoveryRetryCount,
taskDoneRetryCount = excluded.taskDoneRetryCount, taskDoneRetryCount = excluded.taskDoneRetryCount,
@@ -5548,7 +5560,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
async updateTask( async updateTask(
id: string, id: string,
updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record<string, unknown> | null; githubTracking?: import("./types.js").TaskGithubTracking | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null }, updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record<string, unknown> | null; githubTracking?: import("./types.js").TaskGithubTracking | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null },
runContext?: RunMutationContext, runContext?: RunMutationContext,
): Promise<Task> { ): Promise<Task> {
return this.withTaskLock(id, async () => { return this.withTaskLock(id, async () => {
@@ -5857,6 +5869,21 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
} else if (updates.stuckKillCount !== undefined) { } else if (updates.stuckKillCount !== undefined) {
task.stuckKillCount = updates.stuckKillCount; task.stuckKillCount = updates.stuckKillCount;
} }
if (updates.resumeLimboCount === null) {
task.resumeLimboCount = undefined;
} else if (updates.resumeLimboCount !== undefined) {
task.resumeLimboCount = updates.resumeLimboCount;
}
if (updates.resumeLimboTipSha === null) {
task.resumeLimboTipSha = undefined;
} else if (updates.resumeLimboTipSha !== undefined) {
task.resumeLimboTipSha = updates.resumeLimboTipSha;
}
if (updates.resumeLimboStepSignature === null) {
task.resumeLimboStepSignature = undefined;
} else if (updates.resumeLimboStepSignature !== undefined) {
task.resumeLimboStepSignature = updates.resumeLimboStepSignature;
}
if (updates.postReviewFixCount === null) { if (updates.postReviewFixCount === null) {
task.postReviewFixCount = undefined; task.postReviewFixCount = undefined;
} else if (updates.postReviewFixCount !== undefined) { } else if (updates.postReviewFixCount !== undefined) {

View File

@@ -1887,6 +1887,17 @@ export interface Task {
* Incremented by the self-healing manager on each stuck kill. When this reaches * Incremented by the self-healing manager on each stuck kill. When this reaches
* `maxStuckKills`, the task is marked as permanently failed instead of re-queued. */ * `maxStuckKills`, the task is marked as permanently failed instead of re-queued. */
stuckKillCount?: number; stuckKillCount?: number;
/** Number of consecutive reclaim/unpause attempts where no execution progress
* materialized (tip unchanged, step signature unchanged, and no active session).
* Incremented by self-healing for resume-limbo detection and reset when
* progress is observed or recovery escalates to a fresh todo dispatch. */
resumeLimboCount?: number;
/** Branch tip SHA snapshot captured at the last reclaim/unpause attempt used
* by resume-limbo detection to determine whether commits advanced. */
resumeLimboTipSha?: string;
/** Compact execution-progress snapshot captured at the last reclaim/unpause
* attempt (current step + step statuses) for resume-limbo detection. */
resumeLimboStepSignature?: string;
/** Number of times the self-healing manager has auto-revived this task from /** Number of times the self-healing manager has auto-revived this task from
* `in-review` after a failed pre-merge workflow step. Incremented each time the * `in-review` after a failed pre-merge workflow step. Incremented each time the
* `recoverReviewTasksWithFailedPreMergeSteps` scan sends the task back with the * `recoverReviewTasksWithFailedPreMergeSteps` scan sends the task back with the

View File

@@ -0,0 +1,168 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { EventEmitter } from "node:events";
import type { Settings, Task, TaskStore } from "@fusion/core";
import { SelfHealingManager } from "../../self-healing.js";
import * as branchConflictModule from "../../branch-conflicts.js";
import * as worktreePoolModule from "../../worktree-pool.js";
type MutableSettings = Settings & {
autoMerge?: boolean;
globalPause?: boolean;
enginePaused?: boolean;
};
function makeTask(overrides: Partial<Task> = {}): Task {
return {
id: "FN-5704",
title: "resume limbo",
description: "test",
column: "in-progress",
branch: "fusion/fn-5704",
worktree: "/tmp/test/.worktrees/fn-5704",
paused: false,
userPaused: false,
checkedOutBy: undefined,
dependencies: [],
steps: [{ id: "s1", title: "step", status: "in-progress" } as any],
currentStep: 1,
log: [],
createdAt: new Date().toISOString(),
updatedAt: new Date("2026-01-01T00:00:00.000Z").toISOString(),
executionStartedAt: new Date("2026-01-01T00:00:00.000Z").toISOString(),
...overrides,
} as Task;
}
function makeStore(task: Task, settingsOverrides: Partial<MutableSettings> = {}): TaskStore & EventEmitter {
const emitter = new EventEmitter();
const settings = {
autoMerge: true,
globalPause: false,
enginePaused: false,
baseBranch: "main",
mergeStrategy: "direct",
autoRecovery: { mode: "deterministic-only", maxRetries: 3 },
...settingsOverrides,
} as unknown as Settings;
return Object.assign(emitter, {
getSettings: vi.fn(async () => settings),
listTasks: vi.fn(async ({ column }: { column?: string } = {}) => (column === task.column ? [task] : [])),
updateTask: vi.fn(async (_id: string, updates: Partial<Task>) => Object.assign(task, updates)),
moveTask: vi.fn(async (_id: string, column: Task["column"], opts?: Record<string, unknown>) => {
task.column = column;
(task as any).__lastMoveOpts = opts;
return task;
}),
logEntry: vi.fn(async () => undefined),
recordRunAuditEvent: vi.fn(async () => undefined),
appendAgentLog: vi.fn(async () => undefined),
updateSettings: vi.fn(async () => settings),
clearStaleExecutionStartBranchReferences: vi.fn(() => []),
walCheckpoint: vi.fn(() => ({ busy: 0, log: 0, checkpointed: 0 })),
archiveTaskAndCleanup: vi.fn(async () => ({})),
mergeTask: vi.fn(async () => undefined),
getRootDir: vi.fn(() => "/tmp/test"),
}) as unknown as TaskStore & EventEmitter;
}
describe("FN-5704: reclaim self-owned resume limbo escalation", () => {
beforeEach(() => {
vi.restoreAllMocks();
vi.spyOn(worktreePoolModule, "isUsableTaskWorktree").mockResolvedValue(true);
});
it("escalates frozen in-progress reclaim/resume loops to todo with preserve flags and audit event", async () => {
vi.useFakeTimers();
vi.setSystemTime(new Date("2026-01-01T02:00:00.000Z"));
const task = makeTask();
const store = makeStore(task);
vi.spyOn(branchConflictModule, "inspectBranchConflict").mockResolvedValue({
kind: "reclaimable",
taskAttributedCommitCount: 1,
strandedCommits: [{ sha: "c1", authorName: "a", subject: "s", timestamp: Date.now() }],
livePath: task.worktree,
tipSha: "abc123abc123abc123abc123abc123abc123abcd",
} as any);
const manager = new SelfHealingManager(store as any, { rootDir: "/tmp/test" } as any);
await manager.reclaimSelfOwnedBranchConflicts();
expect(task.resumeLimboCount).toBe(0);
expect((store.moveTask as any).mock.calls.length).toBe(0);
await manager.reclaimSelfOwnedBranchConflicts();
expect(task.resumeLimboCount).toBe(1);
await manager.reclaimSelfOwnedBranchConflicts();
expect(store.moveTask).toHaveBeenCalledWith(task.id, "todo", expect.objectContaining({
moveSource: "engine",
preserveWorktree: true,
preserveProgress: true,
preserveResumeState: true,
}));
expect(task.resumeLimboCount).toBe(0);
const limboEvent = (store.recordRunAuditEvent as any).mock.calls.find((call: any[]) => call[0].mutationType === "task:resume-limbo-escalated")?.[0];
expect(limboEvent).toBeTruthy();
expect(limboEvent.target).toBe(task.id);
expect(limboEvent.metadata).toEqual(expect.objectContaining({
taskId: task.id,
frozenTipSha: "abc123abc123abc123abc123abc123abc123abcd",
resumeAttemptCount: 2,
currentStep: 1,
}));
const auditMetadata = limboEvent.metadata;
expect(auditMetadata.idleMs).toBeGreaterThan(0);
vi.useRealTimers();
manager.stop();
});
it("resets limbo counter on progress and avoids escalation", async () => {
const task = makeTask();
const store = makeStore(task);
const inspect = vi.spyOn(branchConflictModule, "inspectBranchConflict");
inspect.mockResolvedValueOnce({ kind: "reclaimable", taskAttributedCommitCount: 1, strandedCommits: [{ sha: "c1", authorName: "a", subject: "s", timestamp: Date.now() }], livePath: task.worktree, tipSha: "sha-1" } as any);
inspect.mockResolvedValueOnce({ kind: "reclaimable", taskAttributedCommitCount: 1, strandedCommits: [{ sha: "c1", authorName: "a", subject: "s", timestamp: Date.now() }], livePath: task.worktree, tipSha: "sha-1" } as any);
inspect.mockResolvedValueOnce({ kind: "reclaimable", taskAttributedCommitCount: 1, strandedCommits: [{ sha: "c2", authorName: "a", subject: "s", timestamp: Date.now() }], livePath: task.worktree, tipSha: "sha-2" } as any);
inspect.mockResolvedValueOnce({ kind: "reclaimable", taskAttributedCommitCount: 1, strandedCommits: [{ sha: "c2", authorName: "a", subject: "s", timestamp: Date.now() }], livePath: task.worktree, tipSha: "sha-2" } as any);
const manager = new SelfHealingManager(store as any, { rootDir: "/tmp/test" } as any);
await manager.reclaimSelfOwnedBranchConflicts();
await manager.reclaimSelfOwnedBranchConflicts();
expect(task.resumeLimboCount).toBe(1);
await manager.reclaimSelfOwnedBranchConflicts();
expect(task.resumeLimboCount).toBe(0);
await manager.reclaimSelfOwnedBranchConflicts();
expect(task.resumeLimboCount).toBe(1);
expect(store.moveTask).not.toHaveBeenCalled();
manager.stop();
});
it("never escalates user-paused tasks", async () => {
const task = makeTask({ userPaused: true });
const store = makeStore(task);
const inspectSpy = vi.spyOn(branchConflictModule, "inspectBranchConflict");
const manager = new SelfHealingManager(store as any, { rootDir: "/tmp/test" } as any);
await manager.reclaimSelfOwnedBranchConflicts();
expect(inspectSpy).not.toHaveBeenCalled();
expect(store.moveTask).not.toHaveBeenCalled();
manager.stop();
});
it("short-circuits reclaim when autoMerge is false", async () => {
const task = makeTask();
const store = makeStore(task, { autoMerge: false });
const inspectSpy = vi.spyOn(branchConflictModule, "inspectBranchConflict");
const manager = new SelfHealingManager(store as any, { rootDir: "/tmp/test" } as any);
const recovered = await manager.reclaimSelfOwnedBranchConflicts();
expect(recovered).toBe(0);
expect(inspectSpy).not.toHaveBeenCalled();
expect(store.moveTask).not.toHaveBeenCalled();
manager.stop();
});
});

View File

@@ -1960,6 +1960,11 @@ export class TaskExecutor {
executorLog.log(`Unpaused ${task.id} in-progress with no session — resuming execution`); executorLog.log(`Unpaused ${task.id} in-progress with no session — resuming execution`);
try { try {
await this.clearResumeFailureState(task); await this.clearResumeFailureState(task);
await this.store.updateTask(task.id, {
resumeLimboCount: 0,
resumeLimboTipSha: null,
resumeLimboStepSignature: null,
});
await this.store.logEntry(task.id, "Resuming execution after unpause", undefined, this.getRunContextFor(task.id)); await this.store.logEntry(task.id, "Resuming execution after unpause", undefined, this.getRunContextFor(task.id));
await this.recoverApprovedStepsOnResume(task.id); await this.recoverApprovedStepsOnResume(task.id);
} catch (clearErr) { } catch (clearErr) {

View File

@@ -462,6 +462,7 @@ export type DatabaseMutationType =
| "task:auto-recover-completion-handoff-limbo-exhausted" | "task:auto-recover-completion-handoff-limbo-exhausted"
| "task:auto-recover-worktree-session-exhausted" | "task:auto-recover-worktree-session-exhausted"
| "task:auto-recover-in-progress-limbo" | "task:auto-recover-in-progress-limbo"
| "task:resume-limbo-escalated"
| "task:orphan-detected-no-action" | "task:orphan-detected-no-action"
/** Metadata: { taskId: string; ignoredStepUpdateCount: number; stuckKillStreak: number; lastReason: "no-progress-churn" } */ /** Metadata: { taskId: string; ignoredStepUpdateCount: number; stuckKillStreak: number; lastReason: "no-progress-churn" } */
| "task:stuck-no-progress-churn-terminalized" | "task:stuck-no-progress-churn-terminalized"

View File

@@ -72,6 +72,7 @@ const DB_CORRUPTION_NOTIFICATION_COOLDOWN_MS = 60 * 60 * 1000;
export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000; export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000;
export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000; export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000;
export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3; export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3;
const MAX_NO_PROGRESS_RESUME_ATTEMPTS = 2;
// listTasks already enforces ACTIVE_TASKS_WHERE (`"deletedAt" IS NULL`), but // listTasks already enforces ACTIVE_TASKS_WHERE (`"deletedAt" IS NULL`), but
// deadlock/stall sweeps still defensively skip soft-deleted rows in case a // deadlock/stall sweeps still defensively skip soft-deleted rows in case a
@@ -106,6 +107,13 @@ async function classifyOwnedLandedEvidenceForSelfHealing(rootDir: string, task:
return classifyOwnedLandedEvidence(rootDir, task, { mergeTargetBranch }); return classifyOwnedLandedEvidence(rootDir, task, { mergeTargetBranch });
} }
function buildResumeLimboStepSignature(task: Task): string {
return JSON.stringify({
currentStep: task.currentStep ?? null,
steps: Array.isArray(task.steps) ? task.steps.map((step) => step.status) : [],
});
}
function formatRecoveryTimestamp(date = new Date()): string { function formatRecoveryTimestamp(date = new Date()): string {
const pad = (value: number) => String(value).padStart(2, "0"); const pad = (value: number) => String(value).padStart(2, "0");
return `${date.getUTCFullYear()}${pad(date.getUTCMonth() + 1)}${pad(date.getUTCDate())}-${pad(date.getUTCHours())}${pad(date.getUTCMinutes())}${pad(date.getUTCSeconds())}`; return `${date.getUTCFullYear()}${pad(date.getUTCMonth() + 1)}${pad(date.getUTCDate())}-${pad(date.getUTCHours())}${pad(date.getUTCMinutes())}${pad(date.getUTCSeconds())}`;
@@ -2328,6 +2336,67 @@ export class SelfHealingManager {
const preservedCommitCount = inspection.kind === "fully-subsumed" const preservedCommitCount = inspection.kind === "fully-subsumed"
? 0 ? 0
: inspection.taskAttributedCommitCount; : inspection.taskAttributedCommitCount;
const stepSignature = buildResumeLimboStepSignature(task);
const hasActiveSessionSignal = Boolean(task.checkedOutBy) || activeTaskIds.has(task.id.toUpperCase());
const hasPriorSnapshot = typeof task.resumeLimboTipSha === "string" && typeof task.resumeLimboStepSignature === "string";
const unchangedSincePriorResume = hasPriorSnapshot
&& task.resumeLimboTipSha === inspection.tipSha
&& task.resumeLimboStepSignature === stepSignature;
const isNoProgressResume = task.column === "in-progress"
&& unchangedSincePriorResume
&& !hasActiveSessionSignal;
const resumeAttemptCount = isNoProgressResume ? (task.resumeLimboCount ?? 0) + 1 : 0;
if (task.column === "in-progress" && isNoProgressResume && resumeAttemptCount >= MAX_NO_PROGRESS_RESUME_ATTEMPTS) {
const idleAnchor = task.executionStartedAt ?? task.columnMovedAt ?? task.updatedAt;
const idleAnchorMs = Date.parse(idleAnchor ?? "");
const idleMs = Number.isFinite(idleAnchorMs) ? Math.max(0, Date.now() - idleAnchorMs) : null;
await this.store.moveTask(task.id, "todo", {
moveSource: "engine",
preserveWorktree: true,
preserveProgress: true,
preserveResumeState: true,
});
await this.store.updateTask(task.id, {
resumeLimboCount: 0,
resumeLimboTipSha: inspection.tipSha,
resumeLimboStepSignature: stepSignature,
});
await this.store.logEntry(
task.id,
`[recovery] resume-limbo-escalated ${task.id} moved to todo after ${resumeAttemptCount} no-progress reclaim/resume attempts`,
JSON.stringify({
frozenTipSha: inspection.tipSha,
idleMs,
resumeAttemptCount,
currentStep: task.currentStep ?? null,
}),
);
try {
await createRunAuditor(this.store, {
runId: generateSyntheticRunId("self-heal", task.id),
agentId: "self-healing",
taskId: task.id,
taskLineageId: task.lineageId,
phase: "reclaim-self-owned-branch-conflicts",
}).database({
type: "task:resume-limbo-escalated",
target: task.id,
metadata: {
taskId: task.id,
frozenTipSha: inspection.tipSha,
idleMs,
resumeAttemptCount,
currentStep: task.currentStep ?? null,
},
});
} catch (auditErr: unknown) {
log.warn(`Failed to write task:resume-limbo-escalated run-audit event for ${task.id}: ${auditErr instanceof Error ? auditErr.message : String(auditErr)}`);
}
recovered++;
continue;
}
await this.store.updateTask(task.id, { await this.store.updateTask(task.id, {
worktree: inspection.livePath, worktree: inspection.livePath,
branch: task.branch, branch: task.branch,
@@ -2335,6 +2404,9 @@ export class SelfHealingManager {
pausedReason: undefined, pausedReason: undefined,
status: null, status: null,
error: null, error: null,
resumeLimboCount: resumeAttemptCount,
resumeLimboTipSha: inspection.tipSha,
resumeLimboStepSignature: stepSignature,
}); });
await this.store.logEntry( await this.store.logEntry(
task.id, task.id,

View File

@@ -743,8 +743,8 @@ describe("RoadmapStore", () => {
}); });
describe("schema version", () => { describe("schema version", () => {
it("schema version is 98 after init", () => { it("schema version is 99 after init", () => {
expect(db.getSchemaVersion()).toBe(98); expect(db.getSchemaVersion()).toBe(99);
}); });
}); });