FN-5704: escalate self-owned execution limbo instead of endless resume
Escalate reclaimable in-progress resume limbo into an explicit failure path to stop infinite self-healing loops. - add resume-limbo escalation handling in self-healing/executor flow so self-owned stuck execution is failed and surfaced - extend core task/run-audit types and retry-reset/store behavior to persist and expose the new escalation state - add reliability interaction coverage for reclaim self-owned resume limbo escalation and update related schema/store/CLI/plugin tests and docs Files changed: AGENTS.md | 1 + docs/architecture.md | 2 + packages/cli/src/commands/__tests__/task.test.ts | 2 + packages/core/src/__tests__/db-migrate.test.ts | 12 +- packages/core/src/__tests__/db.test.ts | 34 ++--- packages/core/src/__tests__/goals-schema.test.ts | 2 +- packages/core/src/__tests__/insight-store.test.ts | 10 +- packages/core/src/__tests__/mission-store.test.ts | 2 +- packages/core/src/__tests__/run-audit.test.ts | 2 +- packages/core/src/__tests__/secrets-schema.test.ts | 6 +- .../core/src/__tests__/store-merge-queue.test.ts | 2 +- packages/core/src/__tests__/task-documents.test.ts | 2 +- packages/core/src/db.ts | 13 +- packages/core/src/manual-retry-reset.ts | 1 + packages/core/src/store.ts | 37 ++++- packages/core/src/types.ts | 11 ++ ...laim-self-owned-resume-limbo-escalation.test.ts | 168 +++++++++++++++++++++ packages/engine/src/executor.ts | 5 + packages/engine/src/run-audit.ts | 1 + packages/engine/src/self-healing.ts | 72 +++++++++ .../src/store/__tests__/roadmap-store.test.ts | 4 +- 21 files changed, 345 insertions(+), 44 deletions(-) Fusion-Task-Id: FN-5704 Fusion-Task-Lineage: c8f73d16-d95b-450f-a514-e7d3c2f7aebe
This commit is contained in:
@@ -158,6 +158,7 @@ When `settings.autoMerge: false`, `in-review` is terminal-until-merged by a huma
|
||||
|
||||
- FN-5432 backstop: `packages/engine/src/__tests__/reliability-interactions/dependency-cycle-reconcile.test.ts` extends FN-5256 coverage with long-cycle ambiguous sweep, write-boundary/sweep race, self-defeating+cycle non-contradiction across one maintenance flow, and audit-event shape regression; core regression cases (long cycle, self-loop via update, incremental-update closes a loop, moveTask seam invariant, DependencyCycleError shape) live in `packages/core/src/__tests__/store-dependency-cycle.test.ts`. User-facing pull/stash audit event behavior (`pull:fast-forward`, `stash:pop-conflict`) is documented in `docs/dashboard-guide.md` under Merge Advance Notice / Smart Pull.
|
||||
- FN-5403 backstop: `packages/engine/src/__tests__/reliability-interactions/engine-stop-aborts-execution.test.ts` locks stop-ordering behavior so engine shutdown aborts executor AI sessions before drain wait and preserves task-row lifecycle semantics.
|
||||
- FN-5704 backstop: `packages/engine/src/__tests__/reliability-interactions/reclaim-self-owned-resume-limbo-escalation.test.ts` guards reclaim/unpause no-progress oscillation recovery by capping repeated no-progress resumes, escalating to preserve-work `todo` rebound, and emitting `task:resume-limbo-escalated` audit metadata while exempting progress/user-paused/autoMerge-off cases.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -1738,6 +1738,7 @@ This section preserves the detailed lifecycle/self-healing contracts that were f
|
||||
- **In-review is terminal-until-merged under `autoMerge: false` (FN-5147)**: when a project sets `settings.autoMerge: false`, `in-review` is the intended resting state until a human merges the PR. No lifecycle-mutating self-healing sweep (`reclaimSelfOwnedBranchConflicts`, `recoverGhostReviewTasks`, `recoverStaleIncompleteReviewTasks`, `recoverInterruptedMergingTasks`, `recoverStuckMergeDeadlocks`, `recoverMissingWorktreeReviewFailures`, `recoverPartialProgressNoTaskDoneFailures`, `recoverCompletionHandoffLimbo`, `recoverMergeableReviewTasks`, `recoverMergedReviewTasks`, `recoverAlreadyMergedReviewTasks`, `recoverOrphanOnlyScopeViolations`, `recoverForeignOnlyContaminatedInReviewTasks`, `recoverReviewTasksWithFailedPreMergeSteps`, `finalizeNoOpReviewTasks`, `surfaceInReviewStalls`, `surfaceInReviewStalled`) may move the task out of `in-review`, mark it `paused`/`failed`, or re-enqueue it for execution. RECONCILE-ONLY sweeps (branch rebind, blocker fan-out, stale-status clears, contamination metadata cleanup, attribution restore, PR refresh, misclassified-failure error clearing) continue to run.
|
||||
- **Auto-merge integration-root default (FN-5279)**: direct auto-merge now defaults `mergeIntegrationWorktree` to `reuse-task-worktree`; merger must pass the reuse handoff gates or emit `merge:reuse-handoff-refused` and leave the task in `in-review` without silently falling back to `cwd-integration-branch` (`cwd-main` remains a deprecated alias normalized to that mode).
|
||||
- **Orphaned execution sweep is observation-only (FN-5337)**: `recoverOrphanedExecutions` only annotates stale in-progress candidates with `task:orphan-detected-no-action` and `[orphan-detected] ... no action (operator-decides)` logs. It must never move `in-progress`/`in-review` backward to `todo` or mutate lease/worktree metadata. Proof-based backward recovery remains exclusively in `recoverInProgressLimbo` (FN-5219), `RestartRecoveryCoordinator`, `recoverMissingWorktreeReviewFailures`, and explicit executor/merger failure paths. Reintroducing lifecycle mutation here requires hard git/session proof gating plus CEO+CTO+PM sign-off.
|
||||
- **Self-owned reclaim resume-limbo escalation (FN-5704)**: `reclaimSelfOwnedBranchConflicts` tracks `resumeLimboCount`, `resumeLimboTipSha`, and `resumeLimboStepSignature` for in-progress reclaim/unpause loops. If reclaim finds no progress (same tip, same step-status signature, and no active-session signal) for `MAX_NO_PROGRESS_RESUME_ATTEMPTS` consecutive sweeps, self-healing escalates by moving the task to `todo` with `preserveWorktree: true`, `preserveProgress: true`, and `preserveResumeState: true` instead of endlessly re-arming resume. Escalation emits `task:resume-limbo-escalated` run-audit metadata (`frozenTipSha`, `idleMs`, `resumeAttemptCount`, `currentStep`) and resets the limbo counter.
|
||||
- **No-progress churn terminalization (FN-5168)**: `StuckTaskDetector` now tracks ignored `fn_task_update` rebuffs via `recordIgnoredStepUpdate(taskId)` and, after one loop/compact-and-resume recovery has already fired in the same `execute()` lifecycle, escalates `ignoredStepUpdateCount >= 25` to the terminal reason `no-progress-churn`. `SelfHealingManager.checkStuckBudget()` maps that reason directly to `STUCK_NO_PROGRESS_CHURN`, emits `task:stuck-no-progress-churn-terminalized` with `{ taskId, ignoredStepUpdateCount, stuckKillStreak, lastReason }`, and parks the task in `in-review` without consuming the normal stuck-kill budget. Under FN-5147 `autoMerge: false`, that failed in-review task remains terminal-until-merged just like `STUCK_LOOP_EXHAUSTED`; the new class adds an earlier bounded exit, not a re-execution path.
|
||||
- **Landed-files attribution (FN-5103)**: Rebase-strategy `mergeDetails.landedFiles` / `filesChanged` / `insertions` / `deletions` are captured from task-attributable commits only via `filterFilesToOwnTaskCommits` (subject-prefix + trailer + bracket-prefix evidence), tagged `landedFilesAttributionRestricted: true`. Zero own commits → `landedFiles: []` and `noOpVerifiedShortCircuit: true`. FN-5304 guard: when `<rebaseBaseSha>..HEAD` reports zero own commits, merger must also validate the source `fusion/<id>` tip; if that source tip still has attributable own commits relative to `rebaseBaseSha`, throw `SilentNoOpAttributionMismatchError`, refuse writing `mergeConfirmed: true`, park the task in `in-review` with `status: "failed"`, and emit `merge:no-op-attribution-mismatch`. If source ref is unavailable, skip with diagnostic + `merge:no-op-attribution-mismatch-skipped` (`reason: "source-ref-unavailable"`). Attribution-helper failures fall back to the unrestricted `rebaseBaseSha..sha` walk and set `landedFilesCaptureFallback: 'attribution-failed'`. Self-healing `recoverDoneTaskMergeMetadata` skips reconcile when `landedFilesAttributionRestricted` or `noOpVerifiedShortCircuit` is set so the narrower set is not overwritten with the full range. Squash-strategy capture is unchanged.
|
||||
- **Soft-delete scheduler invalidation (FN-5137)**: `task:deleted` events must invalidate `AutoClaimSnapshotManager` and clear scheduler bookkeeping (`pausedTaskIds`, `failedTaskIds`, `wasNodeDispatchValidationBlocked`, `wasNodeBlocked`); `executor.execute()` / `resumeOrphaned()` / `resumeTaskForAgent()` refuse any task with `deletedAt` set.
|
||||
@@ -1764,6 +1765,7 @@ Reliability-layer changes are in scope. Interaction regression backstops live in
|
||||
- FN-5147 backstop: `packages/engine/src/__tests__/reliability-interactions/in-review-automerge-off.test.ts` covers `autoMerge: false` + long-quiet in-review + maintenance/startup sweep cycles, asserting no column move / no paused / no status mutation / no requeue, plus explicit regression guards for `surfaceInReviewStalls` and `surfaceInReviewStalled`.
|
||||
- FN-5168 backstop: `packages/engine/src/__tests__/reliability-interactions/non-progress-churn.test.ts` covers loop→compact recovery followed by ignored-step-update churn escalation, terminal `beforeRequeue(false)` behavior, audit/log payloads, and FN-5147 autoMerge-off composition.
|
||||
- FN-5219 backstop: `packages/engine/src/__tests__/reliability-interactions/in-progress-limbo-recovery.test.ts` covers `recoverInProgressLimbo` composition with `recoverOrphanedExecutions` (no double-recovery), `reconcile-task-worktree-metadata` (live rebindable worktree wins), `recoverMissingWorktreeReviewFailures` (in-review vs in-progress disjoint), and executor task-id claim skip, plus an explicit FN-5149 reproduction case.
|
||||
- FN-5704 backstop: `packages/engine/src/__tests__/reliability-interactions/reclaim-self-owned-resume-limbo-escalation.test.ts` covers bounded no-progress reclaim/resume detection, preserve-work escalation to `todo`, `task:resume-limbo-escalated` audit metadata, progress-signal reset behavior, and user-paused/autoMerge-off non-escalation guards.
|
||||
- FN-5337 backstop: `packages/engine/src/__tests__/reliability-interactions/orphan-detected-no-requeue.test.ts` locks observation-only orphan detection across FN-5279 repro metadata desync, worktree-present and worktree-missing candidates, FN-5219 ordering, FN-5147 in-review isolation, FN-5083 branch-cleared composition, lease-manager non-invocation, and per-sweep idempotent audit emission.
|
||||
- FN-5256 backstop: `packages/engine/src/__tests__/reliability-interactions/dependency-cycle-reconcile.test.ts` covers persisted dependency-cycle detection via `reconcileDependencyCycles`, bounded umbrella-back-edge auto-repair, ambiguous-cycle observe-only behavior, composition ordering with `reconcileSelfDefeatingDependencies`, and the post-sweep write-time guard invariant. Core write-boundary regressions (FN-5240/5241/5242 signature, indirect cycle, umbrella back-edge rejection) live in `packages/core/src/__tests__/store-dependency-cycle.test.ts`.
|
||||
- FN-5325 backstop: `packages/engine/src/__tests__/reliability-interactions/scheduler-overlap-priority-inversion.test.ts` covers queued-overlap priority/age deferral, equal-priority age ordering, FN-4969 fanout composition, and one-shot per-pass `scheduler:overlap-priority-inversion` audit surfacing against running lower-priority blockers.
|
||||
|
||||
@@ -2428,6 +2428,7 @@ describe("runTaskRetry", () => {
|
||||
completionHandoffLimboRecoveryCount: 0,
|
||||
mergeAuditBounceCount: 0,
|
||||
mergeRetries: 0,
|
||||
resumeLimboCount: 0,
|
||||
});
|
||||
expect(mockMoveTask).toHaveBeenCalledWith("FN-001", "todo");
|
||||
expect(mockLogEntry).toHaveBeenCalledWith("FN-001", "Retry requested from CLI", "Task reset to todo for retry");
|
||||
@@ -2501,6 +2502,7 @@ describe("runTaskRetry", () => {
|
||||
completionHandoffLimboRecoveryCount: 0,
|
||||
mergeAuditBounceCount: 0,
|
||||
mergeRetries: 0,
|
||||
resumeLimboCount: 0,
|
||||
});
|
||||
expect(mockMoveTask).toHaveBeenCalledWith("FN-001", "todo");
|
||||
expect(mockLogEntry).toHaveBeenCalledWith("FN-001", "Retry requested from CLI", "Task reset to todo for retry");
|
||||
|
||||
@@ -715,7 +715,7 @@ describe("schema migration", () => {
|
||||
|
||||
const row = db.prepare("SELECT deletedAt FROM tasks WHERE id = 'FN-legacy'").get() as { deletedAt: string | null };
|
||||
expect(row.deletedAt).toBeNull();
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
db.close();
|
||||
});
|
||||
@@ -748,7 +748,7 @@ describe("schema migration", () => {
|
||||
{ id: "WS-001", mode: "prompt", gateMode: "advisory" },
|
||||
{ id: "WS-002", mode: "script", gateMode: "advisory" },
|
||||
]);
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
db.close();
|
||||
});
|
||||
@@ -798,7 +798,7 @@ describe("schema migration", () => {
|
||||
reviewerContextRetryCount: 0,
|
||||
reviewerFallbackRetryCount: 0,
|
||||
});
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
db.close();
|
||||
});
|
||||
@@ -827,7 +827,7 @@ describe("schema migration", () => {
|
||||
|
||||
const columns = db.prepare("PRAGMA table_info(milestones)").all() as Array<{ name: string }>;
|
||||
expect(columns.map((column) => column.name)).toContain("acceptanceCriteria");
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
db.close();
|
||||
});
|
||||
@@ -868,7 +868,7 @@ describe("schema migration", () => {
|
||||
const missionColumns = db.prepare("PRAGMA table_info(missions)").all() as Array<{ name: string }>;
|
||||
expect(missionColumns.map((column) => column.name)).toContain("autoMerge");
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
db.close();
|
||||
});
|
||||
|
||||
@@ -902,7 +902,7 @@ describe("schema migration", () => {
|
||||
{ id: "WS-002", mode: "script", enabled: 1, gateMode: "advisory" },
|
||||
{ id: "WS-003", mode: "prompt", enabled: 0, gateMode: "advisory" },
|
||||
]);
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
db.close();
|
||||
});
|
||||
|
||||
@@ -330,7 +330,7 @@ describe("Database", () => {
|
||||
});
|
||||
|
||||
it("seeds schema version", () => {
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
});
|
||||
|
||||
it("includes tokenUsageCacheWriteTokens on freshly initialized tasks table", () => {
|
||||
@@ -389,7 +389,7 @@ describe("Database", () => {
|
||||
|
||||
it("is idempotent - calling init() twice does not fail", () => {
|
||||
expect(() => db.init()).not.toThrow();
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
});
|
||||
it("does not overwrite existing config on re-init", () => {
|
||||
// Update the config
|
||||
@@ -1459,7 +1459,7 @@ describe("schema migrations", () => {
|
||||
db.init();
|
||||
|
||||
// Verify version bumped to 29 (includes v1→v2 through v26→v29)
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
// Verify new columns exist and existing data is intact
|
||||
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
|
||||
@@ -1484,11 +1484,11 @@ describe("schema migrations", () => {
|
||||
const db = new Database(fusionDir);
|
||||
db.init();
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
// Re-init should not fail
|
||||
db.init();
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
db.close();
|
||||
});
|
||||
@@ -1523,7 +1523,7 @@ describe("schema migrations", () => {
|
||||
|
||||
db.init();
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
|
||||
expect(cols.map((col) => col.name)).toContain("priority");
|
||||
@@ -1564,7 +1564,7 @@ describe("schema migrations", () => {
|
||||
|
||||
db.init();
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
|
||||
const colNames = cols.map((col) => col.name);
|
||||
@@ -1636,7 +1636,7 @@ describe("schema migrations", () => {
|
||||
|
||||
db.init();
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
|
||||
const colNames = cols.map((col) => col.name);
|
||||
@@ -1876,7 +1876,7 @@ describe("schema migrations", () => {
|
||||
|
||||
db.init();
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
const cols = db.prepare("PRAGMA table_info(chat_messages)").all() as Array<{ name: string }>;
|
||||
expect(cols.map((col) => col.name)).toContain("attachments");
|
||||
@@ -1950,7 +1950,7 @@ describe("schema migrations", () => {
|
||||
|
||||
db.init();
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'agentRatings'").all() as Array<{ name: string }>;
|
||||
expect(tables).toEqual([{ name: "agentRatings" }]);
|
||||
@@ -1974,7 +1974,7 @@ describe("schema migrations", () => {
|
||||
|
||||
db.init();
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'mission_events'").all() as Array<{ name: string }>;
|
||||
expect(tables).toEqual([{ name: "mission_events" }]);
|
||||
@@ -2078,7 +2078,7 @@ describe("schema migrations", () => {
|
||||
db.init();
|
||||
|
||||
// Verify version bumped to 29
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
// Verify new columns exist and existing data is intact
|
||||
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
|
||||
@@ -2297,7 +2297,7 @@ describe("schema migrations", () => {
|
||||
|
||||
localDb.init();
|
||||
|
||||
expect(localDb.getSchemaVersion()).toBe(98);
|
||||
expect(localDb.getSchemaVersion()).toBe(99);
|
||||
const columns = localDb.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
|
||||
expect(columns.map((column) => column.name)).toContain("tokenUsageCacheWriteTokens");
|
||||
|
||||
@@ -2608,7 +2608,7 @@ describe("createDatabase factory", () => {
|
||||
const db = createDatabase(fusionDir);
|
||||
db.init();
|
||||
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
expect(db.getLastModified()).toBeGreaterThan(0);
|
||||
|
||||
db.close();
|
||||
@@ -2762,7 +2762,7 @@ describe("migration v77 task token budget columns", () => {
|
||||
|
||||
migrated = new Database(fusion);
|
||||
migrated.init();
|
||||
expect(migrated.getSchemaVersion()).toBe(98);
|
||||
expect(migrated.getSchemaVersion()).toBe(99);
|
||||
const rows = migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
|
||||
const names = new Set(rows.map((row) => row.name));
|
||||
expect(names.has("tokenBudgetSoftAlertedAt")).toBe(true);
|
||||
@@ -2808,7 +2808,7 @@ describe("migration v67 drops orphan project auth tables", () => {
|
||||
|
||||
migrated = new Database(fusion);
|
||||
migrated.init();
|
||||
expect(migrated.getSchemaVersion()).toBe(98);
|
||||
expect(migrated.getSchemaVersion()).toBe(99);
|
||||
const tables = migrated
|
||||
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'")
|
||||
.all() as Array<{ name: string }>;
|
||||
@@ -2835,7 +2835,7 @@ describe("migration v67 drops orphan project auth tables", () => {
|
||||
|
||||
try {
|
||||
fresh.init();
|
||||
expect(fresh.getSchemaVersion()).toBe(98);
|
||||
expect(fresh.getSchemaVersion()).toBe(99);
|
||||
const tables = fresh
|
||||
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'")
|
||||
.all() as Array<{ name: string }>;
|
||||
|
||||
@@ -91,6 +91,6 @@ describe("goals schema", () => {
|
||||
});
|
||||
|
||||
it("reports schema version 92", () => {
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1000,7 +1000,7 @@ describe("Migration: pre-33 DB upgrade", () => {
|
||||
// Step 1: Create a fresh database at v33 (runs all migrations up to 33)
|
||||
const db1 = createDatabase(legacyDir);
|
||||
db1.init();
|
||||
expect(db1.getSchemaVersion()).toBe(98);
|
||||
expect(db1.getSchemaVersion()).toBe(99);
|
||||
db1.close();
|
||||
|
||||
// Step 2: Manually downgrade to version 32 and drop insight tables
|
||||
@@ -1035,7 +1035,7 @@ describe("Migration: pre-33 DB upgrade", () => {
|
||||
expect(tableNamesBefore).not.toContain("project_insight_runs");
|
||||
// Now run init — this triggers the v32→v33 migration
|
||||
db3.init();
|
||||
expect(db3.getSchemaVersion()).toBe(98);
|
||||
expect(db3.getSchemaVersion()).toBe(99);
|
||||
|
||||
// Step 4: Verify insight tables exist after migration
|
||||
const tablesAfter = db3.prepare(
|
||||
@@ -1066,12 +1066,12 @@ describe("Migration: pre-33 DB upgrade", () => {
|
||||
try {
|
||||
const db1 = createDatabase(testDir);
|
||||
db1.init();
|
||||
expect(db1.getSchemaVersion()).toBe(98);
|
||||
expect(db1.getSchemaVersion()).toBe(99);
|
||||
db1.close();
|
||||
|
||||
const db2 = createDatabase(testDir);
|
||||
expect(() => db2.init()).not.toThrow();
|
||||
expect(db2.getSchemaVersion()).toBe(98);
|
||||
expect(db2.getSchemaVersion()).toBe(99);
|
||||
db2.close();
|
||||
} finally {
|
||||
rmSync(testDir, { recursive: true, force: true });
|
||||
@@ -1085,7 +1085,7 @@ describe("Migration: pre-33 DB upgrade", () => {
|
||||
// Step 1: Create a fresh DB and run migrations
|
||||
const db1 = createDatabase(compatDir);
|
||||
db1.init();
|
||||
expect(db1.getSchemaVersion()).toBe(98);
|
||||
expect(db1.getSchemaVersion()).toBe(99);
|
||||
|
||||
// Step 2: Strip lifecycle and cancelledAt columns by recreating the
|
||||
// table without them. This simulates a DB that was created before the
|
||||
|
||||
@@ -3200,7 +3200,7 @@ describe("MissionStore", () => {
|
||||
|
||||
describe("Loop State & Validator Run Schema (v31)", () => {
|
||||
it("schema version is 40 after migration", () => {
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
});
|
||||
|
||||
it("mission_features table has loop state columns", () => {
|
||||
|
||||
@@ -584,7 +584,7 @@ describe("Run Audit", () => {
|
||||
});
|
||||
|
||||
it("schema version is bumped to 40", () => {
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -42,7 +42,7 @@ describe("secrets schema migrations", () => {
|
||||
const version = db
|
||||
.prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'")
|
||||
.get() as { value: string };
|
||||
expect(version.value).toBe("98");
|
||||
expect(version.value).toBe("99");
|
||||
} finally {
|
||||
db.close();
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
@@ -105,7 +105,7 @@ describe("secrets schema migrations", () => {
|
||||
const version = db
|
||||
.prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'")
|
||||
.get() as { value: string };
|
||||
expect(version.value).toBe("98");
|
||||
expect(version.value).toBe("99");
|
||||
} finally {
|
||||
db.close();
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
@@ -155,7 +155,7 @@ describe("secrets schema migrations", () => {
|
||||
.prepare("SELECT value FROM __meta WHERE key = 'schemaVersion'")
|
||||
.get() as { value: string };
|
||||
|
||||
expect(projectVersion.value).toBe("98");
|
||||
expect(projectVersion.value).toBe("99");
|
||||
expect(centralVersion.value).toBe("13");
|
||||
} finally {
|
||||
projectDb.close();
|
||||
|
||||
@@ -60,7 +60,7 @@ describe("TaskStore merge queue", () => {
|
||||
expect.arrayContaining(["idx_mergeQueue_lease_ready", "idx_mergeQueue_leaseExpiresAt"]),
|
||||
);
|
||||
|
||||
expect(store.getDatabase().getSchemaVersion()).toBe(98);
|
||||
expect(store.getDatabase().getSchemaVersion()).toBe(99);
|
||||
});
|
||||
|
||||
it("migrates a legacy v88 database and preserves task rows", async () => {
|
||||
|
||||
@@ -51,7 +51,7 @@ describe("TaskStore task documents", () => {
|
||||
|
||||
expect(tableNames.has("task_documents")).toBe(true);
|
||||
expect(tableNames.has("task_document_revisions")).toBe(true);
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
|
||||
const index = db
|
||||
.prepare(
|
||||
|
||||
@@ -149,7 +149,7 @@ export function probeFts5(db: DatabaseSync): boolean {
|
||||
|
||||
// ── Schema Definition ────────────────────────────────────────────────
|
||||
|
||||
const SCHEMA_VERSION = 98;
|
||||
const SCHEMA_VERSION = 99;
|
||||
|
||||
function normalizeTaskComments(
|
||||
steeringComments: SteeringComment[] | undefined,
|
||||
@@ -242,6 +242,9 @@ CREATE TABLE IF NOT EXISTS tasks (
|
||||
planningModelId TEXT,
|
||||
mergeRetries INTEGER,
|
||||
workflowStepRetries INTEGER,
|
||||
resumeLimboCount INTEGER DEFAULT 0,
|
||||
resumeLimboTipSha TEXT,
|
||||
resumeLimboStepSignature TEXT,
|
||||
recoveryRetryCount INTEGER,
|
||||
taskDoneRetryCount INTEGER DEFAULT 0,
|
||||
worktreeSessionRetryCount INTEGER DEFAULT 0,
|
||||
@@ -3726,6 +3729,14 @@ export class Database {
|
||||
});
|
||||
}
|
||||
|
||||
if (version < 99) {
|
||||
this.applyMigration(99, () => {
|
||||
this.addColumnIfMissing("tasks", "resumeLimboCount", "INTEGER DEFAULT 0");
|
||||
this.addColumnIfMissing("tasks", "resumeLimboTipSha", "TEXT");
|
||||
this.addColumnIfMissing("tasks", "resumeLimboStepSignature", "TEXT");
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -2,6 +2,7 @@ import type { Task } from "./types.js";
|
||||
|
||||
export const MANUAL_RETRY_RESET_COUNTER_KEYS = [
|
||||
"stuckKillCount",
|
||||
"resumeLimboCount",
|
||||
"recoveryRetryCount",
|
||||
"taskDoneRetryCount",
|
||||
"worktreeSessionRetryCount",
|
||||
|
||||
@@ -95,6 +95,9 @@ interface TaskRow {
|
||||
mergeRetries: number | null;
|
||||
workflowStepRetries: number | null;
|
||||
stuckKillCount: number | null;
|
||||
resumeLimboCount: number | null;
|
||||
resumeLimboTipSha: string | null;
|
||||
resumeLimboStepSignature: string | null;
|
||||
postReviewFixCount: number | null;
|
||||
recoveryRetryCount: number | null;
|
||||
taskDoneRetryCount: number | null;
|
||||
@@ -1415,6 +1418,9 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
mergeRetries: row.mergeRetries ?? undefined,
|
||||
workflowStepRetries: row.workflowStepRetries ?? undefined,
|
||||
stuckKillCount: row.stuckKillCount ?? undefined,
|
||||
resumeLimboCount: row.resumeLimboCount ?? undefined,
|
||||
resumeLimboTipSha: row.resumeLimboTipSha || undefined,
|
||||
resumeLimboStepSignature: row.resumeLimboStepSignature || undefined,
|
||||
postReviewFixCount: row.postReviewFixCount ?? undefined,
|
||||
recoveryRetryCount: row.recoveryRetryCount ?? undefined,
|
||||
taskDoneRetryCount: row.taskDoneRetryCount ?? undefined,
|
||||
@@ -1924,7 +1930,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
"modelPresetId", "modelProvider", "modelId",
|
||||
"validatorModelProvider", "validatorModelId",
|
||||
"planningModelProvider", "planningModelId",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"error", "summary", "thinkingLevel", "executionMode",
|
||||
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
|
||||
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt",
|
||||
@@ -1973,7 +1979,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
"modelPresetId", "modelProvider", "modelId",
|
||||
"validatorModelProvider", "validatorModelId",
|
||||
"planningModelProvider", "planningModelId",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"error", "summary", "thinkingLevel", "executionMode",
|
||||
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
|
||||
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt",
|
||||
@@ -2039,6 +2045,9 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
task.mergeRetries ?? null,
|
||||
task.workflowStepRetries ?? null,
|
||||
task.stuckKillCount ?? 0,
|
||||
task.resumeLimboCount ?? 0,
|
||||
task.resumeLimboTipSha ?? null,
|
||||
task.resumeLimboStepSignature ?? null,
|
||||
task.postReviewFixCount ?? 0,
|
||||
task.recoveryRetryCount ?? null,
|
||||
task.taskDoneRetryCount ?? 0,
|
||||
@@ -2138,7 +2147,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
id, lineageId, title, description, priority, "column", status, size, reviewLevel, currentStep,
|
||||
worktree, blockedBy, overlapBlockedBy, paused, userPaused, baseBranch, branch, autoMerge, executionStartBranch, baseCommitSha, modelPresetId, modelProvider,
|
||||
modelId, validatorModelProvider, validatorModelId, planningModelProvider, planningModelId, mergeRetries,
|
||||
workflowStepRetries, stuckKillCount, postReviewFixCount, recoveryRetryCount, taskDoneRetryCount, worktreeSessionRetryCount, completionHandoffLimboRecoveryCount, verificationFailureCount, mergeConflictBounceCount, mergeAuditBounceCount, mergeTransientRetryCount, branchConflictRecoveryCount, reviewerContextRetryCount, reviewerFallbackRetryCount, nextRecoveryAt, error,
|
||||
workflowStepRetries, stuckKillCount, resumeLimboCount, resumeLimboTipSha, resumeLimboStepSignature, postReviewFixCount, recoveryRetryCount, taskDoneRetryCount, worktreeSessionRetryCount, completionHandoffLimboRecoveryCount, verificationFailureCount, mergeConflictBounceCount, mergeAuditBounceCount, mergeTransientRetryCount, branchConflictRecoveryCount, reviewerContextRetryCount, reviewerFallbackRetryCount, nextRecoveryAt, error,
|
||||
summary, thinkingLevel, executionMode, tokenUsageInputTokens, tokenUsageOutputTokens, tokenUsageCachedTokens,
|
||||
tokenUsageCacheWriteTokens, tokenUsageTotalTokens, tokenUsageFirstUsedAt, tokenUsageLastUsedAt, tokenBudgetSoftAlertedAt, tokenBudgetHardAlertedAt, tokenBudgetOverride, createdAt, updatedAt, columnMovedAt,
|
||||
firstExecutionAt, cumulativeActiveMs, executionStartedAt, executionCompletedAt,
|
||||
@@ -2165,7 +2174,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
id, lineageId, title, description, priority, "column", status, size, reviewLevel, currentStep,
|
||||
worktree, blockedBy, overlapBlockedBy, paused, userPaused, baseBranch, branch, autoMerge, executionStartBranch, baseCommitSha, modelPresetId, modelProvider,
|
||||
modelId, validatorModelProvider, validatorModelId, planningModelProvider, planningModelId, mergeRetries,
|
||||
workflowStepRetries, stuckKillCount, postReviewFixCount, recoveryRetryCount, taskDoneRetryCount, worktreeSessionRetryCount, completionHandoffLimboRecoveryCount, verificationFailureCount, mergeConflictBounceCount, mergeAuditBounceCount, mergeTransientRetryCount, branchConflictRecoveryCount, reviewerContextRetryCount, reviewerFallbackRetryCount, nextRecoveryAt, error,
|
||||
workflowStepRetries, stuckKillCount, resumeLimboCount, resumeLimboTipSha, resumeLimboStepSignature, postReviewFixCount, recoveryRetryCount, taskDoneRetryCount, worktreeSessionRetryCount, completionHandoffLimboRecoveryCount, verificationFailureCount, mergeConflictBounceCount, mergeAuditBounceCount, mergeTransientRetryCount, branchConflictRecoveryCount, reviewerContextRetryCount, reviewerFallbackRetryCount, nextRecoveryAt, error,
|
||||
summary, thinkingLevel, executionMode, tokenUsageInputTokens, tokenUsageOutputTokens, tokenUsageCachedTokens,
|
||||
tokenUsageCacheWriteTokens, tokenUsageTotalTokens, tokenUsageFirstUsedAt, tokenUsageLastUsedAt, tokenBudgetSoftAlertedAt, tokenBudgetHardAlertedAt, tokenBudgetOverride, createdAt, updatedAt, columnMovedAt,
|
||||
firstExecutionAt, cumulativeActiveMs, executionStartedAt, executionCompletedAt,
|
||||
@@ -2204,6 +2213,9 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
mergeRetries = excluded.mergeRetries,
|
||||
workflowStepRetries = excluded.workflowStepRetries,
|
||||
stuckKillCount = excluded.stuckKillCount,
|
||||
resumeLimboCount = excluded.resumeLimboCount,
|
||||
resumeLimboTipSha = excluded.resumeLimboTipSha,
|
||||
resumeLimboStepSignature = excluded.resumeLimboStepSignature,
|
||||
postReviewFixCount = excluded.postReviewFixCount,
|
||||
recoveryRetryCount = excluded.recoveryRetryCount,
|
||||
taskDoneRetryCount = excluded.taskDoneRetryCount,
|
||||
@@ -5548,7 +5560,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
|
||||
async updateTask(
|
||||
id: string,
|
||||
updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record<string, unknown> | null; githubTracking?: import("./types.js").TaskGithubTracking | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null },
|
||||
updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record<string, unknown> | null; githubTracking?: import("./types.js").TaskGithubTracking | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null },
|
||||
runContext?: RunMutationContext,
|
||||
): Promise<Task> {
|
||||
return this.withTaskLock(id, async () => {
|
||||
@@ -5857,6 +5869,21 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
} else if (updates.stuckKillCount !== undefined) {
|
||||
task.stuckKillCount = updates.stuckKillCount;
|
||||
}
|
||||
if (updates.resumeLimboCount === null) {
|
||||
task.resumeLimboCount = undefined;
|
||||
} else if (updates.resumeLimboCount !== undefined) {
|
||||
task.resumeLimboCount = updates.resumeLimboCount;
|
||||
}
|
||||
if (updates.resumeLimboTipSha === null) {
|
||||
task.resumeLimboTipSha = undefined;
|
||||
} else if (updates.resumeLimboTipSha !== undefined) {
|
||||
task.resumeLimboTipSha = updates.resumeLimboTipSha;
|
||||
}
|
||||
if (updates.resumeLimboStepSignature === null) {
|
||||
task.resumeLimboStepSignature = undefined;
|
||||
} else if (updates.resumeLimboStepSignature !== undefined) {
|
||||
task.resumeLimboStepSignature = updates.resumeLimboStepSignature;
|
||||
}
|
||||
if (updates.postReviewFixCount === null) {
|
||||
task.postReviewFixCount = undefined;
|
||||
} else if (updates.postReviewFixCount !== undefined) {
|
||||
|
||||
@@ -1887,6 +1887,17 @@ export interface Task {
|
||||
* Incremented by the self-healing manager on each stuck kill. When this reaches
|
||||
* `maxStuckKills`, the task is marked as permanently failed instead of re-queued. */
|
||||
stuckKillCount?: number;
|
||||
/** Number of consecutive reclaim/unpause attempts where no execution progress
|
||||
* materialized (tip unchanged, step signature unchanged, and no active session).
|
||||
* Incremented by self-healing for resume-limbo detection and reset when
|
||||
* progress is observed or recovery escalates to a fresh todo dispatch. */
|
||||
resumeLimboCount?: number;
|
||||
/** Branch tip SHA snapshot captured at the last reclaim/unpause attempt used
|
||||
* by resume-limbo detection to determine whether commits advanced. */
|
||||
resumeLimboTipSha?: string;
|
||||
/** Compact execution-progress snapshot captured at the last reclaim/unpause
|
||||
* attempt (current step + step statuses) for resume-limbo detection. */
|
||||
resumeLimboStepSignature?: string;
|
||||
/** Number of times the self-healing manager has auto-revived this task from
|
||||
* `in-review` after a failed pre-merge workflow step. Incremented each time the
|
||||
* `recoverReviewTasksWithFailedPreMergeSteps` scan sends the task back with the
|
||||
|
||||
@@ -0,0 +1,168 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { EventEmitter } from "node:events";
|
||||
import type { Settings, Task, TaskStore } from "@fusion/core";
|
||||
import { SelfHealingManager } from "../../self-healing.js";
|
||||
import * as branchConflictModule from "../../branch-conflicts.js";
|
||||
import * as worktreePoolModule from "../../worktree-pool.js";
|
||||
|
||||
type MutableSettings = Settings & {
|
||||
autoMerge?: boolean;
|
||||
globalPause?: boolean;
|
||||
enginePaused?: boolean;
|
||||
};
|
||||
|
||||
function makeTask(overrides: Partial<Task> = {}): Task {
|
||||
return {
|
||||
id: "FN-5704",
|
||||
title: "resume limbo",
|
||||
description: "test",
|
||||
column: "in-progress",
|
||||
branch: "fusion/fn-5704",
|
||||
worktree: "/tmp/test/.worktrees/fn-5704",
|
||||
paused: false,
|
||||
userPaused: false,
|
||||
checkedOutBy: undefined,
|
||||
dependencies: [],
|
||||
steps: [{ id: "s1", title: "step", status: "in-progress" } as any],
|
||||
currentStep: 1,
|
||||
log: [],
|
||||
createdAt: new Date().toISOString(),
|
||||
updatedAt: new Date("2026-01-01T00:00:00.000Z").toISOString(),
|
||||
executionStartedAt: new Date("2026-01-01T00:00:00.000Z").toISOString(),
|
||||
...overrides,
|
||||
} as Task;
|
||||
}
|
||||
|
||||
function makeStore(task: Task, settingsOverrides: Partial<MutableSettings> = {}): TaskStore & EventEmitter {
|
||||
const emitter = new EventEmitter();
|
||||
const settings = {
|
||||
autoMerge: true,
|
||||
globalPause: false,
|
||||
enginePaused: false,
|
||||
baseBranch: "main",
|
||||
mergeStrategy: "direct",
|
||||
autoRecovery: { mode: "deterministic-only", maxRetries: 3 },
|
||||
...settingsOverrides,
|
||||
} as unknown as Settings;
|
||||
|
||||
return Object.assign(emitter, {
|
||||
getSettings: vi.fn(async () => settings),
|
||||
listTasks: vi.fn(async ({ column }: { column?: string } = {}) => (column === task.column ? [task] : [])),
|
||||
updateTask: vi.fn(async (_id: string, updates: Partial<Task>) => Object.assign(task, updates)),
|
||||
moveTask: vi.fn(async (_id: string, column: Task["column"], opts?: Record<string, unknown>) => {
|
||||
task.column = column;
|
||||
(task as any).__lastMoveOpts = opts;
|
||||
return task;
|
||||
}),
|
||||
logEntry: vi.fn(async () => undefined),
|
||||
recordRunAuditEvent: vi.fn(async () => undefined),
|
||||
appendAgentLog: vi.fn(async () => undefined),
|
||||
updateSettings: vi.fn(async () => settings),
|
||||
clearStaleExecutionStartBranchReferences: vi.fn(() => []),
|
||||
walCheckpoint: vi.fn(() => ({ busy: 0, log: 0, checkpointed: 0 })),
|
||||
archiveTaskAndCleanup: vi.fn(async () => ({})),
|
||||
mergeTask: vi.fn(async () => undefined),
|
||||
getRootDir: vi.fn(() => "/tmp/test"),
|
||||
}) as unknown as TaskStore & EventEmitter;
|
||||
}
|
||||
|
||||
describe("FN-5704: reclaim self-owned resume limbo escalation", () => {
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
vi.spyOn(worktreePoolModule, "isUsableTaskWorktree").mockResolvedValue(true);
|
||||
});
|
||||
|
||||
it("escalates frozen in-progress reclaim/resume loops to todo with preserve flags and audit event", async () => {
|
||||
vi.useFakeTimers();
|
||||
vi.setSystemTime(new Date("2026-01-01T02:00:00.000Z"));
|
||||
const task = makeTask();
|
||||
const store = makeStore(task);
|
||||
vi.spyOn(branchConflictModule, "inspectBranchConflict").mockResolvedValue({
|
||||
kind: "reclaimable",
|
||||
taskAttributedCommitCount: 1,
|
||||
strandedCommits: [{ sha: "c1", authorName: "a", subject: "s", timestamp: Date.now() }],
|
||||
livePath: task.worktree,
|
||||
tipSha: "abc123abc123abc123abc123abc123abc123abcd",
|
||||
} as any);
|
||||
|
||||
const manager = new SelfHealingManager(store as any, { rootDir: "/tmp/test" } as any);
|
||||
await manager.reclaimSelfOwnedBranchConflicts();
|
||||
expect(task.resumeLimboCount).toBe(0);
|
||||
expect((store.moveTask as any).mock.calls.length).toBe(0);
|
||||
|
||||
await manager.reclaimSelfOwnedBranchConflicts();
|
||||
expect(task.resumeLimboCount).toBe(1);
|
||||
await manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(store.moveTask).toHaveBeenCalledWith(task.id, "todo", expect.objectContaining({
|
||||
moveSource: "engine",
|
||||
preserveWorktree: true,
|
||||
preserveProgress: true,
|
||||
preserveResumeState: true,
|
||||
}));
|
||||
expect(task.resumeLimboCount).toBe(0);
|
||||
const limboEvent = (store.recordRunAuditEvent as any).mock.calls.find((call: any[]) => call[0].mutationType === "task:resume-limbo-escalated")?.[0];
|
||||
expect(limboEvent).toBeTruthy();
|
||||
expect(limboEvent.target).toBe(task.id);
|
||||
expect(limboEvent.metadata).toEqual(expect.objectContaining({
|
||||
taskId: task.id,
|
||||
frozenTipSha: "abc123abc123abc123abc123abc123abc123abcd",
|
||||
resumeAttemptCount: 2,
|
||||
currentStep: 1,
|
||||
}));
|
||||
const auditMetadata = limboEvent.metadata;
|
||||
expect(auditMetadata.idleMs).toBeGreaterThan(0);
|
||||
|
||||
vi.useRealTimers();
|
||||
manager.stop();
|
||||
});
|
||||
|
||||
it("resets limbo counter on progress and avoids escalation", async () => {
|
||||
const task = makeTask();
|
||||
const store = makeStore(task);
|
||||
const inspect = vi.spyOn(branchConflictModule, "inspectBranchConflict");
|
||||
inspect.mockResolvedValueOnce({ kind: "reclaimable", taskAttributedCommitCount: 1, strandedCommits: [{ sha: "c1", authorName: "a", subject: "s", timestamp: Date.now() }], livePath: task.worktree, tipSha: "sha-1" } as any);
|
||||
inspect.mockResolvedValueOnce({ kind: "reclaimable", taskAttributedCommitCount: 1, strandedCommits: [{ sha: "c1", authorName: "a", subject: "s", timestamp: Date.now() }], livePath: task.worktree, tipSha: "sha-1" } as any);
|
||||
inspect.mockResolvedValueOnce({ kind: "reclaimable", taskAttributedCommitCount: 1, strandedCommits: [{ sha: "c2", authorName: "a", subject: "s", timestamp: Date.now() }], livePath: task.worktree, tipSha: "sha-2" } as any);
|
||||
inspect.mockResolvedValueOnce({ kind: "reclaimable", taskAttributedCommitCount: 1, strandedCommits: [{ sha: "c2", authorName: "a", subject: "s", timestamp: Date.now() }], livePath: task.worktree, tipSha: "sha-2" } as any);
|
||||
|
||||
const manager = new SelfHealingManager(store as any, { rootDir: "/tmp/test" } as any);
|
||||
await manager.reclaimSelfOwnedBranchConflicts();
|
||||
await manager.reclaimSelfOwnedBranchConflicts();
|
||||
expect(task.resumeLimboCount).toBe(1);
|
||||
await manager.reclaimSelfOwnedBranchConflicts();
|
||||
expect(task.resumeLimboCount).toBe(0);
|
||||
await manager.reclaimSelfOwnedBranchConflicts();
|
||||
expect(task.resumeLimboCount).toBe(1);
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
|
||||
manager.stop();
|
||||
});
|
||||
|
||||
it("never escalates user-paused tasks", async () => {
|
||||
const task = makeTask({ userPaused: true });
|
||||
const store = makeStore(task);
|
||||
const inspectSpy = vi.spyOn(branchConflictModule, "inspectBranchConflict");
|
||||
|
||||
const manager = new SelfHealingManager(store as any, { rootDir: "/tmp/test" } as any);
|
||||
await manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(inspectSpy).not.toHaveBeenCalled();
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
manager.stop();
|
||||
});
|
||||
|
||||
it("short-circuits reclaim when autoMerge is false", async () => {
|
||||
const task = makeTask();
|
||||
const store = makeStore(task, { autoMerge: false });
|
||||
const inspectSpy = vi.spyOn(branchConflictModule, "inspectBranchConflict");
|
||||
|
||||
const manager = new SelfHealingManager(store as any, { rootDir: "/tmp/test" } as any);
|
||||
const recovered = await manager.reclaimSelfOwnedBranchConflicts();
|
||||
|
||||
expect(recovered).toBe(0);
|
||||
expect(inspectSpy).not.toHaveBeenCalled();
|
||||
expect(store.moveTask).not.toHaveBeenCalled();
|
||||
manager.stop();
|
||||
});
|
||||
});
|
||||
@@ -1960,6 +1960,11 @@ export class TaskExecutor {
|
||||
executorLog.log(`Unpaused ${task.id} in-progress with no session — resuming execution`);
|
||||
try {
|
||||
await this.clearResumeFailureState(task);
|
||||
await this.store.updateTask(task.id, {
|
||||
resumeLimboCount: 0,
|
||||
resumeLimboTipSha: null,
|
||||
resumeLimboStepSignature: null,
|
||||
});
|
||||
await this.store.logEntry(task.id, "Resuming execution after unpause", undefined, this.getRunContextFor(task.id));
|
||||
await this.recoverApprovedStepsOnResume(task.id);
|
||||
} catch (clearErr) {
|
||||
|
||||
@@ -462,6 +462,7 @@ export type DatabaseMutationType =
|
||||
| "task:auto-recover-completion-handoff-limbo-exhausted"
|
||||
| "task:auto-recover-worktree-session-exhausted"
|
||||
| "task:auto-recover-in-progress-limbo"
|
||||
| "task:resume-limbo-escalated"
|
||||
| "task:orphan-detected-no-action"
|
||||
/** Metadata: { taskId: string; ignoredStepUpdateCount: number; stuckKillStreak: number; lastReason: "no-progress-churn" } */
|
||||
| "task:stuck-no-progress-churn-terminalized"
|
||||
|
||||
@@ -72,6 +72,7 @@ const DB_CORRUPTION_NOTIFICATION_COOLDOWN_MS = 60 * 60 * 1000;
|
||||
export const STALE_ACTIVE_BRANCH_EXECUTION_GRACE_MS = 10 * 60_000;
|
||||
export const COMPLETION_HANDOFF_LIMBO_GRACE_MS = 5 * 60_000;
|
||||
export const MAX_COMPLETION_HANDOFF_LIMBO_RECOVERIES = 3;
|
||||
const MAX_NO_PROGRESS_RESUME_ATTEMPTS = 2;
|
||||
|
||||
// listTasks already enforces ACTIVE_TASKS_WHERE (`"deletedAt" IS NULL`), but
|
||||
// deadlock/stall sweeps still defensively skip soft-deleted rows in case a
|
||||
@@ -106,6 +107,13 @@ async function classifyOwnedLandedEvidenceForSelfHealing(rootDir: string, task:
|
||||
return classifyOwnedLandedEvidence(rootDir, task, { mergeTargetBranch });
|
||||
}
|
||||
|
||||
function buildResumeLimboStepSignature(task: Task): string {
|
||||
return JSON.stringify({
|
||||
currentStep: task.currentStep ?? null,
|
||||
steps: Array.isArray(task.steps) ? task.steps.map((step) => step.status) : [],
|
||||
});
|
||||
}
|
||||
|
||||
function formatRecoveryTimestamp(date = new Date()): string {
|
||||
const pad = (value: number) => String(value).padStart(2, "0");
|
||||
return `${date.getUTCFullYear()}${pad(date.getUTCMonth() + 1)}${pad(date.getUTCDate())}-${pad(date.getUTCHours())}${pad(date.getUTCMinutes())}${pad(date.getUTCSeconds())}`;
|
||||
@@ -2328,6 +2336,67 @@ export class SelfHealingManager {
|
||||
const preservedCommitCount = inspection.kind === "fully-subsumed"
|
||||
? 0
|
||||
: inspection.taskAttributedCommitCount;
|
||||
const stepSignature = buildResumeLimboStepSignature(task);
|
||||
const hasActiveSessionSignal = Boolean(task.checkedOutBy) || activeTaskIds.has(task.id.toUpperCase());
|
||||
const hasPriorSnapshot = typeof task.resumeLimboTipSha === "string" && typeof task.resumeLimboStepSignature === "string";
|
||||
const unchangedSincePriorResume = hasPriorSnapshot
|
||||
&& task.resumeLimboTipSha === inspection.tipSha
|
||||
&& task.resumeLimboStepSignature === stepSignature;
|
||||
const isNoProgressResume = task.column === "in-progress"
|
||||
&& unchangedSincePriorResume
|
||||
&& !hasActiveSessionSignal;
|
||||
const resumeAttemptCount = isNoProgressResume ? (task.resumeLimboCount ?? 0) + 1 : 0;
|
||||
|
||||
if (task.column === "in-progress" && isNoProgressResume && resumeAttemptCount >= MAX_NO_PROGRESS_RESUME_ATTEMPTS) {
|
||||
const idleAnchor = task.executionStartedAt ?? task.columnMovedAt ?? task.updatedAt;
|
||||
const idleAnchorMs = Date.parse(idleAnchor ?? "");
|
||||
const idleMs = Number.isFinite(idleAnchorMs) ? Math.max(0, Date.now() - idleAnchorMs) : null;
|
||||
await this.store.moveTask(task.id, "todo", {
|
||||
moveSource: "engine",
|
||||
preserveWorktree: true,
|
||||
preserveProgress: true,
|
||||
preserveResumeState: true,
|
||||
});
|
||||
await this.store.updateTask(task.id, {
|
||||
resumeLimboCount: 0,
|
||||
resumeLimboTipSha: inspection.tipSha,
|
||||
resumeLimboStepSignature: stepSignature,
|
||||
});
|
||||
await this.store.logEntry(
|
||||
task.id,
|
||||
`[recovery] resume-limbo-escalated ${task.id} moved to todo after ${resumeAttemptCount} no-progress reclaim/resume attempts`,
|
||||
JSON.stringify({
|
||||
frozenTipSha: inspection.tipSha,
|
||||
idleMs,
|
||||
resumeAttemptCount,
|
||||
currentStep: task.currentStep ?? null,
|
||||
}),
|
||||
);
|
||||
try {
|
||||
await createRunAuditor(this.store, {
|
||||
runId: generateSyntheticRunId("self-heal", task.id),
|
||||
agentId: "self-healing",
|
||||
taskId: task.id,
|
||||
taskLineageId: task.lineageId,
|
||||
phase: "reclaim-self-owned-branch-conflicts",
|
||||
}).database({
|
||||
type: "task:resume-limbo-escalated",
|
||||
target: task.id,
|
||||
metadata: {
|
||||
taskId: task.id,
|
||||
frozenTipSha: inspection.tipSha,
|
||||
idleMs,
|
||||
resumeAttemptCount,
|
||||
currentStep: task.currentStep ?? null,
|
||||
},
|
||||
});
|
||||
} catch (auditErr: unknown) {
|
||||
log.warn(`Failed to write task:resume-limbo-escalated run-audit event for ${task.id}: ${auditErr instanceof Error ? auditErr.message : String(auditErr)}`);
|
||||
}
|
||||
recovered++;
|
||||
continue;
|
||||
}
|
||||
|
||||
await this.store.updateTask(task.id, {
|
||||
worktree: inspection.livePath,
|
||||
branch: task.branch,
|
||||
@@ -2335,6 +2404,9 @@ export class SelfHealingManager {
|
||||
pausedReason: undefined,
|
||||
status: null,
|
||||
error: null,
|
||||
resumeLimboCount: resumeAttemptCount,
|
||||
resumeLimboTipSha: inspection.tipSha,
|
||||
resumeLimboStepSignature: stepSignature,
|
||||
});
|
||||
await this.store.logEntry(
|
||||
task.id,
|
||||
|
||||
@@ -743,8 +743,8 @@ describe("RoadmapStore", () => {
|
||||
});
|
||||
|
||||
describe("schema version", () => {
|
||||
it("schema version is 98 after init", () => {
|
||||
expect(db.getSchemaVersion()).toBe(98);
|
||||
it("schema version is 99 after init", () => {
|
||||
expect(db.getSchemaVersion()).toBe(99);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user