test(FN-4651): complete Step 4/5 verification and docs updates

Fusion-Task-Id: FN-4651
Fusion-Task-Lineage: 14d15555-468d-4690-8efd-a09b31b5d407
This commit is contained in:
Fusion
2026-05-15 12:50:04 -07:00
committed by gsxdsm
parent ebd0be13d9
commit ef1ce1071b
10 changed files with 40 additions and 31 deletions

View File

@@ -0,0 +1,8 @@
---
"@runfusion/fusion": patch
---
Self-healing now auto-requeues `in-review` tasks that failed at session start
with an unusable-worktree error even when zero step progress was recorded.
Bounded by a 3-attempt cap; persistent failures stay in `in-review` for
human inspection.

View File

@@ -203,7 +203,7 @@ Port 4040 is the production dashboard port. A user's live dashboard session is t
## Architecture ## Architecture
- Merge deadlock self-healing now layers `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and `reclaimSelfOwnedBranchConflicts()` in `packages/engine/src/self-healing.ts`, plus the paused-aware in-review scope filter in `packages/engine/src/scheduler.ts` (`inReviewWithWorktree` excludes `paused` tasks). `reclaimSelfOwnedBranchConflicts()` now also recovers paused `branch-conflict-unrecoverable` review rows when ownership is self-proven, auto-reclaims `fusion/<task-id>` branches that are still live-mapped but have zero unique commits vs main by force-removing the stale worktree and deleting the branch, and clears `task.worktree`/`task.branch` so retries recreate a fresh checkout. `inspectBranchConflict()` backs this with a patch-id fallback for degraded/empty `git cherry` output before declaring zero-unique-commit subsumption, treats ghost live-worktree admin entries (mapping path missing on disk) as `stale-resolved`, and classifies tips already reachable from the integration target as `tip-already-merged` so stale cached `baseCommitSha` values are invalidated instead of enumerating main's forward progress as stranded commits. FN-4546 adds a separate `reclaim-stale-active-branches` stage in startup and maintenance sweeps to close the gap where an active task owns a `fusion/<task-id>` branch with zero unique commits but has no usable worktree mapping; that stage now prunes the stale branch, clears task branch/worktree/base metadata, and emits `branch:stale-active-reclaim` audit telemetry. FN-4601 hardens executor recovery on this path: the no-`fn_task_done` retry loop now re-validates live worktree/branch bindings before recreating a session and treats missing/incomplete/unregistered-worktree session-start failures as recoverable by clearing stale metadata and requeueing to `todo` with preserved progress. Orphan `fusion/*` branches are still resolved by prune-or-rescue logic (subsumed branches pruned; unique-commit branches rescued into triage tasks instead of force delete). Completion fan-out now runs synchronously on `in-review → done` via `SelfHealingManager.reconcileCompletedTask()`, so downstream stale `blockedBy` links and residual `fusion/<task-id>` branch/worktree artifacts are reconciled immediately instead of waiting for periodic sweeps. - Merge deadlock self-healing now layers `recoverAlreadyMergedReviewTasks()`, `clearStaleBlockedBy()`, and `reclaimSelfOwnedBranchConflicts()` in `packages/engine/src/self-healing.ts`, plus the paused-aware in-review scope filter in `packages/engine/src/scheduler.ts` (`inReviewWithWorktree` excludes `paused` tasks). `reclaimSelfOwnedBranchConflicts()` now also recovers paused `branch-conflict-unrecoverable` review rows when ownership is self-proven, auto-reclaims `fusion/<task-id>` branches that are still live-mapped but have zero unique commits vs main by force-removing the stale worktree and deleting the branch, and clears `task.worktree`/`task.branch` so retries recreate a fresh checkout. `inspectBranchConflict()` backs this with a patch-id fallback for degraded/empty `git cherry` output before declaring zero-unique-commit subsumption, treats ghost live-worktree admin entries (mapping path missing on disk) as `stale-resolved`, and classifies tips already reachable from the integration target as `tip-already-merged` so stale cached `baseCommitSha` values are invalidated instead of enumerating main's forward progress as stranded commits. FN-4546 adds a separate `reclaim-stale-active-branches` stage in startup and maintenance sweeps to close the gap where an active task owns a `fusion/<task-id>` branch with zero unique commits but has no usable worktree mapping; that stage now prunes the stale branch, clears task branch/worktree/base metadata, and emits `branch:stale-active-reclaim` audit telemetry. FN-4601 hardens executor recovery on this path: the no-`fn_task_done` retry loop now re-validates live worktree/branch bindings before recreating a session and treats missing/incomplete/unregistered-worktree session-start failures as recoverable by clearing stale metadata and requeueing to `todo` with preserved progress. Orphan `fusion/*` branches are still resolved by prune-or-rescue logic (subsumed branches pruned; unique-commit branches rescued into triage tasks instead of force delete). Completion fan-out now runs synchronously on `in-review → done` via `SelfHealingManager.reconcileCompletedTask()`, so downstream stale `blockedBy` links and residual `fusion/<task-id>` branch/worktree artifacts are reconciled immediately instead of waiting for periodic sweeps.
- Restart recovery is coordinated through `RestartRecoveryCoordinator` (`packages/engine/src/restart-recovery-coordinator.ts`), which classifies interrupted `in-progress` runs at runtime startup: no-progress `fn_task_done` failures are safely requeued to `todo`, then remaining orphaned work is resumed via the executor. Session-start unusable-worktree detection now treats all `assertValidWorktreeSession()` variants (`missing worktree`, `incomplete worktree`, and `unregistered git worktree`) as recoverable for self-healing auto-requeue. - Restart recovery is coordinated through `RestartRecoveryCoordinator` (`packages/engine/src/restart-recovery-coordinator.ts`), which classifies interrupted `in-progress` runs at runtime startup: no-progress `fn_task_done` failures are safely requeued to `todo`, then remaining orphaned work is resumed via the executor. Session-start unusable-worktree detection now treats all `assertValidWorktreeSession()` variants (`missing worktree`, `incomplete worktree`, and `unregistered git worktree`) as recoverable for self-healing auto-requeue, including zero-progress `in-review` failures that now retry with a bounded `worktreeSessionRetryCount` cap (`MAX_WORKTREE_SESSION_RETRIES=3`) before escalating for human inspection.
## Engine Process Rules ## Engine Process Rules

View File

@@ -717,7 +717,7 @@ describe("schema migration", () => {
{ id: "WS-001", mode: "prompt", gateMode: "advisory" }, { id: "WS-001", mode: "prompt", gateMode: "advisory" },
{ id: "WS-002", mode: "script", gateMode: "advisory" }, { id: "WS-002", mode: "script", gateMode: "advisory" },
]); ]);
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
db.close(); db.close();
}); });
@@ -767,7 +767,7 @@ describe("schema migration", () => {
reviewerContextRetryCount: 0, reviewerContextRetryCount: 0,
reviewerFallbackRetryCount: 0, reviewerFallbackRetryCount: 0,
}); });
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
db.close(); db.close();
}); });
@@ -796,7 +796,7 @@ describe("schema migration", () => {
const columns = db.prepare("PRAGMA table_info(milestones)").all() as Array<{ name: string }>; const columns = db.prepare("PRAGMA table_info(milestones)").all() as Array<{ name: string }>;
expect(columns.map((column) => column.name)).toContain("acceptanceCriteria"); expect(columns.map((column) => column.name)).toContain("acceptanceCriteria");
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
db.close(); db.close();
}); });
@@ -831,7 +831,7 @@ describe("schema migration", () => {
{ id: "WS-002", mode: "script", enabled: 1, gateMode: "advisory" }, { id: "WS-002", mode: "script", enabled: 1, gateMode: "advisory" },
{ id: "WS-003", mode: "prompt", enabled: 0, gateMode: "advisory" }, { id: "WS-003", mode: "prompt", enabled: 0, gateMode: "advisory" },
]); ]);
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
db.close(); db.close();
}); });

View File

@@ -290,7 +290,7 @@ describe("Database", () => {
}); });
it("seeds schema version", () => { it("seeds schema version", () => {
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
}); });
it("includes tokenUsageCacheWriteTokens on freshly initialized tasks table", () => { it("includes tokenUsageCacheWriteTokens on freshly initialized tasks table", () => {
@@ -318,7 +318,7 @@ describe("Database", () => {
it("is idempotent - calling init() twice does not fail", () => { it("is idempotent - calling init() twice does not fail", () => {
expect(() => db.init()).not.toThrow(); expect(() => db.init()).not.toThrow();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
}); });
it("does not overwrite existing config on re-init", () => { it("does not overwrite existing config on re-init", () => {
// Update the config // Update the config
@@ -1383,7 +1383,7 @@ describe("schema migrations", () => {
db.init(); db.init();
// Verify version bumped to 29 (includes v1→v2 through v26→v29) // Verify version bumped to 29 (includes v1→v2 through v26→v29)
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
// Verify new columns exist and existing data is intact // Verify new columns exist and existing data is intact
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
@@ -1408,11 +1408,11 @@ describe("schema migrations", () => {
const db = new Database(fusionDir); const db = new Database(fusionDir);
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
// Re-init should not fail // Re-init should not fail
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
db.close(); db.close();
}); });
@@ -1447,7 +1447,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
expect(cols.map((col) => col.name)).toContain("priority"); expect(cols.map((col) => col.name)).toContain("priority");
@@ -1488,7 +1488,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const colNames = cols.map((col) => col.name); const colNames = cols.map((col) => col.name);
@@ -1560,7 +1560,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const colNames = cols.map((col) => col.name); const colNames = cols.map((col) => col.name);
@@ -1800,7 +1800,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
const cols = db.prepare("PRAGMA table_info(chat_messages)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(chat_messages)").all() as Array<{ name: string }>;
expect(cols.map((col) => col.name)).toContain("attachments"); expect(cols.map((col) => col.name)).toContain("attachments");
@@ -1874,7 +1874,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'agentRatings'").all() as Array<{ name: string }>; const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'agentRatings'").all() as Array<{ name: string }>;
expect(tables).toEqual([{ name: "agentRatings" }]); expect(tables).toEqual([{ name: "agentRatings" }]);
@@ -1898,7 +1898,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'mission_events'").all() as Array<{ name: string }>; const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'mission_events'").all() as Array<{ name: string }>;
expect(tables).toEqual([{ name: "mission_events" }]); expect(tables).toEqual([{ name: "mission_events" }]);
@@ -2002,7 +2002,7 @@ describe("schema migrations", () => {
db.init(); db.init();
// Verify version bumped to 29 // Verify version bumped to 29
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
// Verify new columns exist and existing data is intact // Verify new columns exist and existing data is intact
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
@@ -2221,7 +2221,7 @@ describe("schema migrations", () => {
localDb.init(); localDb.init();
expect(localDb.getSchemaVersion()).toBe(81); expect(localDb.getSchemaVersion()).toBe(82);
const columns = localDb.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const columns = localDb.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
expect(columns.map((column) => column.name)).toContain("tokenUsageCacheWriteTokens"); expect(columns.map((column) => column.name)).toContain("tokenUsageCacheWriteTokens");
@@ -2532,7 +2532,7 @@ describe("createDatabase factory", () => {
const db = createDatabase(fusionDir); const db = createDatabase(fusionDir);
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
expect(db.getLastModified()).toBeGreaterThan(0); expect(db.getLastModified()).toBeGreaterThan(0);
db.close(); db.close();
@@ -2688,7 +2688,7 @@ describe("migration v77 task token budget columns", () => {
migrated = new Database(fusion); migrated = new Database(fusion);
migrated.init(); migrated.init();
expect(migrated.getSchemaVersion()).toBe(81); expect(migrated.getSchemaVersion()).toBe(82);
const rows = migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const rows = migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const names = new Set(rows.map((row) => row.name)); const names = new Set(rows.map((row) => row.name));
expect(names.has("tokenBudgetSoftAlertedAt")).toBe(true); expect(names.has("tokenBudgetSoftAlertedAt")).toBe(true);
@@ -2734,7 +2734,7 @@ describe("migration v67 drops orphan project auth tables", () => {
migrated = new Database(fusion); migrated = new Database(fusion);
migrated.init(); migrated.init();
expect(migrated.getSchemaVersion()).toBe(81); expect(migrated.getSchemaVersion()).toBe(82);
const tables = migrated const tables = migrated
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'") .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'")
.all() as Array<{ name: string }>; .all() as Array<{ name: string }>;
@@ -2761,7 +2761,7 @@ describe("migration v67 drops orphan project auth tables", () => {
try { try {
fresh.init(); fresh.init();
expect(fresh.getSchemaVersion()).toBe(81); expect(fresh.getSchemaVersion()).toBe(82);
const tables = fresh const tables = fresh
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'") .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'")
.all() as Array<{ name: string }>; .all() as Array<{ name: string }>;

View File

@@ -886,7 +886,7 @@ describe("Migration: pre-33 DB upgrade", () => {
// Step 1: Create a fresh database at v33 (runs all migrations up to 33) // Step 1: Create a fresh database at v33 (runs all migrations up to 33)
const db1 = createDatabase(legacyDir); const db1 = createDatabase(legacyDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(81); expect(db1.getSchemaVersion()).toBe(82);
db1.close(); db1.close();
// Step 2: Manually downgrade to version 32 and drop insight tables // Step 2: Manually downgrade to version 32 and drop insight tables
@@ -921,7 +921,7 @@ describe("Migration: pre-33 DB upgrade", () => {
expect(tableNamesBefore).not.toContain("project_insight_runs"); expect(tableNamesBefore).not.toContain("project_insight_runs");
// Now run init — this triggers the v32→v33 migration // Now run init — this triggers the v32→v33 migration
db3.init(); db3.init();
expect(db3.getSchemaVersion()).toBe(81); expect(db3.getSchemaVersion()).toBe(82);
// Step 4: Verify insight tables exist after migration // Step 4: Verify insight tables exist after migration
const tablesAfter = db3.prepare( const tablesAfter = db3.prepare(
@@ -952,12 +952,12 @@ describe("Migration: pre-33 DB upgrade", () => {
try { try {
const db1 = createDatabase(testDir); const db1 = createDatabase(testDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(81); expect(db1.getSchemaVersion()).toBe(82);
db1.close(); db1.close();
const db2 = createDatabase(testDir); const db2 = createDatabase(testDir);
expect(() => db2.init()).not.toThrow(); expect(() => db2.init()).not.toThrow();
expect(db2.getSchemaVersion()).toBe(81); expect(db2.getSchemaVersion()).toBe(82);
db2.close(); db2.close();
} finally { } finally {
rmSync(testDir, { recursive: true, force: true }); rmSync(testDir, { recursive: true, force: true });
@@ -971,7 +971,7 @@ describe("Migration: pre-33 DB upgrade", () => {
// Step 1: Create a fresh DB and run migrations // Step 1: Create a fresh DB and run migrations
const db1 = createDatabase(compatDir); const db1 = createDatabase(compatDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(81); expect(db1.getSchemaVersion()).toBe(82);
// Step 2: Strip lifecycle and cancelledAt columns by recreating the // Step 2: Strip lifecycle and cancelledAt columns by recreating the
// table without them. This simulates a DB that was created before the // table without them. This simulates a DB that was created before the

View File

@@ -2774,7 +2774,7 @@ describe("MissionStore", () => {
describe("Loop State & Validator Run Schema (v31)", () => { describe("Loop State & Validator Run Schema (v31)", () => {
it("schema version is 40 after migration", () => { it("schema version is 40 after migration", () => {
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
}); });
it("mission_features table has loop state columns", () => { it("mission_features table has loop state columns", () => {

View File

@@ -584,7 +584,7 @@ describe("Run Audit", () => {
}); });
it("schema version is bumped to 40", () => { it("schema version is bumped to 40", () => {
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
}); });
}); });
}); });

View File

@@ -51,7 +51,7 @@ describe("TaskStore task documents", () => {
expect(tableNames.has("task_documents")).toBe(true); expect(tableNames.has("task_documents")).toBe(true);
expect(tableNames.has("task_document_revisions")).toBe(true); expect(tableNames.has("task_document_revisions")).toBe(true);
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
const index = db const index = db
.prepare( .prepare(

View File

@@ -103,6 +103,7 @@ export type DatabaseMutationType =
| "task:auto-recover-finalize-already-on-main" | "task:auto-recover-finalize-already-on-main"
| "task:auto-recover-branch-misbound" | "task:auto-recover-branch-misbound"
| "task:auto-recover-completion-fanout" | "task:auto-recover-completion-fanout"
| "task:auto-recover-worktree-session-exhausted"
| "auto-recovery:classify-decision" | "auto-recovery:classify-decision"
| "auto-recovery:retry-issued" | "auto-recovery:retry-issued"
| "auto-recovery:ai-session-spawned" | "auto-recovery:ai-session-spawned"

View File

@@ -744,7 +744,7 @@ describe("RoadmapStore", () => {
describe("schema version", () => { describe("schema version", () => {
it("schema version is 79 after init", () => { it("schema version is 79 after init", () => {
expect(db.getSchemaVersion()).toBe(81); expect(db.getSchemaVersion()).toBe(82);
}); });
}); });