FN-6281: add bounded graph resume retries

Add a narrow persisted retry path for transient workflow-graph failures during resume after restart or unpause.

- Persist graphResumeRetryCount with schema migration, store round-tripping, manual retry reset, and task serialization coverage.
- Retry only no-progress execute-seam graph failures immediately following resume markers, then clear transient status/error and re-run within a capped budget.
- Preserve terminal graph failure handling for explicit reasons, durable task failures, completed progress, non-resume cases, and exhausted retry budgets.
- Document the resume-limbo retry contract and add a patch changeset for @runfusion/fusion.

Files changed:
 .changeset/fn-6281-graph-resume-retry.md           |   5 +
 docs/architecture.md                               |   1 +
 packages/core/src/__tests__/db-migrate.test.ts     |  30 +--
 packages/core/src/__tests__/db.test.ts             |  44 ++---
 packages/core/src/__tests__/goals-schema.test.ts   |   2 +-
 packages/core/src/__tests__/insight-store.test.ts  |  10 +-
 .../core/src/__tests__/manual-retry-reset.test.ts  |   1 +
 .../src/__tests__/merge-request-record.test.ts     |   2 +-
 packages/core/src/__tests__/mission-store.test.ts  |   2 +-
 packages/core/src/__tests__/run-audit.test.ts      |   4 +-
 .../core/src/__tests__/store-merge-queue.test.ts   |   2 +-
 .../core/src/__tests__/store-persistence.test.ts   |  22 +++
 packages/core/src/__tests__/task-documents.test.ts |   2 +-
 packages/core/src/db.ts                            |  10 +-
 packages/core/src/manual-retry-reset.ts            |   1 +
 packages/core/src/store.ts                         |  14 +-
 packages/core/src/types.ts                         |   5 +
 .../engine/src/__tests__/executor-recovery.test.ts | 201 +++++++++++++++++++++
 packages/engine/src/executor.ts                    |  54 ++++++
 19 files changed, 359 insertions(+), 53 deletions(-)

Fusion-Task-Id: FN-6281

Fusion-Task-Lineage: f9061445-8624-465a-9255-c3d02eac1bdb
This commit is contained in:
gsxdsm
2026-06-12 11:10:12 -07:00
parent 36ba1b9d68
commit 0897b2a0bf
19 changed files with 359 additions and 53 deletions

View File

@@ -0,0 +1,5 @@
---
"@runfusion/fusion": patch
---
Add a bounded persisted auto-retry for transient workflow-graph resume failures after engine restart or unpause, while preserving terminal failures for genuine graph errors.

View File

@@ -1269,6 +1269,7 @@ The columns/traits track moved *board* policy (transitions, capacity, hold, merg
- A `parse-steps` node reads a workflow-declared **artifact** (PROMPT.md is just the default workflow's declared `step-source` artifact) and runs a registry **parser** (`step-headings`, `json-steps`, or a plugin-contributed parser) to write `Task.steps[]`. It is the only graph-side step-list writer and must dominate any `foreach`. Parsers fail closed to a routable `outcome:parse-error`. - A `parse-steps` node reads a workflow-declared **artifact** (PROMPT.md is just the default workflow's declared `step-source` artifact) and runs a registry **parser** (`step-headings`, `json-steps`, or a plugin-contributed parser) to write `Task.steps[]`. It is the only graph-side step-list writer and must dominate any `foreach`. Parsers fail closed to a routable `outcome:parse-error`.
- A `foreach(source:"task-steps")` node instantiates an inline template subgraph once per planned step, with `mode` (sequential/parallel) and `isolation` (shared/worktree) as explicit axes and per-instance run-state pinned + persisted for crash-safe resume. - A `foreach(source:"task-steps")` node instantiates an inline template subgraph once per planned step, with `mode` (sequential/parallel) and `isolation` (shared/worktree) as explicit axes and per-instance run-state pinned + persisted for crash-safe resume.
- Resume-limbo graph failures are retried only through a narrow persisted counter (`Task.graphResumeRetryCount`, max 2). The executor classifies a failure as transient only when it happens immediately after the engine restart/unpause resume log marker, reports no graph `reason`, has no completed step progress, and the task has no durable `lastError`/`failureReason`; it clears transient `status`/`error`, logs the auto-retry, and schedules one more graph execution. Any explicit graph reason, completed step progress, durable task error, missing resume marker, or exhausted counter remains a genuine `status:"failed"` disposition and goes to review handoff, preserving the FN-5704 anti-loop contract.
- A `step-review` node surfaces reviewer verdicts (APPROVE/REVISE/RETHINK/UNAVAILABLE) as outcome edges; `rework` edges (the only legal graph cycles, bounded per instance) route REVISE/RETHINK back to `step-execute`, with RETHINK traversal triggering the reset seam. - A `step-review` node surfaces reviewer verdicts (APPROVE/REVISE/RETHINK/UNAVAILABLE) as outcome edges; `rework` edges (the only legal graph cycles, bounded per instance) route REVISE/RETHINK back to `step-execute`, with RETHINK traversal triggering the reset seam.
- A `code` node runs sandboxed TypeScript (esbuild + child process, clamped timeout, no store handle) for arbitrary computed routing/field logic — the same trust tier as project-local script steps. - A `code` node runs sandboxed TypeScript (esbuild + child process, clamped timeout, no store handle) for arbitrary computed routing/field logic — the same trust tier as project-local script steps.

View File

@@ -715,7 +715,7 @@ describe("schema migration", () => {
const row = db.prepare("SELECT deletedAt FROM tasks WHERE id = 'FN-legacy'").get() as { deletedAt: string | null }; const row = db.prepare("SELECT deletedAt FROM tasks WHERE id = 'FN-legacy'").get() as { deletedAt: string | null };
expect(row.deletedAt).toBeNull(); expect(row.deletedAt).toBeNull();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -748,7 +748,7 @@ describe("schema migration", () => {
{ id: "WS-001", mode: "prompt", gateMode: "advisory" }, { id: "WS-001", mode: "prompt", gateMode: "advisory" },
{ id: "WS-002", mode: "script", gateMode: "advisory" }, { id: "WS-002", mode: "script", gateMode: "advisory" },
]); ]);
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -798,7 +798,7 @@ describe("schema migration", () => {
reviewerContextRetryCount: 0, reviewerContextRetryCount: 0,
reviewerFallbackRetryCount: 0, reviewerFallbackRetryCount: 0,
}); });
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -827,7 +827,7 @@ describe("schema migration", () => {
const columns = db.prepare("PRAGMA table_info(milestones)").all() as Array<{ name: string }>; const columns = db.prepare("PRAGMA table_info(milestones)").all() as Array<{ name: string }>;
expect(columns.map((column) => column.name)).toContain("acceptanceCriteria"); expect(columns.map((column) => column.name)).toContain("acceptanceCriteria");
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -868,7 +868,7 @@ describe("schema migration", () => {
const missionColumns = db.prepare("PRAGMA table_info(missions)").all() as Array<{ name: string }>; const missionColumns = db.prepare("PRAGMA table_info(missions)").all() as Array<{ name: string }>;
expect(missionColumns.map((column) => column.name)).toContain("autoMerge"); expect(missionColumns.map((column) => column.name)).toContain("autoMerge");
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -902,7 +902,7 @@ describe("schema migration", () => {
{ id: "WS-002", mode: "script", enabled: 1, gateMode: "advisory" }, { id: "WS-002", mode: "script", enabled: 1, gateMode: "advisory" },
{ id: "WS-003", mode: "prompt", enabled: 0, gateMode: "advisory" }, { id: "WS-003", mode: "prompt", enabled: 0, gateMode: "advisory" },
]); ]);
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -939,7 +939,7 @@ describe("schema migration", () => {
const indexes = db.prepare("PRAGMA index_list(mission_goals)").all() as Array<{ name: string }>; const indexes = db.prepare("PRAGMA index_list(mission_goals)").all() as Array<{ name: string }>;
expect(indexes.some((index) => index.name === "idxMissionGoalsGoalId")).toBe(true); expect(indexes.some((index) => index.name === "idxMissionGoalsGoalId")).toBe(true);
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -1000,7 +1000,7 @@ describe("schema migration", () => {
expect(customFieldsColumn).toBeDefined(); expect(customFieldsColumn).toBeDefined();
expect(customFieldsColumn?.dflt_value).toBe("'{}'"); expect(customFieldsColumn?.dflt_value).toBe("'{}'");
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -1038,7 +1038,7 @@ describe("schema migration", () => {
const indexes = db.prepare("PRAGMA index_list(workflow_settings)").all() as Array<{ name: string }>; const indexes = db.prepare("PRAGMA index_list(workflow_settings)").all() as Array<{ name: string }>;
expect(indexes.some((index) => index.name === "idx_workflow_settings_project")).toBe(true); expect(indexes.some((index) => index.name === "idx_workflow_settings_project")).toBe(true);
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -1120,7 +1120,7 @@ describe("schema migration", () => {
expect(indexNames).toContain("idx_cli_sessions_chatSessionId"); expect(indexNames).toContain("idx_cli_sessions_chatSessionId");
expect(indexNames).toContain("idx_cli_sessions_project_state"); expect(indexNames).toContain("idx_cli_sessions_project_state");
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -1152,7 +1152,7 @@ describe("schema migration", () => {
.all() as Array<{ name: string }>; .all() as Array<{ name: string }>;
expect(columns.map((column) => column.name)).toContain("cliExecutorAdapterId"); expect(columns.map((column) => column.name)).toContain("cliExecutorAdapterId");
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -1162,7 +1162,7 @@ describe("schema migration", () => {
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table'").all() as Array<{ name: string }>; const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table'").all() as Array<{ name: string }>;
expect(tables.map((row) => row.name)).toContain("cli_sessions"); expect(tables.map((row) => row.name)).toContain("cli_sessions");
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -1219,20 +1219,20 @@ describe("schema migration", () => {
.get() as { migrated_fragment_id: string | null }; .get() as { migrated_fragment_id: string | null };
expect(stepRow.migrated_fragment_id).toBeNull(); expect(stepRow.migrated_fragment_id).toBeNull();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
it("migration 109 is idempotent on re-init", () => { it("migration 109 is idempotent on re-init", () => {
const db = new Database(fusionDir); const db = new Database(fusionDir);
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
// Re-open the same on-disk DB: already at 109, the 109 block must be a no-op. // Re-open the same on-disk DB: already at 109, the 109 block must be a no-op.
const reopened = new Database(fusionDir); const reopened = new Database(fusionDir);
reopened.init(); reopened.init();
expect(reopened.getSchemaVersion()).toBe(115); expect(reopened.getSchemaVersion()).toBe(116);
const workflowColumns = reopened.prepare("PRAGMA table_info(workflows)").all() as Array<{ name: string }>; const workflowColumns = reopened.prepare("PRAGMA table_info(workflows)").all() as Array<{ name: string }>;
expect(workflowColumns.filter((c) => c.name === "kind")).toHaveLength(1); expect(workflowColumns.filter((c) => c.name === "kind")).toHaveLength(1);
const stepColumns = reopened.prepare("PRAGMA table_info(workflow_steps)").all() as Array<{ name: string }>; const stepColumns = reopened.prepare("PRAGMA table_info(workflow_steps)").all() as Array<{ name: string }>;

View File

@@ -334,7 +334,7 @@ describe("Database", () => {
}); });
it("seeds schema version", () => { it("seeds schema version", () => {
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
}); });
it("includes tokenUsageCacheWriteTokens on freshly initialized tasks table", () => { it("includes tokenUsageCacheWriteTokens on freshly initialized tasks table", () => {
@@ -393,7 +393,7 @@ describe("Database", () => {
it("is idempotent - calling init() twice does not fail", () => { it("is idempotent - calling init() twice does not fail", () => {
expect(() => db.init()).not.toThrow(); expect(() => db.init()).not.toThrow();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
}); });
it("does not overwrite existing config on re-init", () => { it("does not overwrite existing config on re-init", () => {
// Update the config // Update the config
@@ -1463,7 +1463,7 @@ describe("schema migrations", () => {
db.init(); db.init();
// Verify version bumped to 29 (includes v1→v2 through v26→v29) // Verify version bumped to 29 (includes v1→v2 through v26→v29)
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
// Verify new columns exist and existing data is intact // Verify new columns exist and existing data is intact
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
@@ -1488,15 +1488,15 @@ describe("schema migrations", () => {
const db = new Database(fusionDir); const db = new Database(fusionDir);
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
// Re-init should not fail // Re-init should not fail
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
// Re-init should not fail // Re-init should not fail
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
db.close(); db.close();
}); });
@@ -1531,7 +1531,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
expect(cols.map((col) => col.name)).toContain("priority"); expect(cols.map((col) => col.name)).toContain("priority");
@@ -1572,7 +1572,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const colNames = cols.map((col) => col.name); const colNames = cols.map((col) => col.name);
@@ -1644,7 +1644,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const colNames = cols.map((col) => col.name); const colNames = cols.map((col) => col.name);
@@ -1884,7 +1884,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
const cols = db.prepare("PRAGMA table_info(chat_messages)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(chat_messages)").all() as Array<{ name: string }>;
expect(cols.map((col) => col.name)).toContain("attachments"); expect(cols.map((col) => col.name)).toContain("attachments");
@@ -1958,7 +1958,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'agentRatings'").all() as Array<{ name: string }>; const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'agentRatings'").all() as Array<{ name: string }>;
expect(tables).toEqual([{ name: "agentRatings" }]); expect(tables).toEqual([{ name: "agentRatings" }]);
@@ -1982,7 +1982,7 @@ describe("schema migrations", () => {
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'mission_events'").all() as Array<{ name: string }>; const tables = db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'mission_events'").all() as Array<{ name: string }>;
expect(tables).toEqual([{ name: "mission_events" }]); expect(tables).toEqual([{ name: "mission_events" }]);
@@ -2086,7 +2086,7 @@ describe("schema migrations", () => {
db.init(); db.init();
// Verify version bumped to 29 // Verify version bumped to 29
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
// Verify new columns exist and existing data is intact // Verify new columns exist and existing data is intact
const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const cols = db.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
@@ -2305,7 +2305,7 @@ describe("schema migrations", () => {
localDb.init(); localDb.init();
expect(localDb.getSchemaVersion()).toBe(115); expect(localDb.getSchemaVersion()).toBe(116);
const columns = localDb.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const columns = localDb.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
expect(columns.map((column) => column.name)).toContain("tokenUsageCacheWriteTokens"); expect(columns.map((column) => column.name)).toContain("tokenUsageCacheWriteTokens");
@@ -2616,7 +2616,7 @@ describe("createDatabase factory", () => {
const db = createDatabase(fusionDir); const db = createDatabase(fusionDir);
db.init(); db.init();
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
expect(db.getLastModified()).toBeGreaterThan(0); expect(db.getLastModified()).toBeGreaterThan(0);
db.close(); db.close();
@@ -2770,7 +2770,7 @@ describe("migration v77 task token budget columns", () => {
migrated = new Database(fusion); migrated = new Database(fusion);
migrated.init(); migrated.init();
expect(migrated.getSchemaVersion()).toBe(115); expect(migrated.getSchemaVersion()).toBe(116);
const rows = migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>; const rows = migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>;
const names = new Set(rows.map((row) => row.name)); const names = new Set(rows.map((row) => row.name));
expect(names.has("tokenBudgetSoftAlertedAt")).toBe(true); expect(names.has("tokenBudgetSoftAlertedAt")).toBe(true);
@@ -2801,7 +2801,7 @@ describe("migration v106 adds tasks.transitionPending (FN-1417)", () => {
const fresh = new Database(fusion); const fresh = new Database(fusion);
try { try {
fresh.init(); fresh.init();
expect(fresh.getSchemaVersion()).toBe(115); expect(fresh.getSchemaVersion()).toBe(116);
const names = new Set( const names = new Set(
(fresh.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>).map((r) => r.name), (fresh.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>).map((r) => r.name),
); );
@@ -2829,7 +2829,7 @@ describe("migration v106 adds tasks.transitionPending (FN-1417)", () => {
migrated = new Database(fusion); migrated = new Database(fusion);
migrated.init(); migrated.init();
expect(migrated.getSchemaVersion()).toBe(115); expect(migrated.getSchemaVersion()).toBe(116);
const names = new Set( const names = new Set(
(migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>).map((r) => r.name), (migrated.prepare("PRAGMA table_info(tasks)").all() as Array<{ name: string }>).map((r) => r.name),
); );
@@ -2855,7 +2855,7 @@ describe("migration v107 adds workflow_run_branches + index (FN-1417)", () => {
const fresh = new Database(fusion); const fresh = new Database(fusion);
try { try {
fresh.init(); fresh.init();
expect(fresh.getSchemaVersion()).toBe(115); expect(fresh.getSchemaVersion()).toBe(116);
const table = fresh const table = fresh
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'workflow_run_branches'") .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'workflow_run_branches'")
.get() as { name: string } | undefined; .get() as { name: string } | undefined;
@@ -2889,7 +2889,7 @@ describe("migration v107 adds workflow_run_branches + index (FN-1417)", () => {
migrated = new Database(fusion); migrated = new Database(fusion);
migrated.init(); migrated.init();
expect(migrated.getSchemaVersion()).toBe(115); expect(migrated.getSchemaVersion()).toBe(116);
const table = migrated const table = migrated
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'workflow_run_branches'") .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name = 'workflow_run_branches'")
.get() as { name: string } | undefined; .get() as { name: string } | undefined;
@@ -2930,7 +2930,7 @@ describe("migration v67 drops orphan project auth tables", () => {
migrated = new Database(fusion); migrated = new Database(fusion);
migrated.init(); migrated.init();
expect(migrated.getSchemaVersion()).toBe(115); expect(migrated.getSchemaVersion()).toBe(116);
const tables = migrated const tables = migrated
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'") .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'")
.all() as Array<{ name: string }>; .all() as Array<{ name: string }>;
@@ -2957,7 +2957,7 @@ describe("migration v67 drops orphan project auth tables", () => {
try { try {
fresh.init(); fresh.init();
expect(fresh.getSchemaVersion()).toBe(115); expect(fresh.getSchemaVersion()).toBe(116);
const tables = fresh const tables = fresh
.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'") .prepare("SELECT name FROM sqlite_master WHERE type='table' AND name LIKE 'project_auth_%'")
.all() as Array<{ name: string }>; .all() as Array<{ name: string }>;

View File

@@ -91,6 +91,6 @@ describe("goals schema", () => {
}); });
it("reports schema version 101", () => { it("reports schema version 101", () => {
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
}); });
}); });

View File

@@ -1000,7 +1000,7 @@ describe("Migration: pre-33 DB upgrade", () => {
// Step 1: Create a fresh database at v33 (runs all migrations up to 33) // Step 1: Create a fresh database at v33 (runs all migrations up to 33)
const db1 = createDatabase(legacyDir); const db1 = createDatabase(legacyDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(115); expect(db1.getSchemaVersion()).toBe(116);
db1.close(); db1.close();
// Step 2: Manually downgrade to version 32 and drop insight tables // Step 2: Manually downgrade to version 32 and drop insight tables
@@ -1035,7 +1035,7 @@ describe("Migration: pre-33 DB upgrade", () => {
expect(tableNamesBefore).not.toContain("project_insight_runs"); expect(tableNamesBefore).not.toContain("project_insight_runs");
// Now run init — this triggers the v32→v33 migration // Now run init — this triggers the v32→v33 migration
db3.init(); db3.init();
expect(db3.getSchemaVersion()).toBe(115); expect(db3.getSchemaVersion()).toBe(116);
// Step 4: Verify insight tables exist after migration // Step 4: Verify insight tables exist after migration
const tablesAfter = db3.prepare( const tablesAfter = db3.prepare(
@@ -1066,12 +1066,12 @@ describe("Migration: pre-33 DB upgrade", () => {
try { try {
const db1 = createDatabase(testDir); const db1 = createDatabase(testDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(115); expect(db1.getSchemaVersion()).toBe(116);
db1.close(); db1.close();
const db2 = createDatabase(testDir); const db2 = createDatabase(testDir);
expect(() => db2.init()).not.toThrow(); expect(() => db2.init()).not.toThrow();
expect(db2.getSchemaVersion()).toBe(115); expect(db2.getSchemaVersion()).toBe(116);
db2.close(); db2.close();
} finally { } finally {
rmSync(testDir, { recursive: true, force: true }); rmSync(testDir, { recursive: true, force: true });
@@ -1085,7 +1085,7 @@ describe("Migration: pre-33 DB upgrade", () => {
// Step 1: Create a fresh DB and run migrations // Step 1: Create a fresh DB and run migrations
const db1 = createDatabase(compatDir); const db1 = createDatabase(compatDir);
db1.init(); db1.init();
expect(db1.getSchemaVersion()).toBe(115); expect(db1.getSchemaVersion()).toBe(116);
// Step 2: Strip lifecycle and cancelledAt columns by recreating the // Step 2: Strip lifecycle and cancelledAt columns by recreating the
// table without them. This simulates a DB that was created before the // table without them. This simulates a DB that was created before the

View File

@@ -53,6 +53,7 @@ describe("buildManualRetryResetPatch", () => {
for (const key of MANUAL_RETRY_RESET_COUNTER_KEYS) { for (const key of MANUAL_RETRY_RESET_COUNTER_KEYS) {
expect(patch[key]).toBe(0); expect(patch[key]).toBe(0);
} }
expect(patch.graphResumeRetryCount).toBe(0);
}); });
it("includes all retry-summary counters in the reset key list", () => { it("includes all retry-summary counters in the reset key list", () => {

View File

@@ -38,7 +38,7 @@ describe("TaskStore merge request record + completion handoff marker", () => {
.all() as Array<{ name: string }>; .all() as Array<{ name: string }>;
expect(tableRows).toEqual([{ name: "completion_handoff_markers" }, { name: "merge_requests" }]); expect(tableRows).toEqual([{ name: "completion_handoff_markers" }, { name: "merge_requests" }]);
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
}); });
it("upserts merge request records", async () => { it("upserts merge request records", async () => {

View File

@@ -3746,7 +3746,7 @@ describe("MissionStore", () => {
describe("Loop State & Validator Run Schema (v31)", () => { describe("Loop State & Validator Run Schema (v31)", () => {
it("schema version is 101 after migration", () => { it("schema version is 101 after migration", () => {
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
}); });
it("mission_features table has loop state columns", () => { it("mission_features table has loop state columns", () => {

View File

@@ -583,8 +583,8 @@ describe("Run Audit", () => {
expect(indexNames).toContain("idxRunAuditEventsTimestamp"); expect(indexNames).toContain("idxRunAuditEventsTimestamp");
}); });
it("schema version is bumped to 115", () => { it("schema version is bumped to 116", () => {
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
}); });
}); });
}); });

View File

@@ -60,7 +60,7 @@ describe("TaskStore merge queue", () => {
expect.arrayContaining(["idx_mergeQueue_lease_ready", "idx_mergeQueue_leaseExpiresAt"]), expect.arrayContaining(["idx_mergeQueue_lease_ready", "idx_mergeQueue_leaseExpiresAt"]),
); );
expect(store.getDatabase().getSchemaVersion()).toBe(115); expect(store.getDatabase().getSchemaVersion()).toBe(116);
}); });
it("migrates a legacy v88 database and preserves task rows", async () => { it("migrates a legacy v88 database and preserves task rows", async () => {

View File

@@ -59,6 +59,28 @@ describe("TaskStore", () => {
}); });
}); });
describe("graphResumeRetryCount persistence", () => {
it("defaults to zero and round-trips updateTask values", async () => {
const task = await harness.store().createTask({ description: "Graph retry counter task" });
expect((await harness.store().getTask(task.id)).graphResumeRetryCount).toBe(0);
const updated = await harness.store().updateTask(task.id, { graphResumeRetryCount: 2 });
expect(updated.graphResumeRetryCount).toBe(2);
expect((await harness.store().getTask(task.id)).graphResumeRetryCount).toBe(2);
});
it("clears graphResumeRetryCount with null", async () => {
const task = await harness.store().createTask({ description: "Graph retry clear task" });
await harness.store().updateTask(task.id, { graphResumeRetryCount: 1 });
const cleared = await harness.store().updateTask(task.id, { graphResumeRetryCount: null });
expect(cleared.graphResumeRetryCount).toBeNull();
expect((await harness.store().getTask(task.id)).graphResumeRetryCount).toBeUndefined();
});
});
describe("agent taskId sync on reassignment", () => { describe("agent taskId sync on reassignment", () => {
it("reassignment clears the old agent taskId and sets the new agent taskId", async () => { it("reassignment clears the old agent taskId and sets the new agent taskId", async () => {
harness.store().close(); harness.store().close();

View File

@@ -51,7 +51,7 @@ describe("TaskStore task documents", () => {
expect(tableNames.has("task_documents")).toBe(true); expect(tableNames.has("task_documents")).toBe(true);
expect(tableNames.has("task_document_revisions")).toBe(true); expect(tableNames.has("task_document_revisions")).toBe(true);
expect(db.getSchemaVersion()).toBe(115); expect(db.getSchemaVersion()).toBe(116);
const index = db const index = db
.prepare( .prepare(

View File

@@ -162,7 +162,7 @@ export function isFts5CorruptionError(error: unknown): boolean {
// ── Schema Definition ──────────────────────────────────────────────── // ── Schema Definition ────────────────────────────────────────────────
const SCHEMA_VERSION = 115; const SCHEMA_VERSION = 116;
const TASKS_FTS_AUTOMERGE = 8; const TASKS_FTS_AUTOMERGE = 8;
const TASKS_FTS_CRISISMERGE = 16; const TASKS_FTS_CRISISMERGE = 16;
@@ -262,6 +262,7 @@ CREATE TABLE IF NOT EXISTS tasks (
mergeRetries INTEGER, mergeRetries INTEGER,
workflowStepRetries INTEGER, workflowStepRetries INTEGER,
resumeLimboCount INTEGER DEFAULT 0, resumeLimboCount INTEGER DEFAULT 0,
graphResumeRetryCount INTEGER DEFAULT 0,
resumeLimboTipSha TEXT, resumeLimboTipSha TEXT,
resumeLimboStepSignature TEXT, resumeLimboStepSignature TEXT,
recoveryRetryCount INTEGER, recoveryRetryCount INTEGER,
@@ -4689,6 +4690,13 @@ export class Database {
}); });
} }
// Migration 116: Bounded transient resume-after-restart graph retries.
if (version < 116) {
this.applyMigration(116, () => {
this.addColumnIfMissing("tasks", "graphResumeRetryCount", "INTEGER DEFAULT 0");
});
}
} }
/** /**

View File

@@ -5,6 +5,7 @@ export const IN_REVIEW_STALL_DEADLOCK_PAUSE_REASON = "in-review-stall-deadlock";
export const MANUAL_RETRY_RESET_COUNTER_KEYS = [ export const MANUAL_RETRY_RESET_COUNTER_KEYS = [
"stuckKillCount", "stuckKillCount",
"resumeLimboCount", "resumeLimboCount",
"graphResumeRetryCount",
"recoveryRetryCount", "recoveryRetryCount",
"taskDoneRetryCount", "taskDoneRetryCount",
"worktreeSessionRetryCount", "worktreeSessionRetryCount",

View File

@@ -203,6 +203,7 @@ interface TaskRow {
workflowStepRetries: number | null; workflowStepRetries: number | null;
stuckKillCount: number | null; stuckKillCount: number | null;
resumeLimboCount: number | null; resumeLimboCount: number | null;
graphResumeRetryCount: number | null;
resumeLimboTipSha: string | null; resumeLimboTipSha: string | null;
resumeLimboStepSignature: string | null; resumeLimboStepSignature: string | null;
postReviewFixCount: number | null; postReviewFixCount: number | null;
@@ -348,6 +349,7 @@ const TASK_COLUMN_DESCRIPTORS: TaskColumnDescriptor[] = [
defineTaskColumn("workflowStepRetries", (task) => task.workflowStepRetries ?? null), defineTaskColumn("workflowStepRetries", (task) => task.workflowStepRetries ?? null),
defineTaskColumn("stuckKillCount", (task) => task.stuckKillCount ?? 0), defineTaskColumn("stuckKillCount", (task) => task.stuckKillCount ?? 0),
defineTaskColumn("resumeLimboCount", (task) => task.resumeLimboCount ?? 0), defineTaskColumn("resumeLimboCount", (task) => task.resumeLimboCount ?? 0),
defineTaskColumn("graphResumeRetryCount", (task) => task.graphResumeRetryCount === undefined ? 0 : task.graphResumeRetryCount),
defineTaskColumn("resumeLimboTipSha", (task) => task.resumeLimboTipSha ?? null), defineTaskColumn("resumeLimboTipSha", (task) => task.resumeLimboTipSha ?? null),
defineTaskColumn("resumeLimboStepSignature", (task) => task.resumeLimboStepSignature ?? null), defineTaskColumn("resumeLimboStepSignature", (task) => task.resumeLimboStepSignature ?? null),
defineTaskColumn("postReviewFixCount", (task) => task.postReviewFixCount ?? 0), defineTaskColumn("postReviewFixCount", (task) => task.postReviewFixCount ?? 0),
@@ -1933,6 +1935,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
workflowStepRetries: row.workflowStepRetries ?? undefined, workflowStepRetries: row.workflowStepRetries ?? undefined,
stuckKillCount: row.stuckKillCount ?? undefined, stuckKillCount: row.stuckKillCount ?? undefined,
resumeLimboCount: row.resumeLimboCount ?? undefined, resumeLimboCount: row.resumeLimboCount ?? undefined,
graphResumeRetryCount: row.graphResumeRetryCount ?? undefined,
resumeLimboTipSha: row.resumeLimboTipSha || undefined, resumeLimboTipSha: row.resumeLimboTipSha || undefined,
resumeLimboStepSignature: row.resumeLimboStepSignature || undefined, resumeLimboStepSignature: row.resumeLimboStepSignature || undefined,
postReviewFixCount: row.postReviewFixCount ?? undefined, postReviewFixCount: row.postReviewFixCount ?? undefined,
@@ -2447,7 +2450,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
"modelPresetId", "modelProvider", "modelId", "modelPresetId", "modelProvider", "modelId",
"validatorModelProvider", "validatorModelId", "validatorModelProvider", "validatorModelId",
"planningModelProvider", "planningModelId", "planningModelProvider", "planningModelId",
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
"error", "summary", "thinkingLevel", "executionMode", "error", "summary", "thinkingLevel", "executionMode",
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride", "tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt", "createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt",
@@ -2496,7 +2499,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
"modelPresetId", "modelProvider", "modelId", "modelPresetId", "modelProvider", "modelId",
"validatorModelProvider", "validatorModelId", "validatorModelProvider", "validatorModelId",
"planningModelProvider", "planningModelId", "planningModelProvider", "planningModelId",
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
"error", "summary", "thinkingLevel", "executionMode", "error", "summary", "thinkingLevel", "executionMode",
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride", "tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt", "createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt",
@@ -7498,7 +7501,7 @@ ${TASK_UPSERT_SQL_ASSIGNMENTS}
async updateTask( async updateTask(
id: string, id: string,
updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; customFields?: Record<string, unknown>; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record<string, unknown> | null; githubTracking?: import("./types.js").TaskGithubTracking | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null }, updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; customFields?: Record<string, unknown>; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; graphResumeRetryCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record<string, unknown> | null; githubTracking?: import("./types.js").TaskGithubTracking | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null },
runContext?: RunMutationContext, runContext?: RunMutationContext,
): Promise<Task> { ): Promise<Task> {
return this.withTaskLock(id, () => this.updateTaskUnlocked(id, updates, runContext)); return this.withTaskLock(id, () => this.updateTaskUnlocked(id, updates, runContext));
@@ -8087,6 +8090,11 @@ ${TASK_UPSERT_SQL_ASSIGNMENTS}
} else if (updates.resumeLimboCount !== undefined) { } else if (updates.resumeLimboCount !== undefined) {
task.resumeLimboCount = updates.resumeLimboCount; task.resumeLimboCount = updates.resumeLimboCount;
} }
if (updates.graphResumeRetryCount === null) {
task.graphResumeRetryCount = null;
} else if (updates.graphResumeRetryCount !== undefined) {
task.graphResumeRetryCount = updates.graphResumeRetryCount;
}
if (updates.resumeLimboTipSha === null) { if (updates.resumeLimboTipSha === null) {
task.resumeLimboTipSha = undefined; task.resumeLimboTipSha = undefined;
} else if (updates.resumeLimboTipSha !== undefined) { } else if (updates.resumeLimboTipSha !== undefined) {

View File

@@ -2265,6 +2265,11 @@ export interface Task {
* Incremented by self-healing for resume-limbo detection and reset when * Incremented by self-healing for resume-limbo detection and reset when
* progress is observed or recovery escalates to a fresh todo dispatch. */ * progress is observed or recovery escalates to a fresh todo dispatch. */
resumeLimboCount?: number; resumeLimboCount?: number;
/** Bounded auto-retry attempts for transient workflow-graph failures observed
* immediately after engine-restart or unpause resume. Reset by manual retry
* and by successful forward progress; capped by the executor before terminal
* `status:"failed"` is recorded to preserve the FN-5704 anti-loop exemption. */
graphResumeRetryCount?: number | null;
/** Branch tip SHA snapshot captured at the last reclaim/unpause attempt used /** Branch tip SHA snapshot captured at the last reclaim/unpause attempt used
* by resume-limbo detection to determine whether commits advanced. */ * by resume-limbo detection to determine whether commits advanced. */
resumeLimboTipSha?: string; resumeLimboTipSha?: string;

View File

@@ -698,6 +698,207 @@ describe("TaskExecutor bounded recovery retries", () => {
expect(store.handoffToReview).not.toHaveBeenCalled(); expect(store.handoffToReview).not.toHaveBeenCalled();
}); });
it("auto-retries a bounded transient resume-after-restart graph failure instead of parking", async () => {
const store = createMockStore();
const task = {
id: "FN-001",
title: "Test",
description: "Test",
column: "in-progress",
status: undefined,
dependencies: [],
steps: [{ name: "Step 1", status: "pending" }],
currentStep: 0,
log: [{ timestamp: new Date().toISOString(), action: "Resumed after engine restart" }],
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
graphResumeRetryCount: 0,
} as Task;
store.getTask.mockResolvedValue({ ...task, paused: false, error: null });
const executor = new TaskExecutor(store, "/tmp/test", {});
const executeSpy = vi.spyOn(executor as any, "execute").mockResolvedValue(undefined);
await (executor as any).handleGraphFailure(task, {
disposition: "failed",
outcome: "failure",
visitedNodeIds: ["execute"],
});
await new Promise((resolve) => setTimeout(resolve, 0));
expect(store.updateTask).toHaveBeenCalledWith(
"FN-001",
{ graphResumeRetryCount: 1, status: null, error: null },
undefined,
);
expect(store.updateTask).not.toHaveBeenCalledWith(
"FN-001",
expect.objectContaining({ status: "failed" }),
expect.anything(),
);
expect(store.handoffToReview).not.toHaveBeenCalled();
expect(executeSpy).toHaveBeenCalledWith(expect.objectContaining({ id: "FN-001" }));
});
it("auto-retries a bounded transient graph failure after unpause resume instead of parking", async () => {
const store = createMockStore();
const task = {
id: "FN-001",
title: "Test",
description: "Test",
column: "in-progress",
status: undefined,
dependencies: [],
steps: [{ name: "Step 1", status: "pending" }],
currentStep: 0,
log: [{ timestamp: new Date().toISOString(), action: "Resuming execution after unpause" }],
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
graphResumeRetryCount: 0,
} as Task;
store.getTask.mockResolvedValue({ ...task, paused: false, error: null });
const executor = new TaskExecutor(store, "/tmp/test", {});
const executeSpy = vi.spyOn(executor as any, "execute").mockResolvedValue(undefined);
await (executor as any).handleGraphFailure(task, {
disposition: "failed",
outcome: "failure",
visitedNodeIds: ["execute"],
});
await new Promise((resolve) => setTimeout(resolve, 0));
expect(store.updateTask).toHaveBeenCalledWith(
"FN-001",
{ graphResumeRetryCount: 1, status: null, error: null },
undefined,
);
expect(store.handoffToReview).not.toHaveBeenCalled();
expect(executeSpy).toHaveBeenCalledWith(expect.objectContaining({ id: "FN-001" }));
});
it("parks a transient resume graph failure once the retry budget is exhausted", async () => {
const store = createMockStore();
const task = {
id: "FN-001",
title: "Test",
description: "Test",
column: "in-progress",
status: undefined,
dependencies: [],
steps: [{ name: "Step 1", status: "pending" }],
currentStep: 0,
log: [{ timestamp: new Date().toISOString(), action: "Resumed after engine restart" }],
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
graphResumeRetryCount: 2,
} as Task;
store.getTask.mockResolvedValue({ ...task, paused: false, error: null });
const warnSpy = vi.spyOn(executorLog, "warn").mockImplementation(() => undefined);
const executor = new TaskExecutor(store, "/tmp/test", {});
const executeSpy = vi.spyOn(executor as any, "execute").mockResolvedValue(undefined);
await (executor as any).handleGraphFailure(task, {
disposition: "failed",
outcome: "failure",
visitedNodeIds: ["execute"],
});
const message = "Workflow graph terminated with failure at node 'execute'";
expect(store.updateTask).toHaveBeenCalledWith("FN-001", { error: message, status: "failed" }, undefined);
expect(store.handoffToReview).toHaveBeenCalledWith(
"FN-001",
expect.objectContaining({ evidence: expect.objectContaining({ reason: "workflow-graph-failed" }) }),
);
expect(executeSpy).not.toHaveBeenCalled();
warnSpy.mockRestore();
});
it.each([
["non-empty execute-seam reason", { result: { reason: "interpreter-error: boom", visitedNodeIds: ["execute"] } }],
["settings/workflow-selection reason before node progress", { result: { reason: "settings-load-failed: boom", visitedNodeIds: [] } }],
["completed step progress", { task: { steps: [{ name: "Step 1", status: "done" }] }, result: { visitedNodeIds: ["execute"] } }],
["lastError", { task: { lastError: "boom" }, result: { visitedNodeIds: ["execute"] } }],
["failureReason", { task: { failureReason: "boom" }, result: { visitedNodeIds: ["execute"] } }],
])("preserves terminal failed handling for genuine graph failure: %s", async (_name, fixture) => {
const store = createMockStore();
const task = {
id: "FN-001",
title: "Test",
description: "Test",
column: "in-progress",
status: undefined,
dependencies: [],
steps: [{ name: "Step 1", status: "pending" }],
currentStep: 0,
log: [{ timestamp: new Date().toISOString(), action: "Resumed after engine restart" }],
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
graphResumeRetryCount: 0,
...(fixture.task ?? {}),
} as Task;
store.getTask.mockResolvedValue({ ...task, paused: false, error: null });
const warnSpy = vi.spyOn(executorLog, "warn").mockImplementation(() => undefined);
const executor = new TaskExecutor(store, "/tmp/test", {});
const executeSpy = vi.spyOn(executor as any, "execute").mockResolvedValue(undefined);
await (executor as any).handleGraphFailure(task, {
disposition: "failed",
outcome: "failure",
...fixture.result,
});
const failedNode = fixture.result.visitedNodeIds.at(-1) ?? "unknown";
const message = `Workflow graph terminated with failure at node '${failedNode}'`;
expect(store.updateTask).toHaveBeenCalledWith("FN-001", { error: message, status: "failed" }, undefined);
expect(store.handoffToReview).toHaveBeenCalledWith(
"FN-001",
expect.objectContaining({ evidence: expect.objectContaining({ reason: "workflow-graph-failed" }) }),
);
expect(executeSpy).not.toHaveBeenCalled();
warnSpy.mockRestore();
});
describe("transient resume-after-restart graph failure classifier", () => {
const makeClassifierTask = (overrides: Partial<Task> = {}) => ({
id: "FN-001",
title: "Test",
description: "Test",
column: "in-progress",
status: undefined,
dependencies: [],
steps: [
{ name: "Step 1", status: "pending" },
{ name: "Step 2", status: "pending" },
],
currentStep: 0,
log: [{ timestamp: new Date().toISOString(), action: "Resumed after engine restart" }],
createdAt: new Date().toISOString(),
updatedAt: new Date().toISOString(),
...overrides,
}) as Task;
const isTransient = (task: Task, result: any) => {
const executor = new TaskExecutor(createMockStore(), "/tmp/test", {});
return (executor as any).isTransientResumeAfterRestartGraphFailure(task, result);
};
it("accepts only the exact no-progress execute-seam post-resume signature", () => {
expect(isTransient(makeClassifierTask(), { visitedNodeIds: ["execute"] })).toBe(true);
expect(isTransient(makeClassifierTask({ log: [{ timestamp: new Date().toISOString(), action: "Resuming execution after unpause" }] }), { visitedNodeIds: ["execute"] })).toBe(true);
expect(isTransient(makeClassifierTask(), { visitedNodeIds: [] })).toBe(true);
});
it.each([
["non-empty reason", makeClassifierTask(), { visitedNodeIds: ["execute"], reason: "settings-load-failed: boom" }],
["non-execute failed node", makeClassifierTask(), { visitedNodeIds: ["planning"] }],
["completed step progress", makeClassifierTask({ steps: [{ name: "Step 1", status: "done" }] }), { visitedNodeIds: ["execute"] }],
["lastError", makeClassifierTask({ lastError: "boom" } as any), { visitedNodeIds: ["execute"] }],
["failureReason", makeClassifierTask({ failureReason: "boom" } as any), { visitedNodeIds: ["execute"] }],
["missing resume log", makeClassifierTask({ log: [{ timestamp: new Date().toISOString(), action: "Started execution" }] }), { visitedNodeIds: ["execute"] }],
])("rejects %s as genuine/non-transient", (_name, task, result) => {
expect(isTransient(task as Task, result)).toBe(false);
});
});
it("preserves genuine in-progress graph failure handling", async () => { it("preserves genuine in-progress graph failure handling", async () => {
const store = createMockStore(); const store = createMockStore();
const task = { const task = {

View File

@@ -275,6 +275,13 @@ const MAX_WORKFLOW_STEP_RETRIES = 3;
const MAX_TASK_DONE_SESSION_RETRIES = 3; const MAX_TASK_DONE_SESSION_RETRIES = 3;
/** Maximum todo requeues after exhausting in-session fn_task_done retries. */ /** Maximum todo requeues after exhausting in-session fn_task_done retries. */
const MAX_TASK_DONE_REQUEUE_RETRIES = 3; const MAX_TASK_DONE_REQUEUE_RETRIES = 3;
/**
* Maximum bounded retries for the narrow resume-after-restart graph transient.
* Budget exhaustion falls through to terminal status:"failed" so FN-5704's
* self-healing anti-loop exemption remains intact for genuine graph failures.
*/
const MAX_TRANSIENT_GRAPH_RESUME_RETRIES = 2;
const TRANSIENT_GRAPH_RESUME_RETRY_BACKOFF_MS = process.env.VITEST || process.env.NODE_ENV === "test" ? 0 : 1_000;
/** How long to wait before recovering a completed task still stuck in in-progress. */ /** How long to wait before recovering a completed task still stuck in in-progress. */
const COMPLETED_TASK_WATCHDOG_MS = 60_000; const COMPLETED_TASK_WATCHDOG_MS = 60_000;
/** How long to wait before retrying a workflow rerun handoff that never reached in-progress. */ /** How long to wait before retrying a workflow rerun handoff that never reached in-progress. */
@@ -3833,6 +3840,11 @@ export class TaskExecutor {
} }
if (result.disposition === "failed") { if (result.disposition === "failed") {
await this.handleGraphFailure(task, result); await this.handleGraphFailure(task, result);
} else if (result.disposition === "completed") {
const live = await this.store.getTask(task.id).catch(() => task);
if ((live.graphResumeRetryCount ?? 0) !== 0) {
await this.store.updateTask(task.id, { graphResumeRetryCount: 0 }, this.getRunContextFor(task.id));
}
} }
return true; return true;
} finally { } finally {
@@ -5970,6 +5982,22 @@ export class TaskExecutor {
} }
} }
private isTransientResumeAfterRestartGraphFailure(live: Task, result: WorkflowGraphTaskRunResult): boolean {
if ((result.reason ?? "").trim().length > 0) return false;
const failedNode = result.visitedNodeIds[result.visitedNodeIds.length - 1];
if (failedNode !== undefined && failedNode !== "execute") return false;
if (live.steps.some((step) => step.status === "done")) return false;
const failureState = live as Task & { lastError?: unknown; failureReason?: unknown };
if (failureState.lastError != null || failureState.failureReason != null) return false;
const latestAction = live.log.at(-1)?.action;
return latestAction === "Resumed after engine restart"
|| latestAction === "Resuming execution after unpause";
}
/** Terminal failure of a graph run: record the error and park the task in /** Terminal failure of a graph run: record the error and park the task in
* review so a human can act — never leave it invisible in in-progress. */ * review so a human can act — never leave it invisible in in-progress. */
private async handleGraphFailure(task: Task, result: WorkflowGraphTaskRunResult): Promise<void> { private async handleGraphFailure(task: Task, result: WorkflowGraphTaskRunResult): Promise<void> {
@@ -5992,6 +6020,32 @@ export class TaskExecutor {
return; return;
} }
const failedNode = result.visitedNodeIds[result.visitedNodeIds.length - 1]; const failedNode = result.visitedNodeIds[result.visitedNodeIds.length - 1];
if (this.isTransientResumeAfterRestartGraphFailure(live, result)) {
const priorRetries = live.graphResumeRetryCount ?? 0;
if (priorRetries < MAX_TRANSIENT_GRAPH_RESUME_RETRIES) {
const nextRetries = priorRetries + 1;
const benignMessage = `Transient resume-after-restart graph failure — auto-retrying (${nextRetries}/${MAX_TRANSIENT_GRAPH_RESUME_RETRIES}) instead of parking`;
executorLog.warn(`${task.id}: ${benignMessage}`);
await this.store.logEntry(task.id, benignMessage, undefined, this.getRunContextFor(task.id));
await this.store.updateTask(task.id, {
graphResumeRetryCount: nextRetries,
status: null,
error: null,
}, this.getRunContextFor(task.id));
const scheduleRetry = () => {
this.execute(live).catch((err) =>
executorLog.error(`Failed transient graph resume retry for ${task.id}:`, err),
);
};
if (TRANSIENT_GRAPH_RESUME_RETRY_BACKOFF_MS > 0) {
const handle = setTimeout(scheduleRetry, TRANSIENT_GRAPH_RESUME_RETRY_BACKOFF_MS);
handle.unref?.();
} else {
setTimeout(scheduleRetry, 0).unref?.();
}
return;
}
}
const message = `Workflow graph terminated with failure at node '${failedNode ?? "unknown"}'`; const message = `Workflow graph terminated with failure at node '${failedNode ?? "unknown"}'`;
executorLog.warn(`${task.id}: ${message}`); executorLog.warn(`${task.id}: ${message}`);
await this.store.logEntry(task.id, message, undefined, this.getRunContextFor(task.id)); await this.store.logEntry(task.id, message, undefined, this.getRunContextFor(task.id));