diff --git a/.changeset/fn-7996-executor-tool-failure-retry.md b/.changeset/fn-7996-executor-tool-failure-retry.md new file mode 100644 index 0000000000..414c2e3c89 --- /dev/null +++ b/.changeset/fn-7996-executor-tool-failure-retry.md @@ -0,0 +1,7 @@ +--- +"@runfusion/fusion": minor +--- + +summary: Auto-retry executor tool-call failures before parking tasks. +category: feature +dev: Adds project-scoped bounded retry settings, durable PostgreSQL claim state, and same-model retry auditing. diff --git a/AGENTS.md b/AGENTS.md index 38c11cc322..5ea77a3f08 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -275,6 +275,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme - Workspace (Phase D U1): self-healing emits `task:reconcile-orphaned-workspace-worktree` when it removes a done/dead workspace task's recorded per-repo worktree from its stored `worktreePath` (guarded by `isPathActive`; no temp-root walk). - FN-7514: the planner overseer's per-task oversight loop (`PlannerRecoveryController.tick`) emits `overseer:oversight-withheld-human-control` when the pure `evaluateOverseerHumanControl` guard withholds ALL oversight action (no steering, retry, targeted-fix, or pending confirmation) for a task that is user-paused (`task.userPaused===true`, or `task.paused===true` with no `pausedReason`) or ineligible for auto-merge processing per `allowsAutoMergeProcessing` (`autoMerge:false`/PR-based human-review terminal contract). The guard runs BEFORE FN-7513's confirmation classification, so a withheld task never records a pending confirmation. Metadata: `{ taskId, reason: "user-paused" | "auto-merge-off-human-review", stage, oversightLevel }`; deduped per (taskId, withheld reason) so it is not re-emitted every poll while the reason is unchanged. - FN-7720: `TaskStore.bypassFailedPreMergeReviewStep` emits `task:bypass-review` when a privileged operator bypasses the latest failed pre-merge review step of an `in-review` task; metadata includes `workflowStepId`, `workflowStepName`, `bypassedFromStatus`, `bypassedFromVerdict`, and the mandatory `reason`. The bypass rewrites the step's `status` to `"skipped"` with `bypassedBy`/`bypassedAt`/`bypassReason`/`bypassedFromStatus` fields; it never fabricates a reviewer `verdict` and clears only the failed-pre-merge-step `getTaskMergeBlocker` reason. Reachable via `fn_task_bypass_review` (CLI/pi-extension operator tool surface only — not executor/reviewer/triage) and `POST /tasks/:id/bypass-review`. +- FN-7996: executor emits `task:execution-tool-failure-retry` for a claimed same-model consecutive-tool-failure retry and `task:execution-tool-failure-retry-exhausted` when the matching run budget is spent. Metadata is ids/counts/outcomes-only; the exhausted event is emitted once through a project-scoped compare-and-set while terminal parking remains idempotent. - FN-8004: `agent:heartbeat-move-skipped-soft-delete` records a heartbeat move that races a soft-deleted task without parking the durable agent. Metadata remains ids/timestamps/source only (`agentId`, optional `taskId`/`deletedAt`, `moveAttemptedAt`, optional `source`); it never stores error prose. diff --git a/docs/architecture.md b/docs/architecture.md index c2f14cfe96..87745c50a5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1311,6 +1311,7 @@ The columns/traits track moved *board* policy (transitions, capacity, hold, merg - A `parse-steps` node reads a workflow-declared **artifact** (PROMPT.md is just the default workflow's declared `step-source` artifact) and runs a registry **parser** (`step-headings`, `json-steps`, or a plugin-contributed parser) to write `Task.steps[]`. It is the only graph-side step-list writer and must dominate any `foreach`. Parsers fail closed to a routable `outcome:parse-error`. - A `foreach(source:"task-steps")` node instantiates an inline template subgraph once per planned step, with `mode` (sequential/parallel) and `isolation` (shared/worktree) as explicit axes and per-instance run-state pinned + persisted for crash-safe resume. - Resume-limbo graph failures are retried only through a narrow persisted counter (`Task.graphResumeRetryCount`, max 2). The executor classifies a failure as transient only when it happens immediately after the engine restart/unpause resume log marker, reports no graph `reason`, has no completed step progress, and the task has no durable `lastError`/`failureReason`; it clears transient `status`/`error`, logs the auto-retry, and schedules one more graph execution. Any explicit graph reason, completed step progress, durable task error, missing resume marker, or exhausted counter remains a genuine `status:"failed"` disposition and goes to review handoff, preserving the FN-5704 anti-loop contract. +- FN-7996 uses the separate durable `Task.consecutiveToolFailureRetryCount` budget for same-model retries after threshold consecutive `tool_error` completions; it never consumes `graphResumeRetryCount`, and exhaustion falls through to the unchanged terminal graph-failure park. - Paused graph exits are benign only while the task is still in `in-progress`; that is the user-pause/engine-pause state where preserving the pause without requeueing is intentional. If the graph reports a pause/abort exit after the task has already advanced to another live column (for example `in-review` after an unpause/resume race), `TaskExecutor.handleGraphFailure()` surfaces the boundary as operator-actionable failure evidence (`status:"failed"`/`error` when no failure is already present, plus a task-log entry) and does **not** move, rewind, or auto-merge the task unless the graph result carries the typed interrupted-node marker. The exceptions are typed in-flight node pause aborts (FN-7214), completed/no-commit finalize-to-review teardown (FN-6625/FN-6644/FN-6647), and benign merge-seam pause/resume aborts (FN-6735). For FN-7214 node aborts, `hard-cancel` and lifted `global-pause` provenance can re-enter the interrupted node through the bounded `graphResumeRetryCount` path; explicit `userPaused`, active global pause, merge/finalize provenance, genuine node failures, `autoMerge:false` human-gated review rows, retry-exhausted tasks, and already-confirmed merges still use the protected operator-action path. For completed finalize handoff, once the persisted task row proves a completed finalize handoff (non-`in-progress`, all steps done/skipped, no live pause/status/error, and the finalize-to-review log entry), a trailing graph abort resolves as an already-advanced benign graph exit even if volatile completion markers were cleared by teardown/restart and later abort provenance was re-marked from `completion-finalize` to `hard-cancel`. For merge-seam aborts, `in-review` tasks with no persisted status/error and no confirmed merge may re-enter bounded auto-merge retry only when the failed graph node is a merge/request-merge seam, the graph value is not conflict/contamination/foreign/retry-exhaustion evidence, project settings allow auto-merge processing (or the task is a shared-branch local integration member), and the merge retry budget is not exhausted. `done` and `archived` remain terminal and keep their column/status, while existing failure details are preserved. - A `step-review` node surfaces reviewer verdicts (APPROVE/REVISE/RETHINK/UNAVAILABLE) as outcome edges; `rework` edges (the only legal graph cycles, bounded per instance) route REVISE/RETHINK back to `step-execute`, with RETHINK traversal triggering the reset seam. - A `code` node runs sandboxed TypeScript (esbuild + child process, clamped timeout, no store handle) for arbitrary computed routing/field logic — the same trust tier as project-local script steps. diff --git a/docs/settings-reference.md b/docs/settings-reference.md index e4a8d06483..7d4f755f69 100644 --- a/docs/settings-reference.md +++ b/docs/settings-reference.md @@ -1746,3 +1746,13 @@ Hard cap → pause with `pausedReason: "token_budget_exceeded"`. Soft cap → on ## Model presets Standardize executor/validator pairs; auto-selectable by task size (Small → Budget, Medium → Normal, Large → Complex). + +### Executor consecutive tool-failure retry + +| Setting | Type/default | Behavior | +| --- | --- | --- | +| `executorToolFailureRetryCount` | integer, `2` | Same-model retries before terminal executor parking; `0` disables this policy entirely. | +| `executorToolFailureRetryBackoffMs` | integer, `2000` | Unref'd delay before the rerun. | +| `executorToolFailureThreshold` | integer, `3` | Consecutive terminal tool failures required to qualify. | + +Values are project-scoped and finite values are floored; count/backoff must be at least `0`, and threshold at least `1`, otherwise their defaults apply. The executor evaluates this bounded policy before its terminal graph-failure park: it counts `tool_error` completion entries, resets only on `tool_result`, and ignores `tool` invocation markers. The detector is scoped to the current executor-run agent-log cursor. Its project-scoped atomic claim prevents concurrent retries and classifies cursor mismatch before an exhausted cap so stale handlers do not park newer work. The exhausted audit is compare-and-set deduplicated while the terminal park remains idempotent. FN-7998 may extend this same-model foundation with escalation. diff --git a/packages/core/src/__tests__/executor-tool-failure-retry-claim.test.ts b/packages/core/src/__tests__/executor-tool-failure-retry-claim.test.ts new file mode 100644 index 0000000000..ef8f760e98 --- /dev/null +++ b/packages/core/src/__tests__/executor-tool-failure-retry-claim.test.ts @@ -0,0 +1,17 @@ +import { describe, expect, it } from "vitest"; +import { + claimNextToolFailureRetryImpl, + markToolFailureRetryExhaustedAuditImpl, +} from "../task-store/remaining-ops-6.js"; + +describe("executor tool-failure retry compatibility backend (FN-7996)", () => { + it("preserves legacy terminal parking when no PostgreSQL claim store is available", async () => { + // FNXC:ExecutorToolFailureRetry 2026-07-16-13:30: the removed SQLite runtime must not throw from the default-enabled retry policy. + const compatibilityStore = { backendMode: false }; + + await expect(claimNextToolFailureRetryImpl(compatibilityStore as never, "FN-7996", 12, 2)) + .resolves.toEqual({ outcome: "exhausted" }); + await expect(markToolFailureRetryExhaustedAuditImpl(compatibilityStore as never, "FN-7996")) + .resolves.toBe(false); + }); +}); diff --git a/packages/core/src/__tests__/manual-retry-reset.test.ts b/packages/core/src/__tests__/manual-retry-reset.test.ts index 3594e32b10..6bba74d4ee 100644 --- a/packages/core/src/__tests__/manual-retry-reset.test.ts +++ b/packages/core/src/__tests__/manual-retry-reset.test.ts @@ -54,6 +54,9 @@ describe("buildManualRetryResetPatch", () => { expect(patch[key]).toBe(0); } expect(patch.graphResumeRetryCount).toBe(0); + expect(patch.consecutiveToolFailureRetryCount).toBe(0); + expect(patch.toolFailureDetectorLogCursor).toBeNull(); + expect(patch.toolFailureRetryExhaustedAuditEmitted).toBe(false); }); it("includes all retry-summary counters in the reset key list", () => { diff --git a/packages/core/src/__tests__/settings-defaults.test.ts b/packages/core/src/__tests__/settings-defaults.test.ts index d539854931..4852fc0e22 100644 --- a/packages/core/src/__tests__/settings-defaults.test.ts +++ b/packages/core/src/__tests__/settings-defaults.test.ts @@ -1,5 +1,5 @@ import { afterEach, describe, expect, it, vi } from "vitest"; -import { DEFAULT_MAX_AUTO_MERGE_RETRIES, resolveMaxAutoMergeRetries } from "../in-review-stall.js"; +import { CONSECUTIVE_TOOL_FAILURE_RETRY_THRESHOLD, DEFAULT_CONSECUTIVE_TOOL_FAILURE_RETRY_BACKOFF_MS, DEFAULT_MAX_CONSECUTIVE_TOOL_FAILURE_RETRIES, DEFAULT_MAX_AUTO_MERGE_RETRIES, resolveConsecutiveToolFailureRetryBackoffMs, resolveConsecutiveToolFailureThreshold, resolveMaxAutoMergeRetries, resolveMaxConsecutiveToolFailureRetries } from "../in-review-stall.js"; import { isExperimentalFeatureEnabled } from "../experimental-features.js"; import { DEFAULT_GLOBAL_SETTINGS, DEFAULT_PROJECT_SETTINGS, GLOBAL_SETTINGS_KEYS, PROJECT_SETTINGS_KEYS, isGlobalOnlySettingsKey } from "../settings-schema.js"; import { isWorkflowColumnsEnabled } from "../workflow-columns-settings.js"; @@ -284,4 +284,17 @@ describe("settings defaults invariants", () => { expect(normalizeMergeIntegrationWorktreeMode(null)).toBe("reuse-task-worktree"); }); }); + it("normalizes executor tool-failure retry settings with floor semantics", () => { + expect(DEFAULT_PROJECT_SETTINGS.executorToolFailureRetryCount).toBe(DEFAULT_MAX_CONSECUTIVE_TOOL_FAILURE_RETRIES); + expect(DEFAULT_PROJECT_SETTINGS.executorToolFailureRetryBackoffMs).toBe(DEFAULT_CONSECUTIVE_TOOL_FAILURE_RETRY_BACKOFF_MS); + expect(DEFAULT_PROJECT_SETTINGS.executorToolFailureThreshold).toBe(CONSECUTIVE_TOOL_FAILURE_RETRY_THRESHOLD); + expect(PROJECT_SETTINGS_KEYS).toEqual(expect.arrayContaining(["executorToolFailureRetryCount", "executorToolFailureRetryBackoffMs", "executorToolFailureThreshold"])); + expect(resolveMaxConsecutiveToolFailureRetries({ executorToolFailureRetryCount: 2.7 })).toBe(2); + expect(resolveMaxConsecutiveToolFailureRetries({ executorToolFailureRetryCount: -1 })).toBe(2); + expect(resolveConsecutiveToolFailureRetryBackoffMs({ executorToolFailureRetryBackoffMs: 2500.9 })).toBe(2500); + expect(resolveConsecutiveToolFailureRetryBackoffMs({ executorToolFailureRetryBackoffMs: Infinity })).toBe(2000); + expect(resolveConsecutiveToolFailureThreshold({ executorToolFailureThreshold: 3.9 })).toBe(3); + expect(resolveConsecutiveToolFailureThreshold({ executorToolFailureThreshold: 0.5 })).toBe(3); + }); + }); diff --git a/packages/core/src/in-review-stall.ts b/packages/core/src/in-review-stall.ts index a4512f16d7..32114e831e 100644 --- a/packages/core/src/in-review-stall.ts +++ b/packages/core/src/in-review-stall.ts @@ -41,6 +41,9 @@ export interface InReviewStallContext { export const DEFAULT_STALE_MERGING_MIN_AGE_MS = 5 * 60_000; /** Historical default for the configurable auto-merge conflict retry cap. */ export const DEFAULT_MAX_AUTO_MERGE_RETRIES = 3; +export const DEFAULT_MAX_CONSECUTIVE_TOOL_FAILURE_RETRIES = 2; +export const DEFAULT_CONSECUTIVE_TOOL_FAILURE_RETRY_BACKOFF_MS = 2_000; +export const CONSECUTIVE_TOOL_FAILURE_RETRY_THRESHOLD = 3; /** * FNXC:AutoMergeRetries 2026-06-17-04:20: @@ -53,6 +56,23 @@ export function resolveMaxAutoMergeRetries(settings?: { maxAutoMergeRetries?: un } return DEFAULT_MAX_AUTO_MERGE_RETRIES; } + +/** FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: normalize the project policy identically in engine and settings UI; finite in-range fractions floor, invalid values retain safe defaults. */ +function resolveNonNegativeInteger(value: unknown, fallback: number): number { + const numeric = Number(value); + return Number.isFinite(numeric) && Math.floor(numeric) >= 0 ? Math.floor(numeric) : fallback; +} +export function resolveMaxConsecutiveToolFailureRetries(settings?: { executorToolFailureRetryCount?: unknown } | null): number { + return resolveNonNegativeInteger(settings?.executorToolFailureRetryCount, DEFAULT_MAX_CONSECUTIVE_TOOL_FAILURE_RETRIES); +} +export function resolveConsecutiveToolFailureRetryBackoffMs(settings?: { executorToolFailureRetryBackoffMs?: unknown } | null): number { + return resolveNonNegativeInteger(settings?.executorToolFailureRetryBackoffMs, DEFAULT_CONSECUTIVE_TOOL_FAILURE_RETRY_BACKOFF_MS); +} +export function resolveConsecutiveToolFailureThreshold(settings?: { executorToolFailureThreshold?: unknown } | null): number { + const numeric = Number(settings?.executorToolFailureThreshold); + return Number.isFinite(numeric) && Math.floor(numeric) >= 1 ? Math.floor(numeric) : CONSECUTIVE_TOOL_FAILURE_RETRY_THRESHOLD; +} + export const IN_REVIEW_STALL_LOG_PREFIX = "In-review stall surfaced ["; export const IN_REVIEW_STALL_DEADLOCK_LOG_PREFIX = "In-review stall auto-disposed ["; export const IN_REVIEW_STALL_TERMINAL_LOG_PREFIX = "In-review stall terminal disposed ["; diff --git a/packages/core/src/index.gate.ts b/packages/core/src/index.gate.ts index 54e1c91fd7..8b062951fb 100644 --- a/packages/core/src/index.gate.ts +++ b/packages/core/src/index.gate.ts @@ -987,6 +987,12 @@ export { DEFAULT_STALE_MERGING_MIN_AGE_MS, DEFAULT_MAX_AUTO_MERGE_RETRIES, resolveMaxAutoMergeRetries, + DEFAULT_MAX_CONSECUTIVE_TOOL_FAILURE_RETRIES, + DEFAULT_CONSECUTIVE_TOOL_FAILURE_RETRY_BACKOFF_MS, + CONSECUTIVE_TOOL_FAILURE_RETRY_THRESHOLD, + resolveMaxConsecutiveToolFailureRetries, + resolveConsecutiveToolFailureRetryBackoffMs, + resolveConsecutiveToolFailureThreshold, } from "./in-review-stall.js"; export type { InReviewStallSignal, InReviewStallCode, ProviderErrorClassification } from "./in-review-stall.js"; export { diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index ad5a081ba0..5bd5690ba2 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -1021,6 +1021,12 @@ export { DEFAULT_STALE_MERGING_MIN_AGE_MS, DEFAULT_MAX_AUTO_MERGE_RETRIES, resolveMaxAutoMergeRetries, + DEFAULT_MAX_CONSECUTIVE_TOOL_FAILURE_RETRIES, + DEFAULT_CONSECUTIVE_TOOL_FAILURE_RETRY_BACKOFF_MS, + CONSECUTIVE_TOOL_FAILURE_RETRY_THRESHOLD, + resolveMaxConsecutiveToolFailureRetries, + resolveConsecutiveToolFailureRetryBackoffMs, + resolveConsecutiveToolFailureThreshold, } from "./in-review-stall.js"; export type { InReviewStallSignal, InReviewStallCode, ProviderErrorClassification } from "./in-review-stall.js"; export { diff --git a/packages/core/src/manual-retry-reset.ts b/packages/core/src/manual-retry-reset.ts index 6229807569..eeca2655a6 100644 --- a/packages/core/src/manual-retry-reset.ts +++ b/packages/core/src/manual-retry-reset.ts @@ -7,6 +7,7 @@ export const MANUAL_RETRY_RESET_COUNTER_KEYS = [ "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", + "consecutiveToolFailureRetryCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", @@ -42,6 +43,8 @@ export function buildAutoPauseClearPatch( export function buildManualRetryResetPatch(options?: { resetMergeRetries?: boolean }): Partial { const patch: Partial = { nextRecoveryAt: null as unknown as Task["nextRecoveryAt"], + toolFailureDetectorLogCursor: null, + toolFailureRetryExhaustedAuditEmitted: false, }; for (const key of MANUAL_RETRY_RESET_COUNTER_KEYS) { diff --git a/packages/core/src/postgres/migrations/0013_executor_tool_failure_retry.sql b/packages/core/src/postgres/migrations/0013_executor_tool_failure_retry.sql new file mode 100644 index 0000000000..86f289f66b --- /dev/null +++ b/packages/core/src/postgres/migrations/0013_executor_tool_failure_retry.sql @@ -0,0 +1,4 @@ +-- FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: durable bounded retry state is project-task scoped in the PostgreSQL production backend. +ALTER TABLE project.tasks ADD COLUMN IF NOT EXISTS consecutive_tool_failure_retry_count integer DEFAULT 0; +ALTER TABLE project.tasks ADD COLUMN IF NOT EXISTS tool_failure_detector_log_cursor integer; +ALTER TABLE project.tasks ADD COLUMN IF NOT EXISTS tool_failure_retry_exhausted_audit_emitted integer DEFAULT 0; diff --git a/packages/core/src/postgres/schema-applier.ts b/packages/core/src/postgres/schema-applier.ts index 1118143e69..5d5225224a 100644 --- a/packages/core/src/postgres/schema-applier.ts +++ b/packages/core/src/postgres/schema-applier.ts @@ -72,6 +72,8 @@ Version 0012 makes the persisted pin timestamp available on databases that already applied the baseline before Direct conversations can be pinned. */ export const CHAT_SESSION_PINS_VERSION = "0012"; +/** FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: upgrades existing PostgreSQL task rows before retry-state reads. */ +export const EXECUTOR_TOOL_FAILURE_RETRY_VERSION = "0013"; /** Bookkeeping table for the fresh Drizzle migration history. */ export const MIGRATION_BOOKKEEPING_TABLE = "fusion_schema_migrations"; @@ -138,6 +140,11 @@ const CHAT_SESSION_PINS_MIGRATION_PATH = join( "migrations", "0012_chat_session_pins.sql", ); +const EXECUTOR_TOOL_FAILURE_RETRY_MIGRATION_PATH = join( + __dirname, + "migrations", + "0013_executor_tool_failure_retry.sql", +); /** * Ensure the migration bookkeeping table exists. Lives in the public schema so @@ -219,6 +226,7 @@ export async function applySchemaBaseline( const importTranslationCacheAlreadyApplied = applied.includes(IMPORT_TRANSLATION_CACHE_VERSION); const ownerProjectIdSplitAlreadyApplied = applied.includes(OWNER_PROJECT_ID_SPLIT_VERSION); const chatSessionPinsAlreadyApplied = applied.includes(CHAT_SESSION_PINS_VERSION); + const executorToolFailureRetryAlreadyApplied = applied.includes(EXECUTOR_TOOL_FAILURE_RETRY_VERSION); let schemaChanged = false; if (!baselineAlreadyApplied) { @@ -485,6 +493,15 @@ export async function applySchemaBaseline( schemaChanged = true; } + if (!executorToolFailureRetryAlreadyApplied) { + const executorToolFailureRetrySql = await readFile(EXECUTOR_TOOL_FAILURE_RETRY_MIGRATION_PATH, "utf8"); + await tx.execute(sql.raw(executorToolFailureRetrySql)); + await tx.execute( + sql`INSERT INTO public.${sql.identifier(MIGRATION_BOOKKEEPING_TABLE)} (version) VALUES (${EXECUTOR_TOOL_FAILURE_RETRY_VERSION}) ON CONFLICT (version) DO NOTHING`, + ); + schemaChanged = true; + } + return { applied: schemaChanged, pluginHooksRun: pluginHooks.length }; }); } diff --git a/packages/core/src/postgres/schema/project.ts b/packages/core/src/postgres/schema/project.ts index e2deab66bb..5d15d8ec9b 100644 --- a/packages/core/src/postgres/schema/project.ts +++ b/packages/core/src/postgres/schema/project.ts @@ -100,6 +100,9 @@ export const tasks = projectSchema.table("tasks", { workflowStepRetries: integer("workflow_step_retries"), resumeLimboCount: integer("resume_limbo_count").default(0), graphResumeRetryCount: integer("graph_resume_retry_count").default(0), + consecutiveToolFailureRetryCount: integer("consecutive_tool_failure_retry_count").default(0), + toolFailureDetectorLogCursor: integer("tool_failure_detector_log_cursor"), + toolFailureRetryExhaustedAuditEmitted: integer("tool_failure_retry_exhausted_audit_emitted").default(0), resumeLimboTipSha: text("resume_limbo_tip_sha"), resumeLimboStepSignature: text("resume_limbo_step_signature"), // FNXC:WorkflowLifecycle 2026-07-12 (merge port from main): FN-7863 execute self-requeue streak. diff --git a/packages/core/src/settings-schema.ts b/packages/core/src/settings-schema.ts index dd1f6add67..6e04d1727a 100644 --- a/packages/core/src/settings-schema.ts +++ b/packages/core/src/settings-schema.ts @@ -493,6 +493,9 @@ export const DEFAULT_PROJECT_SETTINGS = { * Project settings own the auto-merge conflict retry cap because existing engine/dashboard consumers already resolve project settings; the default imports core's stall-detection fallback to keep every surface on the historical value of 3. */ maxAutoMergeRetries: DEFAULT_MAX_AUTO_MERGE_RETRIES, + executorToolFailureRetryCount: 2, + executorToolFailureRetryBackoffMs: 2000, + executorToolFailureThreshold: 3, /** * FNXC:Merge 2026-06-26-00:00: * New and unconfigured projects default AI merge to sync a dirty checked-out integration branch, restoring the legacy stash → fast-forward → restore landing behavior. Explicit persisted merger.allowDirtyLocalCheckoutSync values still win, and no existing-project migration stamps this default into storage. diff --git a/packages/core/src/store.ts b/packages/core/src/store.ts index 16c5d4a473..594ec24a3e 100644 --- a/packages/core/src/store.ts +++ b/packages/core/src/store.ts @@ -98,7 +98,7 @@ import { pgRowToTaskRow as pgRowToTaskRowExternal, rowToTask as rowToTaskExterna import { moveTaskImpl, handoffToReviewImpl, moveTaskInternalImpl } from "./task-store/moves.js"; import { recordGoalCitationsImpl, insertTaskWithFtsRecoveryImpl2, assertTaskIdAvailableImpl, atomicWriteTaskJsonImpl2, createTaskWithDistributedReservationImpl, toStoredWorkflowStepImpl, ensureWorkflowStepForTemplateImpl, resolveEnabledWorkflowStepsImpl, setTaskBranchGroupImpl, getTaskColumnsImpl, prepareWorkflowMovePolicyPreflightImpl, updateTaskCustomFieldsImpl, listWorkflowPromptOverridesForProjectImpl, listWorkflowWorkItemsForTaskImpl, listDueWorkflowWorkItemsImpl, rewriteBlockedByResidueDependentsForRemovalImpl, getAllDocumentsImpl, deleteWorkflowStepImpl, toWorkflowDefinitionImpl, materializeDefaultWorkflowStepsImpl, reconcileTaskCustomFieldsForSchemaImpl, getTaskMovedCountsByDayImpl, getGoalStoreImpl, upsertTaskCommitAssociationImpl } from "./task-store/remaining-ops-4.js"; import { applyLegacyWorkflowStepOverridesImpl, applyTaskPatchImpl, archiveDbImpl, assertNoDependencyCycleImpl, atomicCreateTaskJsonImpl, buildActiveTaskDependencyLookupImpl, buildArchivedAgentLogFieldsImpl, buildTaskIdIntegrityFallbackReportImpl, createBranchGroupImpl, dbImpl, detectAndCacheTaskIdIntegrityReportImpl, findLiveDependentsImpl, findLiveLineageChildrenImpl, getLegacyWorkflowStepSnapshotImpl, getMalformedTaskMetadataReasonImpl, getMergeQueuedTaskIdsAsyncImpl, insertRunAuditEventRowImpl, insertTaskImpl, invokeTaskCreatedHookImpl, isTaskArchivedImpl, isTaskIdPresentInArchivedTasksTableImpl, logTaskCreateConflictImpl, maybeResolveTombstonedTaskIdImpl, mergeTaskIdIntegrityReportsImpl, optionalGroupIdSetImpl, patchTaskRowInTransactionImpl, readConfigFastImpl, readConfigImpl, readPromptForArchiveImpl, readTaskFromDbImpl, reconcileDistributedTaskIdStateOnOpenImpl, recordActivityFromListenerImpl, recordDependencyCycleRejectedAuditImpl, refreshTaskIdIntegrityReportImpl, resolveLocalNodeIdForTaskAllocationImpl, runTaskFtsWriteWithRecoveryImpl, scanAndRecordCitationsImpl, taskIdExistsAnywhereImpl, throwSoftDeletedWriteBlockedImpl, toBuiltInWorkflowStepImpl, trackDeferredTaskCreatedWorkImpl, upsertTaskImpl, withConfigLockImpl, withTaskLockImpl, withWorktreeAllocationLockImpl } from "./task-store/remaining-ops-5.js"; -import { clearNearDuplicateReferencesToFailSoftImpl, clearWorkflowRunStepInstancesImpl, computeMovedSettingsTargetWorkflowIdsImpl, ensureBranchGroupForSourceImpl, ensurePrEntityForSourceImpl, findRecentTasksByContentFingerprintImpl, getActiveMergingTaskImpl, getActivePrEntityBySourceImpl, getBranchGroupByBranchNameImpl, getBranchGroupBySourceImpl, getBranchGroupImpl, getBranchProgressByTaskImpl, getMutationsForRunImpl, getPrEntityByNumberImpl, getPrEntityImpl, getPrThreadStateImpl, getTasksByAssignedAgentImpl, getWorkflowPromptOverridesAsyncImpl, getWorkflowSettingValuesAsyncImpl, getWorkflowSettingValuesImpl, getWorkflowSettingsProjectIdImpl, getWorkflowWorkItemImpl, insertCompletionHandoffWorkflowWorkAuditImpl, listActivePrEntitiesImpl, listBranchGroupsImpl, listPrThreadStatesImpl, listTasksByBranchGroupImpl, listWorkflowSettingValuesForProjectImpl, loadWorkflowRunBranchesImpl, loadWorkflowRunStepInstancesImpl, mergeCustomFieldPatchImpl, normalizeMergeRequestStateImpl, normalizeWorkflowWorkItemKindImpl, normalizeWorkflowWorkItemStateImpl, parseWorkflowPromptOverrideJsonImpl, recordPrThreadOutcomeImpl, resetAllStepsToPendingImpl, resetPromptCheckboxesImpl, resolveWorkflowMoveActorImpl, resolveWorkflowSettingDeclarationsImpl, saveWorkflowRunStepInstanceImpl, transitionMergeRequestStateImpl, transitionWorkflowWorkItemSyncImpl, updateTaskImpl, updateWorkflowPromptOverridesImpl, upsertMergeRequestRecordImpl, workflowStateForMergeRequestStateImpl } from "./task-store/remaining-ops-6.js"; +import { claimNextToolFailureRetryImpl, clearNearDuplicateReferencesToFailSoftImpl, clearWorkflowRunStepInstancesImpl, computeMovedSettingsTargetWorkflowIdsImpl, ensureBranchGroupForSourceImpl, ensurePrEntityForSourceImpl, findRecentTasksByContentFingerprintImpl, getActiveMergingTaskImpl, getActivePrEntityBySourceImpl, getBranchGroupByBranchNameImpl, getBranchGroupBySourceImpl, getBranchGroupImpl, getBranchProgressByTaskImpl, getMutationsForRunImpl, getPrEntityByNumberImpl, getPrEntityImpl, getPrThreadStateImpl, getTasksByAssignedAgentImpl, getWorkflowPromptOverridesAsyncImpl, getWorkflowSettingValuesAsyncImpl, getWorkflowSettingValuesImpl, getWorkflowSettingsProjectIdImpl, getWorkflowWorkItemImpl, insertCompletionHandoffWorkflowWorkAuditImpl, listActivePrEntitiesImpl, listBranchGroupsImpl, listPrThreadStatesImpl, listTasksByBranchGroupImpl, listWorkflowSettingValuesForProjectImpl, loadWorkflowRunBranchesImpl, loadWorkflowRunStepInstancesImpl, markToolFailureRetryExhaustedAuditImpl, mergeCustomFieldPatchImpl, normalizeMergeRequestStateImpl, normalizeWorkflowWorkItemKindImpl, normalizeWorkflowWorkItemStateImpl, parseWorkflowPromptOverrideJsonImpl, recordPrThreadOutcomeImpl, resetAllStepsToPendingImpl, resetPromptCheckboxesImpl, resolveWorkflowMoveActorImpl, resolveWorkflowSettingDeclarationsImpl, saveWorkflowRunStepInstanceImpl, transitionMergeRequestStateImpl, transitionWorkflowWorkItemSyncImpl, updateTaskImpl, updateWorkflowPromptOverridesImpl, upsertMergeRequestRecordImpl, workflowStateForMergeRequestStateImpl } from "./task-store/remaining-ops-6.js"; import { addPrInfoImpl, addSteeringCommentImpl, archiveAllDoneImpl, cleanupStaleMergeQueueRowsImpl, clearCompletionHandoffAcceptedMarkerImpl, clearDoneTransientFieldsImpl, clearStaleExecutionStartBranchReferencesImpl, computeWorkflowColumnsGraduationReportImpl, deleteTaskCommentImpl, deleteTaskDocumentImpl, emitUsageEventImpl, enqueueMergeQueueImpl, getAgentLogCountImpl, getAgentLogsImpl, getArtifactImpl, getArtifactsImpl, getAttachmentImpl, getCompletionHandoffAcceptedMarkerImpl, getTaskDocumentImpl, getTaskDocumentRevisionsImpl, getTaskDocumentsImpl, insertArtifactRowImpl, linkGithubIssueImpl, listWorkflowWorkItemsForTaskSyncImpl, moveToDoneImpl, parseDependenciesFromPromptImpl, parseFileScopeFromPromptImpl, parseStepsFromPromptImpl, peekMergeQueueHeadImpl, peekMergeQueueImpl, readPreArchiveColumnFromTaskFileImpl, recordPluginActivationImpl, recordRunAuditEventBackendImpl, removePrInfoByNumberImpl, resolvePrimaryPrInfoImpl, resolveUnarchiveTargetColumnImpl, rewriteLineageChildrenForRemovalImpl, runGitCommandImpl, stopWatchingImpl, syncAgentTaskLinkOnReassignmentImpl, updateArtifactImpl, updateGithubTrackingImpl, updatePrInfoByNumberImpl, updateTaskCommentImpl, upsertPrInfoByNumberImpl, writeArtifactDataImpl } from "./task-store/remaining-ops-7.js"; import { approveCliAutonomyImpl, approveWorkflowCliCommandImpl, cleanupOrphanedMaterializedStepsImpl, consumePluginGateVerdictsImpl, getAgentLogsByTimeRangeImpl, getDatabaseHealthImpl, getDistributedTaskIdAllocatorImpl, getExperimentSessionStoreImpl, getInReviewDurationEventsImpl, getMissionStoreImpl, getPluginStoreImpl, getSecretsStoreImpl, getSettingsSyncImpl, getTaskMergedTaskIdsImpl, getTaskWorkflowSelectionImpl, getImportTranslationImpl, recordImportTranslationImpl, pruneImportTranslationsImpl, type ImportTranslationCacheKey, type ImportTranslationCacheEntry, getVerificationCacheHitImpl, getWorkflowDefinitionImpl, healthCheckImpl, importLegacyAgentLogsOnceImpl, insertWorkflowDefinitionSyncImpl, isCliAutonomyApprovedImpl, isPluginInstalledImpl, isWorkflowCliCommandApprovedImpl, listWorkflowDefinitionsImpl, materializeExplicitWorkflowStepsImpl, materializeWorkflowStepsImpl, migrateActiveArchivedTasksToArchiveDbImpl, migrateLegacyArchiveEntriesToArchiveDbImpl, nextWorkflowDefinitionIdImpl, occupantsByColumnForWorkflowImpl, parseWorkflowLayoutImpl, pruneAgentLogFilesImpl, purgeTaskWorkflowSelectionRowsImpl, readAllWorkflowDefinitionsImpl, readRawProjectSettingsImpl, recordPluginGateVerdictImpl, recordVerificationCachePassImpl, removeMaterializedSelectionImpl, resolvePluginWorkflowStepImpl, resolveTaskWorkflowIrSyncImpl, revokeCliAutonomyImpl, selectTaskWorkflowAndReconcileImpl, writeTaskWorkflowSelectionImpl, getTaskWorkflowSelectionAsyncImpl, } from "./task-store/remaining-ops-8.js"; import { getTaskCommitAssociationsByLineageIdImpl, replaceLegacyTaskCommitAssociationsImpl } from "./task-store/task-commit-associations.js"; @@ -1166,10 +1166,16 @@ export class TaskStore extends EventEmitter { } async updateTask( id: string, - updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; workspaceWorktrees?: import("./types.js").Task["workspaceWorktrees"]; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; customFields?: Record; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; executeRequeueLoopCount?: number | null; graphResumeRetryCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; executeRequeueLoopSignature?: string | null; postReviewFixCount?: number | null; planReviewReplanCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; validatorThinkingLevel?: string | null; planningThinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record | null; githubTracking?: import("./types.js").TaskGithubTracking | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null; workflowTransitionNotification?: import("./types.js").WorkflowTransitionNotificationMarker | undefined; plannerOversightLevel?: string | null; sessionAdvisorEnabled?: boolean | null; approvedPlanFingerprint?: string | null }, runContext?: RunMutationContext, + updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; workspaceWorktrees?: import("./types.js").Task["workspaceWorktrees"]; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; customFields?: Record; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; executeRequeueLoopCount?: number | null; graphResumeRetryCount?: number | null; consecutiveToolFailureRetryCount?: number | null; toolFailureDetectorLogCursor?: number | null; toolFailureRetryExhaustedAuditEmitted?: boolean | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; executeRequeueLoopSignature?: string | null; postReviewFixCount?: number | null; planReviewReplanCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; validatorThinkingLevel?: string | null; planningThinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record | null; githubTracking?: import("./types.js").TaskGithubTracking | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null; workflowTransitionNotification?: import("./types.js").WorkflowTransitionNotificationMarker | undefined; plannerOversightLevel?: string | null; sessionAdvisorEnabled?: boolean | null; approvedPlanFingerprint?: string | null }, runContext?: RunMutationContext, ): Promise { return updateTaskImpl(this, id, updates, runContext); } + async claimNextToolFailureRetry(taskId: string, expectedCursor: number, maxRetries: number): Promise { + return claimNextToolFailureRetryImpl(this, taskId, expectedCursor, maxRetries); + } + async markToolFailureRetryExhaustedAudit(taskId: string): Promise { + return markToolFailureRetryExhaustedAuditImpl(this, taskId); + } async updateTaskAtomic( id: string, updater: ( current: Task, ) => Parameters[1] | null | undefined | Promise[1] | null | undefined>, runContext?: RunMutationContext, ): Promise { return updateTaskAtomicImpl(this, id, updater, runContext); } diff --git a/packages/core/src/task-store/persistence.ts b/packages/core/src/task-store/persistence.ts index 4f73442a34..fff820594c 100644 --- a/packages/core/src/task-store/persistence.ts +++ b/packages/core/src/task-store/persistence.ts @@ -47,6 +47,9 @@ export interface TaskRow { stuckKillCount: number | null; resumeLimboCount: number | null; graphResumeRetryCount: number | null; + consecutiveToolFailureRetryCount: number | null; + toolFailureDetectorLogCursor: number | null; + toolFailureRetryExhaustedAuditEmitted: number | null; resumeLimboTipSha: string | null; resumeLimboStepSignature: string | null; executeRequeueLoopCount: number | null; @@ -228,6 +231,10 @@ export const TASK_COLUMN_DESCRIPTORS: TaskColumnDescriptor[] = [ defineTaskColumn("stuckKillCount", (task) => task.stuckKillCount ?? 0), defineTaskColumn("resumeLimboCount", (task) => task.resumeLimboCount ?? 0), defineTaskColumn("graphResumeRetryCount", (task) => task.graphResumeRetryCount === undefined ? 0 : task.graphResumeRetryCount), + defineTaskColumn("consecutiveToolFailureRetryCount", (task) => task.consecutiveToolFailureRetryCount ?? 0), + defineTaskColumn("toolFailureDetectorLogCursor", (task) => task.toolFailureDetectorLogCursor ?? null), + // FNXC:ExecutorToolFailureRetry 2026-07-16-13:30: PostgreSQL stores this CAS marker as integer (0/1), matching its schema and legacy SQLite flag representation. + defineTaskColumn("toolFailureRetryExhaustedAuditEmitted", (task) => task.toolFailureRetryExhaustedAuditEmitted ? 1 : 0), defineTaskColumn("resumeLimboTipSha", (task) => task.resumeLimboTipSha ?? null), defineTaskColumn("resumeLimboStepSignature", (task) => task.resumeLimboStepSignature ?? null), // FNXC:WorkflowLifecycle 2026-07-12 (merge port from main): FN-7863 progress-anchored execute self-requeue streak. diff --git a/packages/core/src/task-store/remaining-ops-2.ts b/packages/core/src/task-store/remaining-ops-2.ts index 5db5f02629..9e26342776 100644 --- a/packages/core/src/task-store/remaining-ops-2.ts +++ b/packages/core/src/task-store/remaining-ops-2.ts @@ -44,7 +44,7 @@ export function getTaskSelectClauseWithActivityLogLimitImpl(store: TaskStore, li "modelPresetId", "modelProvider", "modelId", "validatorModelProvider", "validatorModelId", "planningModelProvider", "planningModelId", - "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "executeRequeueLoopSignature", "postReviewFixCount", "planReviewReplanCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", + "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", "consecutiveToolFailureRetryCount", "toolFailureDetectorLogCursor", "toolFailureRetryExhaustedAuditEmitted", "resumeLimboTipSha", "resumeLimboStepSignature", "executeRequeueLoopSignature", "postReviewFixCount", "planReviewReplanCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", "error", "summary", "thinkingLevel", "validatorThinkingLevel", "planningThinkingLevel", "executionMode", "tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenUsageModelProvider", "tokenUsageModelId", "tokenUsagePerModel", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride", "createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt", diff --git a/packages/core/src/task-store/remaining-ops-3.ts b/packages/core/src/task-store/remaining-ops-3.ts index 03cc269540..6d9a5e97dd 100644 --- a/packages/core/src/task-store/remaining-ops-3.ts +++ b/packages/core/src/task-store/remaining-ops-3.ts @@ -34,7 +34,7 @@ export function getTaskSelectClauseImpl2(store: TaskStore, slim: boolean, tableA "modelPresetId", "modelProvider", "modelId", "validatorModelProvider", "validatorModelId", "planningModelProvider", "planningModelId", - "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "executeRequeueLoopSignature", "postReviewFixCount", "planReviewReplanCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", + "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", "consecutiveToolFailureRetryCount", "toolFailureDetectorLogCursor", "toolFailureRetryExhaustedAuditEmitted", "resumeLimboTipSha", "resumeLimboStepSignature", "executeRequeueLoopSignature", "postReviewFixCount", "planReviewReplanCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", "error", "summary", "thinkingLevel", "validatorThinkingLevel", "planningThinkingLevel", "executionMode", "tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenUsageModelProvider", "tokenUsageModelId", "tokenUsagePerModel", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride", "createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "executionStartedAt", "executionCompletedAt", diff --git a/packages/core/src/task-store/remaining-ops-6.ts b/packages/core/src/task-store/remaining-ops-6.ts index ec1e169055..59077ec148 100644 --- a/packages/core/src/task-store/remaining-ops-6.ts +++ b/packages/core/src/task-store/remaining-ops-6.ts @@ -624,7 +624,7 @@ export async function resetPromptCheckboxesImpl(store: TaskStore, dir: string): export async function updateTaskImpl(store: TaskStore, id: string, - updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; workspaceWorktrees?: import("../types.js").Task["workspaceWorktrees"]; status?: string | null; dependencies?: string[]; steps?: import("../types.js").TaskStep[]; customFields?: Record; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("../types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("../types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("../types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; executeRequeueLoopCount?: number | null; graphResumeRetryCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; executeRequeueLoopSignature?: string | null; postReviewFixCount?: number | null; planReviewReplanCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; validatorThinkingLevel?: string | null; planningThinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("../types.js").TaskReview | null; reviewState?: import("../types.js").TaskReviewState | null; workflowStepResults?: import("../types.js").WorkflowStepResult[] | null; mergeDetails?: import("../types.js").MergeDetails | null; sourceIssue?: import("../types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record | null; githubTracking?: import("../types.js").TaskGithubTracking | null; tokenUsage?: import("../types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null; workflowTransitionNotification?: import("../types.js").WorkflowTransitionNotificationMarker | undefined; sessionAdvisorEnabled?: boolean | null }, runContext?: RunMutationContext, + updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; workspaceWorktrees?: import("../types.js").Task["workspaceWorktrees"]; status?: string | null; dependencies?: string[]; steps?: import("../types.js").TaskStep[]; customFields?: Record; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("../types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("../types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("../types.js").ExecutionMode | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; executeRequeueLoopCount?: number | null; graphResumeRetryCount?: number | null; consecutiveToolFailureRetryCount?: number | null; toolFailureDetectorLogCursor?: number | null; toolFailureRetryExhaustedAuditEmitted?: boolean | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; executeRequeueLoopSignature?: string | null; postReviewFixCount?: number | null; planReviewReplanCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; validatorThinkingLevel?: string | null; planningThinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("../types.js").TaskReview | null; reviewState?: import("../types.js").TaskReviewState | null; workflowStepResults?: import("../types.js").WorkflowStepResult[] | null; mergeDetails?: import("../types.js").MergeDetails | null; sourceIssue?: import("../types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record | null; githubTracking?: import("../types.js").TaskGithubTracking | null; tokenUsage?: import("../types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; missionId?: string | null; sliceId?: string | null; workflowTransitionNotification?: import("../types.js").WorkflowTransitionNotificationMarker | undefined; sessionAdvisorEnabled?: boolean | null }, runContext?: RunMutationContext, ): Promise { /* FNXC:StateMachine 2026-07-07-12:00: @@ -1339,3 +1339,66 @@ export async function getWorkflowWorkItemImpl(store: TaskStore, id: string): Pro const row = store.db.prepare("SELECT * FROM workflow_work_items WHERE id = ?").get(id) as WorkflowWorkItemRow | undefined; return row ? store.rowToWorkflowWorkItem(row) : null; } + +export type ToolFailureRetryClaim = + | { outcome: "claimed"; attempt: number } + | { outcome: "already-claimed-for-run" } + | { outcome: "exhausted" }; + +/** + * FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: + * PostgreSQL owns the durable per-run claim. The cursor predicate is deliberately + * evaluated before the cap after a miss: an older handler must never park a newer run. + */ +export async function claimNextToolFailureRetryImpl( + store: TaskStore, + taskId: string, + expectedCursor: number, + maxRetries: number, +): Promise { + /* + * FNXC:ExecutorToolFailureRetry 2026-07-16-13:30: + * PostgreSQL is the only runtime TaskStore backend. Keep compatibility-mode + * stores on the legacy terminal-park path instead of throwing from a + * default-enabled retry policy: they have no durable atomic claim primitive. + */ + if (!store.backendMode) return { outcome: "exhausted" }; + const layer = store.asyncLayer!; + const projectId = layer.projectId?.trim() || "__legacy_unscoped__"; + const rows = await layer.db.update(schema.project.tasks).set({ + consecutiveToolFailureRetryCount: sql`coalesce(${schema.project.tasks.consecutiveToolFailureRetryCount}, 0) + 1`, + toolFailureDetectorLogCursor: null, + updatedAt: new Date().toISOString(), + }).where(and( + eq(schema.project.tasks.projectId, projectId), + eq(schema.project.tasks.id, taskId), + eq(schema.project.tasks.toolFailureDetectorLogCursor, expectedCursor), + sql`coalesce(${schema.project.tasks.consecutiveToolFailureRetryCount}, 0) < ${maxRetries}`, + )).returning({ attempt: schema.project.tasks.consecutiveToolFailureRetryCount }); + if (rows.length > 0) return { outcome: "claimed", attempt: rows[0]!.attempt ?? 1 }; + const [current] = await layer.db.select({ + cursor: schema.project.tasks.toolFailureDetectorLogCursor, + count: schema.project.tasks.consecutiveToolFailureRetryCount, + }).from(schema.project.tasks).where(and(eq(schema.project.tasks.projectId, projectId), eq(schema.project.tasks.id, taskId))); + // Cursor mismatch MUST win before cap: stale handlers silently defer to the newer run. + if (!current || current.cursor !== expectedCursor) return { outcome: "already-claimed-for-run" }; + if ((current.count ?? 0) >= maxRetries) return { outcome: "exhausted" }; + return { outcome: "already-claimed-for-run" }; +} + +/** CAS for the single exhaustion audit; terminal parking intentionally remains idempotent. */ +export async function markToolFailureRetryExhaustedAuditImpl(store: TaskStore, taskId: string): Promise { + // FNXC:ExecutorToolFailureRetry 2026-07-16-13:30: non-PostgreSQL compatibility stores safely skip the deduplicated audit and retain legacy terminal parking. + if (!store.backendMode) return false; + const layer = store.asyncLayer!; + const projectId = layer.projectId?.trim() || "__legacy_unscoped__"; + const rows = await layer.db.update(schema.project.tasks).set({ + toolFailureRetryExhaustedAuditEmitted: 1, + updatedAt: new Date().toISOString(), + }).where(and( + eq(schema.project.tasks.projectId, projectId), + eq(schema.project.tasks.id, taskId), + sql`(${schema.project.tasks.toolFailureRetryExhaustedAuditEmitted} is null or ${schema.project.tasks.toolFailureRetryExhaustedAuditEmitted} = 0)`, + )).returning({ id: schema.project.tasks.id }); + return rows.length > 0; +} diff --git a/packages/core/src/task-store/serialization.ts b/packages/core/src/task-store/serialization.ts index a25cdc3abd..e0ac529a27 100644 --- a/packages/core/src/task-store/serialization.ts +++ b/packages/core/src/task-store/serialization.ts @@ -101,6 +101,9 @@ export function rowToTask(row: TaskRow): Task { stuckKillCount: row.stuckKillCount ?? undefined, resumeLimboCount: row.resumeLimboCount ?? undefined, graphResumeRetryCount: row.graphResumeRetryCount ?? undefined, + consecutiveToolFailureRetryCount: row.consecutiveToolFailureRetryCount ?? undefined, + toolFailureDetectorLogCursor: row.toolFailureDetectorLogCursor ?? undefined, + toolFailureRetryExhaustedAuditEmitted: row.toolFailureRetryExhaustedAuditEmitted ? true : undefined, resumeLimboTipSha: row.resumeLimboTipSha || undefined, resumeLimboStepSignature: row.resumeLimboStepSignature || undefined, executeRequeueLoopCount: row.executeRequeueLoopCount ?? undefined, diff --git a/packages/core/src/task-store/task-update.ts b/packages/core/src/task-store/task-update.ts index 5f9ab31ea3..7bfa68257b 100644 --- a/packages/core/src/task-store/task-update.ts +++ b/packages/core/src/task-store/task-update.ts @@ -365,6 +365,12 @@ export async function updateTaskUnlockedImpl(store: TaskStore, id: string, updat } else if (updates.graphResumeRetryCount !== undefined) { task.graphResumeRetryCount = updates.graphResumeRetryCount; } + if (updates.consecutiveToolFailureRetryCount === null) task.consecutiveToolFailureRetryCount = null; + else if (updates.consecutiveToolFailureRetryCount !== undefined) task.consecutiveToolFailureRetryCount = updates.consecutiveToolFailureRetryCount; + if (updates.toolFailureDetectorLogCursor === null) task.toolFailureDetectorLogCursor = null; + else if (updates.toolFailureDetectorLogCursor !== undefined) task.toolFailureDetectorLogCursor = updates.toolFailureDetectorLogCursor; + if (updates.toolFailureRetryExhaustedAuditEmitted === null) task.toolFailureRetryExhaustedAuditEmitted = null; + else if (updates.toolFailureRetryExhaustedAuditEmitted !== undefined) task.toolFailureRetryExhaustedAuditEmitted = updates.toolFailureRetryExhaustedAuditEmitted; if (updates.resumeLimboTipSha === null) { task.resumeLimboTipSha = undefined; } else if (updates.resumeLimboTipSha !== undefined) { diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index 1ffaa0a9f0..e67b3ffe18 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -1613,6 +1613,15 @@ export interface Task { * and by successful forward progress; capped by the executor before terminal * `status:"failed"` is recorded to preserve the FN-5704 anti-loop exemption. */ graphResumeRetryCount?: number | null; + /** + * FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: + * FN-7996 persists the bounded same-model retry budget for consecutive terminal tool errors. The executor atomically claims it per run cursor so concurrent failures cannot exceed the configured cap. + */ + consecutiveToolFailureRetryCount?: number | null; + /** Agent-log boundary captured at executor-run start; only later terminal outcomes qualify. */ + toolFailureDetectorLogCursor?: number | null; + /** Durable compare-and-set marker which permits one exhaustion audit per retry window. */ + toolFailureRetryExhaustedAuditEmitted?: boolean | null; /** Branch tip SHA snapshot captured at the last reclaim/unpause attempt used * by resume-limbo detection to determine whether commits advanced. */ resumeLimboTipSha?: string; @@ -3036,6 +3045,13 @@ export interface ProjectSettings { /** Maximum number of concurrent AI agents across all activity types * (triage specification, task execution, and merge operations). */ maxConcurrent: number; + /** + * FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: + * Bounded same-model retry before the executor terminal park. Tool markers are ignored, terminal tool_error counts, tool_result resets; per-run cursor claims prevent concurrent over-retry and count 0 preserves prior behavior. Values are floored and the backoff timer is unref'd. FN-7998 consumes this stable policy shape for escalation. + */ + executorToolFailureRetryCount?: number; + executorToolFailureRetryBackoffMs?: number; + executorToolFailureThreshold?: number; /** * FNXC:VerificationConcurrency 2026-07-15-03:35: * Max concurrent verification subprocesses (fn_run_verification / merge testCommand builds) across all tasks in this process. Caps stacked monorepo typecheck/build pegging CPU when many tasks are in-progress. Default 1. Raise only on high-core hosts. diff --git a/packages/dashboard/app/components/SettingsModal.tsx b/packages/dashboard/app/components/SettingsModal.tsx index 85ae5ce4d1..eeb39db7d8 100644 --- a/packages/dashboard/app/components/SettingsModal.tsx +++ b/packages/dashboard/app/components/SettingsModal.tsx @@ -464,6 +464,12 @@ function resolveMaxAutoMergeRetriesForSettingsForm(settings?: { maxAutoMergeRetr return Number.isFinite(configured) && configured > 0 ? Math.floor(configured) : 3; } +/** FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: zero is an intentional project setting that disables retries/backoff, so do not use a truthy fallback when normalizing the form. */ +function resolveNonNegativeExecutorToolFailureSetting(value: unknown, fallback: number): number { + const configured = Number(value); + return Number.isFinite(configured) && configured >= 0 ? Math.floor(configured) : fallback; +} + export const SETTINGS_SECTIONS: SettingsSection[] = [ { id: "__preferences_header", label: "Preferences", labelKey: "settings.nav.preferencesHeader", scope: undefined, isGroupHeader: true }, { id: "appearance", label: "Appearance", labelKey: "settings.nav.appearance", scope: "global", searchableText: ["theme", "color", "sidebar", "dock", "task popup", "task popups", "board list popups", "popup view attachment", "open tasks as popups", "quick chat"] }, @@ -1127,6 +1133,9 @@ export function SettingsModal({ planApprovalMode: "auto-approve-all", mergeStrategy: "direct", maxAutoMergeRetries: 3, + executorToolFailureRetryCount: 2, + executorToolFailureRetryBackoffMs: 2000, + executorToolFailureThreshold: 3, mergeIntegrationWorktree: "reuse-task-worktree", mergeAdvanceAutoSync: "stash-and-ff", merger: { mode: "ai", maxReviewPasses: 3, allowDirtyLocalCheckoutSync: true }, @@ -1708,6 +1717,9 @@ export function SettingsModal({ mergeIntegrationWorktree: normalizeMergeIntegrationWorktreeMode(s.mergeIntegrationWorktree), mergeAdvanceAutoSync: normalizeMergeAdvanceAutoSyncMode(s.mergeAdvanceAutoSync), maxAutoMergeRetries: resolveMaxAutoMergeRetriesForSettingsForm(s), + executorToolFailureRetryCount: resolveNonNegativeExecutorToolFailureSetting(s.executorToolFailureRetryCount, 2), + executorToolFailureRetryBackoffMs: resolveNonNegativeExecutorToolFailureSetting(s.executorToolFailureRetryBackoffMs, 2000), + executorToolFailureThreshold: Math.max(1, Math.floor(Number(s.executorToolFailureThreshold ?? 3) || 3)), worktreeCopyFiles: Array.isArray(s.worktreeCopyFiles) ? s.worktreeCopyFiles : [], }; setForm(normalizedSettings); @@ -3321,6 +3333,9 @@ export function SettingsModal({ onFailure: form.worktrunk?.onFailure ?? "fail", }, maxAutoMergeRetries: resolveMaxAutoMergeRetriesForSettingsForm(form), + executorToolFailureRetryCount: resolveNonNegativeExecutorToolFailureSetting(form.executorToolFailureRetryCount, 2), + executorToolFailureRetryBackoffMs: resolveNonNegativeExecutorToolFailureSetting(form.executorToolFailureRetryBackoffMs, 2000), + executorToolFailureThreshold: Math.max(1, Math.floor(Number(form.executorToolFailureThreshold ?? 3) || 3)), taskPrefix: form.taskPrefix?.trim() || undefined, githubTrackingDefaultRepo: form.githubTrackingDefaultRepo?.trim() || undefined, /* diff --git a/packages/dashboard/app/components/settings/section-keys.ts b/packages/dashboard/app/components/settings/section-keys.ts index 170ff95a90..3e773f150a 100644 --- a/packages/dashboard/app/components/settings/section-keys.ts +++ b/packages/dashboard/app/components/settings/section-keys.ts @@ -115,6 +115,9 @@ const PROJECT_SECTION_KEYS: Record = { "autoArchiveDoneAfterMs", "autoArchiveDoneTasksEnabled", "engineerBacklogAutoClaim", + "executorToolFailureRetryCount", + "executorToolFailureRetryBackoffMs", + "executorToolFailureThreshold", "groupOverlappingFiles", "heartbeatScopeDiscipline", "ignoreHiddenOverlapPaths", diff --git a/packages/dashboard/app/components/settings/sections/SchedulingSection.search.ts b/packages/dashboard/app/components/settings/sections/SchedulingSection.search.ts index 997a59c486..1c8fd7d4db 100644 --- a/packages/dashboard/app/components/settings/sections/SchedulingSection.search.ts +++ b/packages/dashboard/app/components/settings/sections/SchedulingSection.search.ts @@ -39,6 +39,33 @@ export const schedulingSearchEntries: SettingsSearchEntry[] = [ helpFallback: "Maximum concurrent planning agents. Default: 2.", keywords: ["parallelism", "capacity", "spec"], }, + { + sectionId: "scheduling", + key: "executorToolFailureRetryCount", + labelKey: "settings.scheduling.executorToolFailureRetryCount", + labelFallback: "Executor tool-failure retries", + helpKey: "settings.scheduling.executorToolFailureRetryCountHelp", + helpFallback: "Same-model retries after consecutive tool-call failures. Set 0 to disable. Default: 2.", + keywords: ["executor", "tool error", "auto retry", "same model", "failure"], + }, + { + sectionId: "scheduling", + key: "executorToolFailureRetryBackoffMs", + labelKey: "settings.scheduling.executorToolFailureRetryBackoffMs", + labelFallback: "Tool-failure retry backoff (ms)", + helpKey: "settings.scheduling.executorToolFailureRetryBackoffMsHelp", + helpFallback: "Unref'd wait before retrying. Default: 2000.", + keywords: ["executor", "delay", "wait", "auto retry", "tool error"], + }, + { + sectionId: "scheduling", + key: "executorToolFailureThreshold", + labelKey: "settings.scheduling.executorToolFailureThreshold", + labelFallback: "Consecutive tool failures", + helpKey: "settings.scheduling.executorToolFailureThresholdHelp", + helpFallback: "Terminal tool errors required before retrying. Default: 3.", + keywords: ["executor", "tool error", "threshold", "auto retry"], + }, { sectionId: "scheduling", key: "pollIntervalMs", diff --git a/packages/dashboard/app/components/settings/sections/SchedulingSection.tsx b/packages/dashboard/app/components/settings/sections/SchedulingSection.tsx index 43e7009863..d5a6620cd3 100644 --- a/packages/dashboard/app/components/settings/sections/SchedulingSection.tsx +++ b/packages/dashboard/app/components/settings/sections/SchedulingSection.tsx @@ -33,6 +33,10 @@ export function SchedulingSection({ form, setForm, concurrencyLoading = false, o const { t } = useTranslation("app"); return (<>

{t("settings.scheduling.scheduling", "Scheduling")}

+ {/* FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: project controls tune the bounded same-model retry before terminal executor parking; values floor to core's resolver contract. */} + setForm((f) => ({ ...f, executorToolFailureRetryCount: Math.max(0, Math.floor(v ?? 2)) } as SettingsFormState))} /> + setForm((f) => ({ ...f, executorToolFailureRetryBackoffMs: Math.max(0, Math.floor(v ?? 2000)) } as SettingsFormState))} /> + setForm((f) => ({ ...f, executorToolFailureThreshold: Math.max(1, Math.floor(v ?? 3)) } as SettingsFormState))} /> = { pollIntervalMs: "scheduling.pollIntervalMsHint", heartbeatScopeDiscipline: "scheduling.strictDefault", engineerBacklogAutoClaim: "scheduling.backlogNoTaskAutoClaimIsExecutorOnly", + executorToolFailureRetryCount: "scheduling.executorToolFailureRetryCountHelp", + executorToolFailureRetryBackoffMs: "scheduling.executorToolFailureRetryBackoffMsHelp", + executorToolFailureThreshold: "scheduling.executorToolFailureThresholdHelp", taskStuckTimeoutMs: "scheduling.timeoutInMinutesForDetectingStuckTasksWhen", staleHighFanoutBlockerAgeThresholdMs: "scheduling.escalateHighFanOutBlockersOnlyAfterThey", preserveProgressOnStuckRequeue: "scheduling.whenTheStuckDetectorKillsAndReQueues", diff --git a/packages/engine/src/__tests__/executor-tool-failure-retry.test.ts b/packages/engine/src/__tests__/executor-tool-failure-retry.test.ts new file mode 100644 index 0000000000..cadf838213 --- /dev/null +++ b/packages/engine/src/__tests__/executor-tool-failure-retry.test.ts @@ -0,0 +1,160 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import type { TaskDetail } from "@fusion/core"; +import "./executor-test-helpers.js"; +import { TaskExecutor } from "../executor.js"; +import { createMockStore, resetExecutorMocks } from "./executor-test-helpers.js"; + +const now = "2026-07-16T00:00:00.000Z"; + +function makeTask(overrides: Partial = {}): TaskDetail { + return { + id: "FN-7996", + title: "Tool failure retry", + description: "Reproduce executor tool errors", + column: "in-progress", + dependencies: [], + steps: [{ name: "Implement", status: "in-progress" }], + currentStep: 0, + log: [], + branch: "fusion/fn-7996", + baseBranch: "main", + worktree: "/tmp/fusion-fn-7996", + status: null, + error: null, + paused: false, + userPaused: false, + toolFailureDetectorLogCursor: 0, + autoMerge: true, + mergeRetries: 0, + createdAt: now, + updatedAt: now, + ...overrides, + } as TaskDetail; +} + +function graphFailure() { + return { + disposition: "failed" as const, + outcome: "failure" as const, + visitedNodeIds: ["steps#0:step-execute"], + context: { "node:steps#0:step-execute:value": "failure" }, + }; +} + +function makeHarness(options: { retries: number; entries: Array<{ type: string }> }) { + const store = createMockStore(); + const task = makeTask(); + store.getTask.mockResolvedValue(task); + store.getSettings.mockResolvedValue({ + maxConcurrent: 2, + maxWorktrees: 4, + pollIntervalMs: 15_000, + autoMerge: true, + executorToolFailureRetryCount: options.retries, + executorToolFailureRetryBackoffMs: 0, + executorToolFailureThreshold: 3, + }); + store.getAgentLogCount = vi.fn().mockResolvedValue(options.entries.length); + store.getAgentLogs = vi.fn().mockResolvedValue(options.entries); + store.claimNextToolFailureRetry = vi.fn().mockResolvedValue({ outcome: "claimed", attempt: 1 }); + store.updateTaskAtomic = vi.fn(async (_id: string, updater: (current: TaskDetail) => Partial | null) => { + const updates = updater(task); + if (updates) Object.assign(task, updates); + return task; + }); + store.markToolFailureRetryExhaustedAudit = vi.fn().mockResolvedValue(true); + store.recordRunAuditEvent = vi.fn().mockResolvedValue(undefined); + const executor = new TaskExecutor(store, "/tmp/test"); + (executor as any).graphToolFailureRunCursors.set(task.id, 0); + return { executor, store, task }; +} + +describe("executor consecutive tool-failure retry (FN-7996)", () => { + beforeEach(() => { + resetExecutorMocks(); + vi.useFakeTimers(); + }); + + afterEach(() => vi.useRealTimers()); + + it("retries a qualifying terminal step failure and records metadata-only audit evidence", async () => { + const { executor, store, task } = makeHarness({ + retries: 2, + entries: [{ type: "tool_error" }, { type: "tool_error" }, { type: "tool_error" }], + }); + const execute = vi.spyOn(executor as any, "execute").mockResolvedValue(undefined); + + await (executor as any).handleGraphFailure(task, graphFailure()); + await vi.advanceTimersByTimeAsync(0); + + expect(execute).toHaveBeenCalledWith(task); + expect(store.updateTask).not.toHaveBeenCalledWith(task.id, expect.objectContaining({ status: "failed" }), expect.anything()); + expect(store.updateTask).not.toHaveBeenCalledWith(task.id, expect.objectContaining({ graphResumeRetryCount: expect.anything() }), expect.anything()); + expect(store.recordRunAuditEvent).toHaveBeenCalledWith(expect.objectContaining({ + mutationType: "task:execution-tool-failure-retry", + metadata: { + taskId: task.id, + nodeId: "steps#0:step-execute", + attempt: 1, + maxAttempts: 2, + consecutiveToolFailures: 3, + mode: "same-model", + }, + })); + }); + + it("parks unchanged after a spent retry budget and emits one exhaustion audit", async () => { + const { executor, store, task } = makeHarness({ + retries: 2, + entries: [{ type: "tool_error" }, { type: "tool_error" }, { type: "tool_error" }], + }); + store.claimNextToolFailureRetry.mockResolvedValue({ outcome: "exhausted" }); + + await (executor as any).handleGraphFailure(task, graphFailure()); + + expect(store.recordRunAuditEvent).toHaveBeenCalledWith(expect.objectContaining({ + mutationType: "task:execution-tool-failure-retry-exhausted", + metadata: expect.objectContaining({ taskId: task.id, attempts: 2, limit: 2, outcome: "terminal-park" }), + })); + expect(store.updateTaskAtomic).toHaveBeenCalledWith(task.id, expect.any(Function), undefined); + expect(task).toMatchObject({ + status: "failed", + error: "Workflow graph terminated with failure at node 'steps#0:step-execute'", + }); + }); + + it("does not let an exhausted stale handler park a newer cursor-owned run", async () => { + const { executor, store, task } = makeHarness({ + retries: 2, + entries: [{ type: "tool_error" }, { type: "tool_error" }, { type: "tool_error" }], + }); + store.claimNextToolFailureRetry.mockResolvedValue({ outcome: "exhausted" }); + const newRun = makeTask({ toolFailureDetectorLogCursor: 99 }); + store.updateTaskAtomic.mockImplementation(async (_id: string, updater: (current: TaskDetail) => Partial | null) => { + // A new execution captured its own log cursor after the old run's claim exhausted. + expect(updater(newRun)).toBeNull(); + return newRun; + }); + + await (executor as any).handleGraphFailure(task, graphFailure()); + + expect(newRun).toMatchObject({ status: null, error: null, toolFailureDetectorLogCursor: 99 }); + expect(store.markToolFailureRetryExhaustedAudit).not.toHaveBeenCalled(); + expect(store.recordRunAuditEvent).not.toHaveBeenCalledWith(expect.objectContaining({ + mutationType: "task:execution-tool-failure-retry-exhausted", + })); + expect(store.updateTask).not.toHaveBeenCalledWith(task.id, expect.objectContaining({ status: "failed" }), expect.anything()); + }); + + it("preserves the immediate legacy park when disabled or errors are not consecutive", async () => { + const disabled = makeHarness({ retries: 0, entries: [{ type: "tool_error" }, { type: "tool_error" }, { type: "tool_error" }] }); + await (disabled.executor as any).handleGraphFailure(disabled.task, graphFailure()); + expect(disabled.store.claimNextToolFailureRetry).not.toHaveBeenCalled(); + expect(disabled.store.updateTask).toHaveBeenCalledWith(disabled.task.id, expect.objectContaining({ status: "failed" }), undefined); + + const interleaved = makeHarness({ retries: 2, entries: [{ type: "tool_error" }, { type: "tool_result" }, { type: "tool_error" }, { type: "tool_error" }] }); + await (interleaved.executor as any).handleGraphFailure(interleaved.task, graphFailure()); + expect(interleaved.store.claimNextToolFailureRetry).not.toHaveBeenCalled(); + expect(interleaved.store.updateTask).toHaveBeenCalledWith(interleaved.task.id, expect.objectContaining({ status: "failed" }), undefined); + }); +}); diff --git a/packages/engine/src/executor.ts b/packages/engine/src/executor.ts index 385fdc5d6e..a8df3fc7a9 100644 --- a/packages/engine/src/executor.ts +++ b/packages/engine/src/executor.ts @@ -13,7 +13,7 @@ import { existsSync, lstatSync, realpathSync } from "node:fs"; import { readFile, rm, writeFile } from "node:fs/promises"; import type { TaskStore, Task, TaskDetail, TaskTokenUsage, StepStatus, Settings, WorkflowStep, MissionStore, AsyncMissionStore, Slice, AgentState, AgentCapability, RunMutationContext, AgentHeartbeatConfig, Agent, AgentMemoryInclusionMode, ProjectSettings, MergeResult, WorkflowIrNode, WorkflowIrNodeKind, WorkflowStepResult as CoreWorkflowStepResult, ThinkingLevel } from "@fusion/core"; import { getUnmetSchedulingDependencies } from "./scheduler.js"; -import { RetryStormError, serializeRetryStormError, isExperimentalFeatureEnabled, resolveWorkflowIrForTask, resolveColumnAgentBinding, resolveEffectiveAgent, instanceNodeId, getWorkflowExtensionRegistry, getBuiltinWorkflow, parseNoOpCompletionMarker, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, isLiveSharedBranchGroupMemberIntegration, resolveMaxAutoMergeRetries, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, COMPLETION_SUMMARY_NODE_ID, upsertWorkflowStepResult, AWAITING_APPROVAL_PAUSE_REASON, THINKING_LEVELS, AgentStore, resolveExecutorFallbackModel } from "@fusion/core"; +import { RetryStormError, serializeRetryStormError, isExperimentalFeatureEnabled, resolveWorkflowIrForTask, resolveColumnAgentBinding, resolveEffectiveAgent, instanceNodeId, getWorkflowExtensionRegistry, getBuiltinWorkflow, parseNoOpCompletionMarker, allowsAutoMergeProcessing, resolveEffectiveAutoMerge, isLiveSharedBranchGroupMemberIntegration, resolveMaxAutoMergeRetries, resolveMaxConsecutiveToolFailureRetries, resolveConsecutiveToolFailureRetryBackoffMs, resolveConsecutiveToolFailureThreshold, resolveOptionalStepRevisionBudget, resolveOptionalReviewRevisionBudget, COMPLETION_SUMMARY_NODE_ID, upsertWorkflowStepResult, AWAITING_APPROVAL_PAUSE_REASON, THINKING_LEVELS, AgentStore, resolveExecutorFallbackModel } from "@fusion/core"; import { finalizeProvenAutoMergeTask } from "./auto-merge-finalization.js"; import { mergeEffectiveSettings } from "./effective-settings.js"; import { moveTaskToReplanColumn, resolveReplanTargetColumn } from "./replan-target.js"; @@ -5022,6 +5022,8 @@ export class TaskExecutor { * steps, no review handoff) and hands control back to the graph runner. * Doubles as the re-entrancy guard for graph routing. */ private graphCompletionInterceptors = new Map void>(); + /** Per graph-run agent-log boundary; passed to failure handling rather than trusting stale task snapshots. */ + private graphToolFailureRunCursors = new Map(); /** Step-inversion (KTD-2/KTD-8, U6/U8): graph-owned step-execute can pin * step-session physics for workflows that need a hard per-step boundary @@ -5145,6 +5147,16 @@ export class TaskExecutor { workflowGraphExecutor graduated from Experimental. Every task routes through the graph runner by default, and stale persisted experimentalFeatures.workflowGraphExecutor=false values are ignored so the product no longer has a user-facing or runtime graph-engine kill switch. */ settings = { ...settings }; + /* + * FNXC:ExecutorToolFailureRetry 2026-07-16-12:00: + * Capture a count cursor without reading the task log. Failure handling receives this + * execution-local boundary, so a stale task snapshot cannot accidentally qualify an old run. + */ + if (resolveMaxConsecutiveToolFailureRetries(settings) > 0) { + const cursor = await this.store.getAgentLogCount(task.id); + this.graphToolFailureRunCursors.set(task.id, cursor); + await this.store.updateTask(task.id, { toolFailureDetectorLogCursor: cursor }, this.getRunContextFor(task.id)); + } let selection: { workflowId: string; stepIds: string[] } | undefined; if ( typeof this.store.getTaskWorkflowSelectionAsync !== "function" @@ -5406,8 +5418,8 @@ export class TaskExecutor { if ((live as TaskDetail).mergeDetails?.mergeConfirmed === true && (live as TaskDetail).column !== "done") { await this.finalizeMergeConfirmedWorkflowGraphTask(task.id, "graph-completed"); } - if ((live.graphResumeRetryCount ?? 0) !== 0) { - await this.store.updateTask(task.id, { graphResumeRetryCount: 0 }, this.getRunContextFor(task.id)); + if ((live.graphResumeRetryCount ?? 0) !== 0 || (live.consecutiveToolFailureRetryCount ?? 0) !== 0) { + await this.store.updateTask(task.id, { graphResumeRetryCount: 0, consecutiveToolFailureRetryCount: 0, toolFailureDetectorLogCursor: null, toolFailureRetryExhaustedAuditEmitted: false }, this.getRunContextFor(task.id)); } } return true; @@ -5442,6 +5454,7 @@ export class TaskExecutor { this.activeWorkflowGraphAbortControllers.delete(task.id); } this.graphRouting.delete(task.id); + this.graphToolFailureRunCursors.delete(task.id); // Clear per-run step-inversion pins (KTD-8: pinned only for the run's life). this.graphStepSessionPinned.delete(task.id); this.graphStepRunOnce.delete(task.id); @@ -9081,6 +9094,24 @@ export class TaskExecutor { return true; } + private async hasTrailingConsecutiveToolFailures(taskId: string, cursor: number | null | undefined, threshold: number): Promise { + if (cursor == null) return false; + const currentCount = await this.store.getAgentLogCount(taskId); + if (currentCount <= cursor) return false; + const entries = await this.store.getAgentLogs(taskId, { limit: currentCount - cursor }); + let failures = 0; + for (let index = entries.length - 1; index >= 0; index -= 1) { + const type = entries[index]!.type; + if (type === "tool_result") return false; + if (type === "tool_error") { + failures += 1; + if (failures >= threshold) return true; + } + // Invocation markers and non-completion entries intentionally do not reset the run. + } + return false; + } + /** Terminal failure of a graph run: record the error and park the task in * review so a human can act — never leave it invisible in in-progress. */ private async handleGraphFailure(task: Task, result: WorkflowGraphTaskRunResult): Promise { @@ -9659,6 +9690,56 @@ export class TaskExecutor { return; } const message = `Workflow graph terminated with failure at node '${failedNode ?? "unknown"}'`; + const maxToolFailureRetries = resolveMaxConsecutiveToolFailureRetries(await this.store.getSettings()); + const isExecuteFailure = failedNode === "execute" || failedNode?.endsWith(":step-execute") === true || failedNode === "step-execute"; + if (maxToolFailureRetries > 0 && isExecuteFailure && !live.paused && !live.userPaused && !live.deletedAt && live.column === "in-progress") { + // Prefer the execution-local boundary; recovery paths refetch durable state rather than use the stale failure snapshot. + const cursor = this.graphToolFailureRunCursors.get(task.id) ?? (await this.store.getTask(task.id))?.toolFailureDetectorLogCursor; + const threshold = resolveConsecutiveToolFailureThreshold(await this.store.getSettings()); + if (await this.hasTrailingConsecutiveToolFailures(task.id, cursor, threshold)) { + const claim = await this.store.claimNextToolFailureRetry(task.id, cursor!, maxToolFailureRetries); + if (claim.outcome === "claimed") { + await this.store.updateTask(task.id, { status: null, error: null }, this.getRunContextFor(task.id)); + await this.store.logEntry(task.id, `Consecutive tool-call failures — auto-retrying same model (${claim.attempt}/${maxToolFailureRetries}) instead of parking`, undefined, this.getRunContextFor(task.id)); + await this.store.recordRunAuditEvent?.({ taskId: task.id, agentId: "executor", runId: generateSyntheticRunId("tool-failure-retry", task.id), domain: "database", mutationType: "task:execution-tool-failure-retry", target: task.id, metadata: { taskId: task.id, nodeId: failedNode ?? "unknown", attempt: claim.attempt, maxAttempts: maxToolFailureRetries, consecutiveToolFailures: threshold, mode: "same-model" } }); + const schedule = () => { void (async () => { const resume = await this.store.getTask(task.id); if (resume && !resume.deletedAt && !resume.paused && !resume.userPaused && resume.column === "in-progress") await this.execute(resume); })().catch((error) => executorLog.error(`${task.id}: tool-failure retry failed`, error)); }; + const delay = resolveConsecutiveToolFailureRetryBackoffMs(await this.store.getSettings()); + setTimeout(schedule, delay).unref?.(); + return; + } + if (claim.outcome === "already-claimed-for-run") { await this.store.getTask(task.id); return; } + /* + FNXC:ExecutorToolFailureRetry 2026-07-16-20:45: + Exhaustion belongs to the graph run that supplied `cursor`, not a later run + that may have begun while this handler awaited its durable claim. Revalidate + the cursor under TaskStore's per-task atomic lock while applying the terminal + state; only that successful CAS may emit the exhaustion audit. This keeps an + old terminal handler from parking a newer in-progress executor run. + */ + let cursorOwnedTerminalPark = false; + await this.store.updateTaskAtomic(task.id, (current) => { + if ( + current.toolFailureDetectorLogCursor !== cursor + || current.column !== "in-progress" + || current.paused + || current.userPaused + || current.deletedAt + ) { + return null; + } + cursorOwnedTerminalPark = true; + return { error: message, status: "failed" }; + }, this.getRunContextFor(task.id)); + if (!cursorOwnedTerminalPark) return; + if (await this.store.markToolFailureRetryExhaustedAudit(task.id)) { + await this.store.recordRunAuditEvent?.({ taskId: task.id, agentId: "executor", runId: generateSyntheticRunId("tool-failure-retry-exhausted", task.id), domain: "database", mutationType: "task:execution-tool-failure-retry-exhausted", target: task.id, metadata: { taskId: task.id, nodeId: failedNode ?? "unknown", attempts: maxToolFailureRetries, limit: maxToolFailureRetries, outcome: "terminal-park" } }); + } + executorLog.warn(`${task.id}: ${message}`); + await this.store.logEntry(task.id, message, undefined, this.getRunContextFor(task.id)); + await this.persistTokenUsage(task.id); + return; + } + } executorLog.warn(`${task.id}: ${message}`); await this.store.logEntry(task.id, message, undefined, this.getRunContextFor(task.id)); // status "failed" doubles as the self-healing exemption: review-task diff --git a/packages/i18n/locales/en/app.json b/packages/i18n/locales/en/app.json index b650582490..be608e7d92 100644 --- a/packages/i18n/locales/en/app.json +++ b/packages/i18n/locales/en/app.json @@ -6628,6 +6628,12 @@ "enableAutomaticTaskArchiving": " Enable automatic task archiving ", "enablePlanStalenessEnforcement": " Enable plan staleness enforcement ", "escalateHighFanOutBlockersOnlyAfterThey": "Escalate high fan-out blockers only after they remain in in-progress or in-review for this many hours (age source: columnMovedAt, fallback updatedAt). Default: 2 hours.", + "executorToolFailureRetryCount": "Executor tool-failure retries", + "executorToolFailureRetryCountHelp": "Same-model retries after consecutive tool-call failures. Set 0 to disable. Default: 2.", + "executorToolFailureRetryBackoffMs": "Tool-failure retry backoff (ms)", + "executorToolFailureRetryBackoffMsHelp": "Unref'd wait before retrying. Default: 2000.", + "executorToolFailureThreshold": "Consecutive tool failures", + "executorToolFailureThresholdHelp": "Terminal tool errors required before retrying. Default: 3.", "fullAgentLog": "Full agent log", "globalMaxConcurrent": "Global Max Concurrent", "heartbeatScopeDiscipline": "Heartbeat Scope Discipline",