diff --git a/AGENTS.md b/AGENTS.md index 48ed81930d..592cbe738d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -226,6 +226,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme - FN-7787: `createResolvedAgentSession` enriches `session:runtime-resolved` with `noModelResolved: true` and `runtimeBuiltInFallbackModel` when a non-mock/non-test session reaches runtime creation without a complete provider/model pair; this is a visibility signal for runtime built-in fallback usage, not a fabricated model-resolution verdict. - FN-7835/FN-7844/FN-7859: durable-agent error-state recovery emits `agent:auto-recover-error-state` when either the heartbeat timer or the self-healing sweep clears a transient, non-operator-actionable `error` and retries; metadata stays ids/counts/outcomes-only (`agentId`, attempt, limit, source), where `source` is `timer`/`automation`/`self-healing`. Both entry paths share the `heartbeatErrorRecovery` budget (self-healing keeps `durableErrorRecovery` only for cooldown/stale-path bookkeeping) and emit `agent:error-retry-exhausted` when the shared budget is exhausted and the agent is parked `paused` with `pauseReason:"error-retry-exhausted"`. Non-recoverable durable heartbeat errors (permanent/usage-limit/operator-actionable, excluding stale worktree/module-resolution suppression and disabled/ephemeral/actively-executing agents) emit `agent:error-parked-unrecoverable` with ids/counts/outcomes-only metadata (`agentId`, `source`, optional `attempts`, `limit`) and park `paused` with `pauseReason:"error-unrecoverable"` for human repair. - FN-7802: self-healing emits `task:reconcile-missing-worktree-merge-active` when it proves an `in-review` merge-active task (`merging`/`merging-pr`/`merging-fix`) is stranded by an unusable-worktree session-start failure, clears stale `worktree`/`branch`/`sessionFile`, resets the worktree-session retry budget, increments `recoveryRetryCount` as the bounded stale-metadata clear counter, and requeues to `todo`; it emits `task:reconcile-missing-worktree-merge-active-no-action` when `autoMerge:false`, workspace-task ownership, or triple-proof blocks the backward move. +- FN-7863: executor emits `task:execution-dispatch-loop-terminalized` when an execute-node self-requeue loop reaches `MAX_EXECUTE_REQUEUE_LOOP_CYCLES` with an unchanged progress signature; metadata stays ids/counts/outcomes-only (`taskId`, `cycleCount`, `maxCycles`, `progressSignature`, `failureValue`) and the task is visibly failed with `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` while preserving worktree/branch/step progress. - FN-7011: self-healing emits `task:reconcile-engine-downtime-active-timing` when startup recovery shifts active task segment anchors to exclude proven engine-process downtime, and `task:reconcile-engine-downtime-active-timing-no-action` when no active task qualifies. - FN-5419: git run-audit now includes `pull:fast-forward` and `stash:pop-conflict`; dashboard git surfaces now include the extended `POST /api/git/pull` integration-worktree path plus companion `POST /api/git/stash-resolve`, `POST /api/git/stash-drop`, and `POST /api/git/stash-apply` routes. - FN-6292: self-healing emits `task:reconcile-dependency-blocking-lease` when it rebounds an in-progress holder whose stale file-scope lease blocks an unmet dependency, and `task:reconcile-dependency-blocking-lease-no-action` when triple-proof blocks that backward move. diff --git a/docs/architecture.md b/docs/architecture.md index d46327542b..b015628a46 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -687,6 +687,7 @@ Runtime action-gate flow (v1): - `recoverGhostReviewTasks()` is a fallback only for idle, non-terminal `in-review` states. Terminal/actionable states (notably `status: "failed"`) are preserved and **not** auto-kicked back to `todo`. - `recoverPausedAbortFailures()` clears executor pause/resume abort parks only when the durable row is safe to recover. `todo`/`in-progress` rows are requeued for normal scheduling, while clean `in-review` rows (completed steps, not paused/user-paused/executing, auto-merge eligible, no confirmed or terminal merge evidence) have `status`/`error` cleared in place so review progression can continue. FN-7749 adds the manual-hold exception to the prior `autoMerge:false` guard: a benign hard-cancel pause/resume abort at a merge-region/manual-hold node is the healthy Merge & Close resting state, so already-parked rows of that exact shape are cleared in place without moving backward (FN-5147-compliant). User hard-cancel, global/user pause, terminal merge, live-execution, and other `autoMerge:false` guards remain operator-actionable. Successful recovery emits `task:auto-recover-paused-abort-park` with `preservedInReview` metadata. - Workflow graph pause/resume is node-reentrant for typed engine-internal interruptions. When `WorkflowGraphExecutor` sees the graph abort signal or a node returns `value: "aborted"`, it stamps the interrupted node and `engine-pause` abort kind into graph context. `TaskExecutor` then uses the existing bounded `graphResumeRetryCount` budget to clear the transient abort, suppress failure notification with an `Auto-recovered:` task log, and re-enter the graph/task only under the same safety guards: no user/active global pause, no merge/finalize provenance, no genuine node failure, no terminal merge value, no `autoMerge:false` protected review row, and no active execution owner. Global-pause provenance from the graph-controller abort is re-entrant once the global pause has been lifted because it represents the same in-flight node interruption. Generic legacy pause-abort parks without the typed node marker remain operator-action failures except for the narrow `in-review`/`plan` stale-replay shape: hard-cancel pause provenance, `node:plan:value === "aborted"`, no typed interrupted node, no active task/user/global pause, no terminal merge value, no confirmed merge, auto-merge eligibility, and only a clean row or the exact stale plan pause-abort failure. That path logs `stale replay ignored`, clears only the stale failure state when present, preserves `in-review`, and never re-enters planning or moves the task to `todo`. + - FN-7863 adds a progress-anchored guard at the execute-node self-requeue funnel (`failedNode === "execute"` and either the live row is `todo` or the in-process self-requeue marker proves a stale `in-progress` read). The guard persists `executeRequeueLoopCount` plus `executeRequeueLoopSignature` (`currentStep` + step statuses), warns at `EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD = 3`, and terminalizes at `MAX_EXECUTE_REQUEUE_LOOP_CYCLES = 6` with `status:"failed"` and an `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` error. It preserves worktree/branch/step progress, skips paused/user-paused/done/archived terminalization, resets on signature progress, manual retry, successful completion, forward moves, and unpause, and complements rather than replaces the scheduler's wall-clock `dispatchStormCount` guard (fast flapping stays scheduler-owned; slow no-progress flapping is caught here). - `reattach-orphaned-assigned-executions` is a forward-resume safety net for durable-agent assignments. During startup recovery and periodic maintenance, after orphaned-agent and stale-heartbeat-run repairs, self-healing finds `in-progress` tasks with an `assignedAgentId` whose agent has no active heartbeat run and no active executor session after the orphan grace window. It re-dispatches in place via `executor.resumeTaskForAgent(agentId)` (the same seam used by clean `HeartbeatMonitor.onRunCompleted` and guarded by executor double-execution checks), emits `task:reattach-orphaned-execution`, and never moves the task backward. This complements engine-start `executor.resumeOrphaned()` and leaves unassigned/role-based execution recovery to the existing startup/limbo/stuck-task paths. - Durable `Agent.taskId` is a running assignment for parked `todo`/`triage` task rows only when the agent has live proof: a fresh active heartbeat run or an executor-active/tracked heartbeat signal. Scheduler overlap requeues, task move sync, self-healing, and Reports Health Check share this invariant: stale durable links are cleared or rendered as stale while `status: "queued"` and `overlapBlockedBy` remain on the task row so file-scope lease blocking is not weakened. `fn_list_agents` and `fn_agent_show` render the linked task column next to `Current Task` (for example `Current Task: FN-1234 (triage)` or `Current Task: FN-1234 (not active — done)`) so parked-column planning ownership is not misread as in-progress execution drift. - Mission validation has a dedicated stale-run reaper: startup recovery and Batch 2 maintenance call `reapStaleMissionValidatorRuns()` when wired by the runtime, using `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). The sweep terminates ownerless `mission_validator_runs.status='running'` rows as `error`, writes the reap reason into `summary`, leaves `lastValidatorRunId` pointing at the now-terminal run, and emits run-audit telemetry with `mutationType: "mission:validator-run-reaped"` plus `runId`/`featureId`/`missionId`/`triggerType`/`elapsedMs` metadata. Active mission features move to `loopState="needs_fix"` + `lastValidatorStatus="error"` unless their parent mission is already `complete`/`archived`. @@ -749,6 +750,7 @@ Guardrails: this routine does **not** retry merges, does **not** apply to mixed/ - FN-6782/FN-6796: `task:auto-recover-paused-abort-park` records self-healing recovery of pause-abort operator parks. Metadata includes the source column and whether recovery preserved a clean `in-review` row instead of requeueing to `todo`. - FN-7069: `task:reconcile-phantom-committed-reservation` records task-store startup or self-healing cleanup of committed-reservation-without-task phantoms. Metadata includes `reservationStatus: "committed"` plus pruned `activityLog` and `agents` counts; `runAuditEvents` and the committed reservation are intentionally retained for auditability and ID permanence. - FN-7074: `task:reservation-commit-rolled-back` records preventive create-path rollback when a distributed reservation was committed with the task-row insert but a later create materialization step failed. Metadata includes `{ reservationId, nodeId, reason: "failed-create", error }`; the task row/partial directory are removed and the reservation is moved to `aborted` so FN-7069 should not need to clean up a new phantom. + - FN-7863: `task:execution-dispatch-loop-terminalized` records executor terminalization of a no-progress execute-node self-requeue loop. Metadata includes `{ taskId, cycleCount, maxCycles, progressSignature, failureValue }`; the task row carries `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` and remains in its visible failed state with committed/step progress preserved. - FN-4956: Layer 3 merge-conflict arbitration now scope-partitions conflicted files before AI resolution. Out-of-scope conflicts are deterministically resolved to the integration branch (`git checkout --ours`) and unstaged, while only in-scope conflicts flow to AI. Integration branch defaults are resolved via `resolveIntegrationBranch(rootDir, settings)`. Audit events: `merge:layer3:foreign-file-skipped` and `merge:layer3:scope-override-bypass`. - FN-5655 goal anchoring observability adds `database`-domain mutation types `goal:injection-applied`, `goal:injection-skipped`, and `goal:retrieval-invoked` so Slice 2 cite-rate tracking has a prompt-independent signal. Metadata uses counts/IDs only (`count`, `lane`, `toolName`, optional `truncated`/`reason`/`notFound`) and never stores prompt bodies or goal titles/descriptions. These events surface through `GET /api/agents/:id/runs/:runId/audit` and support the existing `startTime`/`endTime` filters. diff --git a/packages/core/src/__tests__/store-persistence.test.ts b/packages/core/src/__tests__/store-persistence.test.ts index de28919286..976442ea77 100644 --- a/packages/core/src/__tests__/store-persistence.test.ts +++ b/packages/core/src/__tests__/store-persistence.test.ts @@ -59,6 +59,51 @@ describe("TaskStore", () => { }); }); + /* + * FNXC:WorkflowLifecycle 2026-07-12-00:00: + * FN-7863's execute self-requeue loop guard is progress-anchored, so both the no-progress streak + * and the compact step-status signature must round-trip through SQLite and list surfaces. + */ + describe("executeRequeueLoop persistence", () => { + it("round-trips execute loop counters through updateTask, getTask, and listTasks", async () => { + const task = await harness.store().createTask({ description: "Execute requeue loop task" }); + const updated = await harness.store().updateTask(task.id, { + executeRequeueLoopCount: 3, + executeRequeueLoopSignature: JSON.stringify({ currentStep: 1, steps: ["done", "pending"] }), + }); + + expect(updated.executeRequeueLoopCount).toBe(3); + expect(updated.executeRequeueLoopSignature).toBe('{"currentStep":1,"steps":["done","pending"]}'); + + const detail = await harness.store().getTask(task.id); + expect(detail.executeRequeueLoopCount).toBe(3); + expect(detail.executeRequeueLoopSignature).toBe('{"currentStep":1,"steps":["done","pending"]}'); + + const listed = (await harness.store().listTasks()).find((candidate) => candidate.id === task.id); + expect(listed?.executeRequeueLoopCount).toBe(3); + expect(listed?.executeRequeueLoopSignature).toBe('{"currentStep":1,"steps":["done","pending"]}'); + }); + + it("clears execute loop state with explicit null updates", async () => { + const task = await harness.store().createTask({ description: "Execute requeue loop clear task" }); + await harness.store().updateTask(task.id, { + executeRequeueLoopCount: 2, + executeRequeueLoopSignature: "sig", + }); + + const cleared = await harness.store().updateTask(task.id, { + executeRequeueLoopCount: null, + executeRequeueLoopSignature: null, + }); + expect(cleared.executeRequeueLoopCount).toBeUndefined(); + expect(cleared.executeRequeueLoopSignature).toBeUndefined(); + + const detail = await harness.store().getTask(task.id); + expect(detail.executeRequeueLoopCount).toBe(0); + expect(detail.executeRequeueLoopSignature).toBeUndefined(); + }); + }); + /* * FNXC:PlanApproval 2026-07-04-22:41: * FN-7569 — approvedPlanFingerprint must survive create/update/null-clear round trips through diff --git a/packages/core/src/db.ts b/packages/core/src/db.ts index 488a215d60..3990d14e47 100644 --- a/packages/core/src/db.ts +++ b/packages/core/src/db.ts @@ -184,7 +184,7 @@ export function isFts5CorruptionError(error: unknown): boolean { // ── Schema Definition ──────────────────────────────────────────────── -const SCHEMA_VERSION = 141; +const SCHEMA_VERSION = 142; const TASKS_FTS_AUTOMERGE = 8; const TASKS_FTS_CRISISMERGE = 16; @@ -285,9 +285,11 @@ CREATE TABLE IF NOT EXISTS tasks ( mergeRetries INTEGER, workflowStepRetries INTEGER, resumeLimboCount INTEGER DEFAULT 0, + executeRequeueLoopCount INTEGER DEFAULT 0, graphResumeRetryCount INTEGER DEFAULT 0, resumeLimboTipSha TEXT, resumeLimboStepSignature TEXT, + executeRequeueLoopSignature TEXT, recoveryRetryCount INTEGER, taskDoneRetryCount INTEGER DEFAULT 0, worktreeSessionRetryCount INTEGER DEFAULT 0, @@ -5673,6 +5675,19 @@ export class Database { }); } + if (version < 142) { + /* + * FNXC:WorkflowLifecycle 2026-07-12-00:00: + * FN-7863 stores the progress-anchored execute self-requeue streak so slow + * execute→pause-abort→todo loops survive scheduler cadence gaps and terminalize + * visibly instead of burning executor slots indefinitely. + */ + this.applyMigration(142, () => { + this.addColumnIfMissing("tasks", "executeRequeueLoopCount", "INTEGER DEFAULT 0"); + this.addColumnIfMissing("tasks", "executeRequeueLoopSignature", "TEXT"); + }); + } + } /** diff --git a/packages/core/src/manual-retry-reset.ts b/packages/core/src/manual-retry-reset.ts index 903b89131f..97095d4a73 100644 --- a/packages/core/src/manual-retry-reset.ts +++ b/packages/core/src/manual-retry-reset.ts @@ -5,6 +5,7 @@ export const IN_REVIEW_STALL_DEADLOCK_PAUSE_REASON = "in-review-stall-deadlock"; export const MANUAL_RETRY_RESET_COUNTER_KEYS = [ "stuckKillCount", "resumeLimboCount", + "executeRequeueLoopCount", "graphResumeRetryCount", "recoveryRetryCount", "taskDoneRetryCount", diff --git a/packages/core/src/store.ts b/packages/core/src/store.ts index 2d937f76b5..2e32c4d2e1 100644 --- a/packages/core/src/store.ts +++ b/packages/core/src/store.ts @@ -251,9 +251,11 @@ interface TaskRow { workflowStepRetries: number | null; stuckKillCount: number | null; resumeLimboCount: number | null; + executeRequeueLoopCount: number | null; graphResumeRetryCount: number | null; resumeLimboTipSha: string | null; resumeLimboStepSignature: string | null; + executeRequeueLoopSignature: string | null; postReviewFixCount: number | null; recoveryRetryCount: number | null; taskDoneRetryCount: number | null; @@ -419,9 +421,11 @@ const TASK_COLUMN_DESCRIPTORS: TaskColumnDescriptor[] = [ defineTaskColumn("workflowStepRetries", (task) => task.workflowStepRetries ?? null), defineTaskColumn("stuckKillCount", (task) => task.stuckKillCount ?? 0), defineTaskColumn("resumeLimboCount", (task) => task.resumeLimboCount ?? 0), + defineTaskColumn("executeRequeueLoopCount", (task) => task.executeRequeueLoopCount ?? 0), defineTaskColumn("graphResumeRetryCount", (task) => task.graphResumeRetryCount === undefined ? 0 : task.graphResumeRetryCount), defineTaskColumn("resumeLimboTipSha", (task) => task.resumeLimboTipSha ?? null), defineTaskColumn("resumeLimboStepSignature", (task) => task.resumeLimboStepSignature ?? null), + defineTaskColumn("executeRequeueLoopSignature", (task) => task.executeRequeueLoopSignature ?? null), defineTaskColumn("postReviewFixCount", (task) => task.postReviewFixCount ?? 0), defineTaskColumn("recoveryRetryCount", (task) => task.recoveryRetryCount ?? null), defineTaskColumn("taskDoneRetryCount", (task) => task.taskDoneRetryCount ?? 0), @@ -2152,9 +2156,11 @@ export class TaskStore extends EventEmitter { workflowStepRetries: row.workflowStepRetries ?? undefined, stuckKillCount: row.stuckKillCount ?? undefined, resumeLimboCount: row.resumeLimboCount ?? undefined, + executeRequeueLoopCount: row.executeRequeueLoopCount ?? undefined, graphResumeRetryCount: row.graphResumeRetryCount ?? undefined, resumeLimboTipSha: row.resumeLimboTipSha || undefined, resumeLimboStepSignature: row.resumeLimboStepSignature || undefined, + executeRequeueLoopSignature: row.executeRequeueLoopSignature || undefined, postReviewFixCount: row.postReviewFixCount ?? undefined, recoveryRetryCount: row.recoveryRetryCount ?? undefined, taskDoneRetryCount: row.taskDoneRetryCount ?? undefined, @@ -2738,7 +2744,7 @@ export class TaskStore extends EventEmitter { "modelPresetId", "modelProvider", "modelId", "validatorModelProvider", "validatorModelId", "planningModelProvider", "planningModelId", - "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", + "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "executeRequeueLoopSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", "error", "summary", "thinkingLevel", "executionMode", "plannerOversightLevel", "awaitingApprovalReason", "approvedPlanFingerprint", "tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenUsageModelProvider", "tokenUsageModelId", "tokenUsagePerModel", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride", "createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "columnDwellMs", "executionStartedAt", "executionCompletedAt", @@ -2834,7 +2840,7 @@ export class TaskStore extends EventEmitter { "modelPresetId", "modelProvider", "modelId", "validatorModelProvider", "validatorModelId", "planningModelProvider", "planningModelId", - "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", + "mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "executeRequeueLoopSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt", "error", "summary", "thinkingLevel", "executionMode", "plannerOversightLevel", "awaitingApprovalReason", "approvedPlanFingerprint", "tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenUsageModelProvider", "tokenUsageModelId", "tokenUsagePerModel", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride", "createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "columnDwellMs", "executionStartedAt", "executionCompletedAt", @@ -8597,7 +8603,7 @@ ${TASK_UPSERT_SQL_ASSIGNMENTS} async updateTask( id: string, - updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; workspaceWorktrees?: import("./types.js").Task["workspaceWorktrees"]; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; customFields?: Record; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; plannerOversightLevel?: import("./types.js").PlannerOversightLevel | null; awaitingApprovalReason?: import("./types.js").Task["awaitingApprovalReason"] | null; approvedPlanFingerprint?: string | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; graphResumeRetryCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record | null; githubTracking?: import("./types.js").TaskGithubTracking | null; gitlabTracking?: (Omit & { item?: import("./types.js").TaskGitLabTrackedItem | null }) | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; workflowTransitionNotification?: import("./types.js").Task["workflowTransitionNotification"] | null; missionId?: string | null; sliceId?: string | null }, + updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; workspaceWorktrees?: import("./types.js").Task["workspaceWorktrees"]; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; customFields?: Record; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; plannerOversightLevel?: import("./types.js").PlannerOversightLevel | null; awaitingApprovalReason?: import("./types.js").Task["awaitingApprovalReason"] | null; approvedPlanFingerprint?: string | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; executeRequeueLoopCount?: number | null; graphResumeRetryCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; executeRequeueLoopSignature?: string | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record | null; githubTracking?: import("./types.js").TaskGithubTracking | null; gitlabTracking?: (Omit & { item?: import("./types.js").TaskGitLabTrackedItem | null }) | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; workflowTransitionNotification?: import("./types.js").Task["workflowTransitionNotification"] | null; missionId?: string | null; sliceId?: string | null }, runContext?: RunMutationContext, ): Promise { /* @@ -9343,6 +9349,11 @@ ${TASK_UPSERT_SQL_ASSIGNMENTS} } else if (updates.resumeLimboCount !== undefined) { task.resumeLimboCount = updates.resumeLimboCount; } + if (updates.executeRequeueLoopCount === null) { + task.executeRequeueLoopCount = undefined; + } else if (updates.executeRequeueLoopCount !== undefined) { + task.executeRequeueLoopCount = updates.executeRequeueLoopCount; + } if (updates.graphResumeRetryCount === null) { task.graphResumeRetryCount = null; } else if (updates.graphResumeRetryCount !== undefined) { @@ -9358,6 +9369,11 @@ ${TASK_UPSERT_SQL_ASSIGNMENTS} } else if (updates.resumeLimboStepSignature !== undefined) { task.resumeLimboStepSignature = updates.resumeLimboStepSignature; } + if (updates.executeRequeueLoopSignature === null) { + task.executeRequeueLoopSignature = undefined; + } else if (updates.executeRequeueLoopSignature !== undefined) { + task.executeRequeueLoopSignature = updates.executeRequeueLoopSignature; + } if (updates.postReviewFixCount === null) { task.postReviewFixCount = undefined; } else if (updates.postReviewFixCount !== undefined) { diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index 252ef11e04..3973133163 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -2452,6 +2452,14 @@ export interface Task { * Incremented by self-healing for resume-limbo detection and reset when * progress is observed or recovery escalates to a fresh todo dispatch. */ resumeLimboCount?: number; + /** + * FNXC:WorkflowLifecycle 2026-07-12-00:00: + * FN-7863 bounds execute-node self-requeue loops by counting consecutive requeues + * that preserve the same execution-progress signature. Reset this counter on real + * progress, forward moves, and manual retry; the executor caps it before writing + * terminal status:"failed" so committed work and step progress remain visible. + */ + executeRequeueLoopCount?: number; /** Bounded auto-retry attempts for transient workflow-graph failures observed * immediately after engine-restart or unpause resume. Reset by manual retry * and by successful forward progress; capped by the executor before terminal @@ -2463,6 +2471,9 @@ export interface Task { /** Compact execution-progress snapshot captured at the last reclaim/unpause * attempt (current step + step statuses) for resume-limbo detection. */ resumeLimboStepSignature?: string; + /** Compact execution-progress snapshot captured at the last execute-node + * self-requeue (current step + step statuses) for FN-7863 loop detection. */ + executeRequeueLoopSignature?: string; /** Number of times workflow remediation has auto-revived this task after * failed pre-merge review feedback. Incremented each time the engine sends the * task back with failure feedback injected. Capped only when the workflow step diff --git a/packages/engine/src/__tests__/reliability-interactions/execute-requeue-loop-guard.test.ts b/packages/engine/src/__tests__/reliability-interactions/execute-requeue-loop-guard.test.ts new file mode 100644 index 0000000000..2a6524504a --- /dev/null +++ b/packages/engine/src/__tests__/reliability-interactions/execute-requeue-loop-guard.test.ts @@ -0,0 +1,188 @@ +import { describe, expect, it, vi } from "vitest"; +import type { TaskDetail } from "@fusion/core"; +import "../executor-test-helpers.js"; +import { + EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD, + MAX_EXECUTE_REQUEUE_LOOP_CYCLES, + TaskExecutor, +} from "../../executor.js"; +import { createMockStore, resetExecutorMocks } from "../executor-test-helpers.js"; + +const now = "2026-07-12T00:00:00.000Z"; + +function task(overrides: Partial = {}): TaskDetail { + return { + id: "FN-7863-T", + title: "Execute requeue loop", + description: "Bound execute self-requeue loops", + column: "todo", + dependencies: [], + steps: [{ name: "Implement", status: "pending" }], + currentStep: 0, + log: [], + branch: "fusion/fn-7863", + baseBranch: "main", + worktree: "/tmp/fusion-fn-7863", + status: null, + error: null, + paused: false, + userPaused: false, + autoMerge: true, + mergeRetries: 0, + createdAt: now, + updatedAt: now, + ...overrides, + } as TaskDetail; +} + +function harness(initial: TaskDetail) { + resetExecutorMocks(); + const store = createMockStore(); + let live = { ...initial } as TaskDetail; + store.getTask.mockImplementation(async () => live); + store.updateTask.mockImplementation(async (_id: string, updates: Partial) => { + live = { ...live, ...updates } as TaskDetail; + return live; + }); + store.recordRunAuditEvent = vi.fn().mockResolvedValue(undefined); + const executor = new TaskExecutor(store, "/tmp/test"); + return { + store, + executor, + get live() { + return live; + }, + setLive(patch: Partial) { + live = { ...live, ...patch } as TaskDetail; + }, + }; +} + +async function failAtExecute(executor: TaskExecutor, taskSnapshot: TaskDetail) { + await (executor as any).handleGraphFailure(taskSnapshot, { + disposition: "failed", + outcome: "failure", + visitedNodeIds: ["execute"], + context: { "node:execute:value": "implementation-incomplete" }, + }); +} + +describe("execute requeue loop guard", () => { + it("terminalizes unchanged todo execute self-requeues and preserves progress", async () => { + const h = harness(task({ + id: "FN-7863-TODO", + column: "todo", + steps: [ + { name: "Preflight", status: "done" }, + { name: "Implement", status: "in-progress" }, + ], + currentStep: 1, + })); + + for (let i = 0; i < MAX_EXECUTE_REQUEUE_LOOP_CYCLES; i += 1) { + await failAtExecute(h.executor, h.live); + } + + expect(h.store.updateTask).toHaveBeenCalledWith( + "FN-7863-TODO", + expect.objectContaining({ + status: "failed", + error: expect.stringMatching(/^EXECUTION_DISPATCH_LOOP_EXHAUSTED:/), + executeRequeueLoopCount: MAX_EXECUTE_REQUEUE_LOOP_CYCLES, + }), + undefined, + ); + expect(h.store.recordRunAuditEvent).toHaveBeenCalledWith(expect.objectContaining({ + mutationType: "task:execution-dispatch-loop-terminalized", + metadata: expect.objectContaining({ + taskId: "FN-7863-TODO", + cycleCount: MAX_EXECUTE_REQUEUE_LOOP_CYCLES, + maxCycles: MAX_EXECUTE_REQUEUE_LOOP_CYCLES, + failureValue: "implementation-incomplete", + }), + })); + expect(h.store.moveTask).not.toHaveBeenCalled(); + const terminalUpdate = h.store.updateTask.mock.calls.find((call: any[]) => call[1]?.status === "failed")?.[1]; + expect(terminalUpdate).not.toHaveProperty("worktree"); + expect(terminalUpdate).not.toHaveProperty("branch"); + expect(terminalUpdate).not.toHaveProperty("steps"); + const lastLog = h.store.logEntry.mock.calls.at(-1)?.[1] as string; + expect(lastLog).toMatch(/^EXECUTION_DISPATCH_LOOP_EXHAUSTED:/); + expect(lastLog).not.toContain("executor recovery preserved"); + }); + + it("terminalizes the stale in-progress self-requeue marker path", async () => { + const h = harness(task({ id: "FN-7863-STALE", column: "in-progress" })); + (h.executor as any).graphRouting.add("FN-7863-STALE"); + (h.executor as any).markGraphExecuteSelfRequeued("FN-7863-STALE"); + + for (let i = 0; i < MAX_EXECUTE_REQUEUE_LOOP_CYCLES; i += 1) { + await failAtExecute(h.executor, h.live); + } + + expect(h.store.updateTask).toHaveBeenCalledWith( + "FN-7863-STALE", + expect.objectContaining({ status: "failed", error: expect.stringMatching(/^EXECUTION_DISPATCH_LOOP_EXHAUSTED:/) }), + undefined, + ); + }); + + it("resets the streak on real step progress and never terminalizes", async () => { + const h = harness(task({ id: "FN-7863-PROGRESS", column: "todo" })); + + for (let i = 0; i < MAX_EXECUTE_REQUEUE_LOOP_CYCLES + 3; i += 1) { + h.setLive({ currentStep: i, steps: [{ name: `Step ${i}`, status: i % 2 === 0 ? "pending" : "in-progress" }] as any }); + await failAtExecute(h.executor, h.live); + } + + expect(h.store.updateTask).not.toHaveBeenCalledWith( + "FN-7863-PROGRESS", + expect.objectContaining({ status: "failed" }), + expect.anything(), + ); + expect(h.live.executeRequeueLoopCount).toBe(1); + }); + + it("emits a visible warning at the threshold without terminalizing", async () => { + const h = harness(task({ id: "FN-7863-WARN", column: "todo" })); + + for (let i = 0; i < EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD; i += 1) { + await failAtExecute(h.executor, h.live); + } + + expect(h.store.logEntry).toHaveBeenCalledWith( + "FN-7863-WARN", + expect.stringContaining(`Execution dispatch loop building: ${EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD}/${MAX_EXECUTE_REQUEUE_LOOP_CYCLES}`), + undefined, + undefined, + ); + expect(h.store.updateTask).not.toHaveBeenCalledWith( + "FN-7863-WARN", + expect.objectContaining({ status: "failed" }), + expect.anything(), + ); + }); + + it.each([ + ["userPaused", { userPaused: true }], + ["paused", { paused: true }], + ])("does not terminalize %s tasks from the benign branch", async (_label, patch) => { + const h = harness(task({ id: `FN-7863-${_label}`, column: "todo", ...patch })); + + for (let i = 0; i < MAX_EXECUTE_REQUEUE_LOOP_CYCLES; i += 1) { + await failAtExecute(h.executor, h.live); + } + + expect(h.store.updateTask).not.toHaveBeenCalledWith( + h.live.id, + expect.objectContaining({ status: "failed" }), + expect.anything(), + ); + expect(h.store.logEntry).toHaveBeenCalledWith( + h.live.id, + expect.stringContaining("paused awaiting explicit unpause"), + undefined, + undefined, + ); + }); +}); diff --git a/packages/engine/src/executor.ts b/packages/engine/src/executor.ts index 87f00891ff..3305b9c94e 100644 --- a/packages/engine/src/executor.ts +++ b/packages/engine/src/executor.ts @@ -490,6 +490,10 @@ const MAX_WORKFLOW_STEP_RETRIES = 3; const MAX_TASK_DONE_SESSION_RETRIES = 3; /** Maximum todo requeues after exhausting in-session fn_task_done retries. */ const MAX_TASK_DONE_REQUEUE_RETRIES = 3; +/** Maximum no-progress execute-node self-requeues before terminalizing the loop. */ +export const MAX_EXECUTE_REQUEUE_LOOP_CYCLES = 6; +/** Low-water mark for surfacing a visible warning before loop terminalization. */ +export const EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD = 3; /** * Maximum bounded retries for the narrow resume-after-restart graph transient. * Budget exhaustion falls through to terminal status:"failed" so FN-5704's @@ -506,6 +510,13 @@ const LOOP_COMPACTION_TIMEOUT_MS = 60_000; const TASK_DONE_REFUSAL_SUFFIX = "Either finish the work and resubmit, or do not call fn_task_done — exit the session and the engine will requeue."; +export function buildExecuteRequeueLoopSignature(live: TaskDetail): string { + return JSON.stringify({ + currentStep: live.currentStep ?? null, + steps: live.steps?.map((step) => step.status) ?? [], + }); +} + const TRANSIENT_WORKTREE_TASK_JSON_ENOENT_PATTERN = /ENOENT:\s+no such file or directory,\s+open\s+'([^']+\/\.fusion\/tasks\/([^/]+)\/task\.json)'/; export function isTransientMissingTaskJsonError(error: unknown, task: Pick): boolean { @@ -8935,7 +8946,57 @@ export class TaskExecutor { FNXC:WorkflowLifecycle 2026-06-23-21:19: Also honor the in-process self-requeue marker. Upgrade/restart races and minimal stores can return a stale `in-progress` live row even after the inner executor already moved the task to `todo`; stale reads must not strand progressing tasks in review. + + FNXC:WorkflowLifecycle 2026-07-12-00:00: + FN-7863: the scheduler's wall-clock dispatchStormCount guard only increments when re-dispatches happen inside its short window; slow execute→pause-abort→todo loops reset that counter every cycle. Count this funnel by execution-progress signature instead, warn early for board-visible monitoring, and terminalize only non-paused live tasks after the bounded no-progress cap while preserving worktree/branch/step progress. */ + const signature = buildExecuteRequeueLoopSignature(live); + const nextCount = live.executeRequeueLoopSignature === signature + ? (live.executeRequeueLoopCount ?? 0) + 1 + : 1; + if (live.executeRequeueLoopCount !== nextCount || live.executeRequeueLoopSignature !== signature) { + await this.store.updateTask(task.id, { + executeRequeueLoopCount: nextCount, + executeRequeueLoopSignature: signature, + }, this.getRunContextFor(task.id)); + } + if (nextCount === EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD) { + const warningMessage = `Execution dispatch loop building: ${nextCount}/${MAX_EXECUTE_REQUEUE_LOOP_CYCLES} no-progress execute re-queues`; + executorLog.warn(`${task.id}: ${warningMessage}`); + await this.store.logEntry(task.id, warningMessage, undefined, this.getRunContextFor(task.id)); + } + const canTerminalizeExecuteLoop = live.userPaused !== true + && live.paused !== true + && live.column !== "done" + && live.column !== "archived"; + if (nextCount >= MAX_EXECUTE_REQUEUE_LOOP_CYCLES && canTerminalizeExecuteLoop) { + const terminalError = `EXECUTION_DISPATCH_LOOP_EXHAUSTED: execute node re-queued task to todo ${nextCount} times with no forward progress (last value=${failureValue ?? "no-value"}). No further automatic retries will run. Manually retry, decompose, or rescope the task.`; + await this.store.updateTask(task.id, { + status: "failed", + error: terminalError, + executeRequeueLoopCount: nextCount, + executeRequeueLoopSignature: signature, + }, this.getRunContextFor(task.id)); + await this.store.recordRunAuditEvent?.({ + taskId: task.id, + agentId: "executor", + runId: generateSyntheticRunId("execution-dispatch-loop", task.id), + domain: "database", + mutationType: "task:execution-dispatch-loop-terminalized", + target: task.id, + metadata: { + taskId: task.id, + cycleCount: nextCount, + maxCycles: MAX_EXECUTE_REQUEUE_LOOP_CYCLES, + progressSignature: signature, + failureValue: failureValue ?? null, + }, + }); + executorLog.warn(`${task.id}: ${terminalError}`); + await this.store.logEntry(task.id, terminalError, undefined, this.getRunContextFor(task.id)); + await this.persistTokenUsage(task.id); + return; + } const benignMessage = `Workflow graph execute node ended after executor re-queued task to todo (${failureValue ?? "no-value"}) — executor recovery preserved`; executorLog.log(`${task.id}: ${benignMessage}`); await this.store.logEntry(task.id, benignMessage, undefined, this.getRunContextFor(task.id)); @@ -10253,7 +10314,7 @@ export class TaskExecutor { } // Reset retry counters on success - await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null }); + await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null, executeRequeueLoopCount: null, executeRequeueLoopSignature: null }); if (await this.shouldDeferCompletionForGlobalPause(task.id, "before in-review transition after step-session completion")) { return; } @@ -11085,7 +11146,7 @@ export class TaskExecutor { } // Reset retry counters on success - await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null }); + await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null, executeRequeueLoopCount: null, executeRequeueLoopSignature: null }); if (await this.shouldDeferCompletionForGlobalPause(task.id, "before in-review transition after task completion (post-reset)")) { return; } @@ -11369,7 +11430,7 @@ export class TaskExecutor { await this.store.logEntry(task.id, "Fast mode — pre-merge workflow steps skipped", undefined, this.getRunContextFor(task.id)); } - await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null }); + await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null, executeRequeueLoopCount: null, executeRequeueLoopSignature: null }); if (await this.shouldDeferCompletionForGlobalPause(task.id, "before in-review transition after task completion retry")) { return; } diff --git a/packages/engine/src/run-audit.ts b/packages/engine/src/run-audit.ts index 8442375854..0d7439cab4 100644 --- a/packages/engine/src/run-audit.ts +++ b/packages/engine/src/run-audit.ts @@ -551,6 +551,8 @@ export type DatabaseMutationType = | "task:stuck-no-progress-churn-terminalized" /** Metadata: { taskId, cycleCount, windowMs, lastMoveSource } */ | "task:dispatch-oscillation-terminalized" + /** Metadata: { taskId, cycleCount, maxCycles, progressSignature, failureValue } */ + | "task:execution-dispatch-loop-terminalized" | "task:auto-recover-starved-refinement" /** Metadata: { rawDiffFileCount: number; attributedFileCount: number; foreignCommitCount: number; foreignCommitShas: string[]; source: string } */ | "task:worktree-contamination-detected" diff --git a/packages/engine/src/scheduler.ts b/packages/engine/src/scheduler.ts index 07a44ca138..1255856461 100644 --- a/packages/engine/src/scheduler.ts +++ b/packages/engine/src/scheduler.ts @@ -766,10 +766,12 @@ export class Scheduler { } } else if (to === "in-review" || to === "done" || to === "archived") { this.recentEngineTodoRequeues.delete(task.id); - if (task.dispatchStormCount != null || task.lastDispatchAt != null) { + if (task.dispatchStormCount != null || task.lastDispatchAt != null || task.executeRequeueLoopCount != null || task.executeRequeueLoopSignature != null) { void this.store.updateTask(task.id, { dispatchStormCount: null, lastDispatchAt: null, + executeRequeueLoopCount: null, + executeRequeueLoopSignature: null, }).catch((error) => { schedulerLog.warn(`Failed to reset dispatch oscillation state for ${task.id} on move to ${to}: ${error instanceof Error ? error.message : String(error)}`); }); @@ -812,10 +814,12 @@ export class Scheduler { } else if (this.pausedTaskIds.has(task.id)) { // Task was paused, now unpaused — trigger scheduling this.pausedTaskIds.delete(task.id); - if (task.userPaused === false && (task.dispatchStormCount != null || task.lastDispatchAt != null)) { + if (task.userPaused === false && (task.dispatchStormCount != null || task.lastDispatchAt != null || task.executeRequeueLoopCount != null || task.executeRequeueLoopSignature != null)) { void this.store.updateTask(task.id, { dispatchStormCount: null, lastDispatchAt: null, + executeRequeueLoopCount: null, + executeRequeueLoopSignature: null, }).catch((error) => { schedulerLog.warn(`Failed to reset dispatch oscillation state for ${task.id} on unpause: ${error instanceof Error ? error.message : String(error)}`); });