FN-7863: add bounded execute-node self-requeue loop guard
Bounds the execute->pause-abort->todo dispatch loop so a task can no longer requeue forever with no visible signal or terminal state. - Track a progress-anchored `executeRequeueLoopCount`/`executeRequeueLoopSignature` pair on the task row (current step + step statuses) so slow no-progress requeue cycles are counted independently of the scheduler's wall-clock `dispatchStormCount` guard. - Warn visibly in the task log at `EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD` (3) and terminalize non-paused, non-terminal tasks at `MAX_EXECUTE_REQUEUE_LOOP_CYCLES` (6) with `status:"failed"` and an `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` error, preserving worktree/branch/step progress. - Emit a new `task:execution-dispatch-loop-terminalized` run-audit mutation type with ids/counts/outcomes-only metadata. - Reset the loop counters on real progress, manual retry, forward moves (in-review/done/archived), and unpause, in both the executor and scheduler. - Add DB migration 142 (`executeRequeueLoopCount`, `executeRequeueLoopSignature` columns) plus store read/write/reset plumbing. - Add reliability-interactions coverage for the new loop guard and extend store-persistence tests for the new columns. - Document the new behavior in AGENTS.md and docs/architecture.md. Files changed: AGENTS.md | 1 + docs/architecture.md | 2 + packages/core/src/__tests__/store-persistence.test.ts | 45 +++++ packages/core/src/db.ts | 17 +- packages/core/src/manual-retry-reset.ts | 1 + packages/core/src/store.ts | 22 ++- packages/core/src/types.ts | 11 ++ .../execute-requeue-loop-guard.test.ts | 188 +++++++++++++++ packages/engine/src/executor.ts | 67 +++++++- packages/engine/src/run-audit.ts | 2 + packages/engine/src/scheduler.ts | 8 +- 11 files changed, 355 insertions(+), 9 deletions(-) Fusion-Task-Id: FN-7863 Fusion-Task-Lineage: db40507f-5851-435e-8854-c1ed695b4154 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
@@ -226,6 +226,7 @@ Scoped exception (FN-5819): shared-branch-group members (`branchContext.assignme
|
||||
- FN-7787: `createResolvedAgentSession` enriches `session:runtime-resolved` with `noModelResolved: true` and `runtimeBuiltInFallbackModel` when a non-mock/non-test session reaches runtime creation without a complete provider/model pair; this is a visibility signal for runtime built-in fallback usage, not a fabricated model-resolution verdict.
|
||||
- FN-7835/FN-7844/FN-7859: durable-agent error-state recovery emits `agent:auto-recover-error-state` when either the heartbeat timer or the self-healing sweep clears a transient, non-operator-actionable `error` and retries; metadata stays ids/counts/outcomes-only (`agentId`, attempt, limit, source), where `source` is `timer`/`automation`/`self-healing`. Both entry paths share the `heartbeatErrorRecovery` budget (self-healing keeps `durableErrorRecovery` only for cooldown/stale-path bookkeeping) and emit `agent:error-retry-exhausted` when the shared budget is exhausted and the agent is parked `paused` with `pauseReason:"error-retry-exhausted"`. Non-recoverable durable heartbeat errors (permanent/usage-limit/operator-actionable, excluding stale worktree/module-resolution suppression and disabled/ephemeral/actively-executing agents) emit `agent:error-parked-unrecoverable` with ids/counts/outcomes-only metadata (`agentId`, `source`, optional `attempts`, `limit`) and park `paused` with `pauseReason:"error-unrecoverable"` for human repair.
|
||||
- FN-7802: self-healing emits `task:reconcile-missing-worktree-merge-active` when it proves an `in-review` merge-active task (`merging`/`merging-pr`/`merging-fix`) is stranded by an unusable-worktree session-start failure, clears stale `worktree`/`branch`/`sessionFile`, resets the worktree-session retry budget, increments `recoveryRetryCount` as the bounded stale-metadata clear counter, and requeues to `todo`; it emits `task:reconcile-missing-worktree-merge-active-no-action` when `autoMerge:false`, workspace-task ownership, or triple-proof blocks the backward move.
|
||||
- FN-7863: executor emits `task:execution-dispatch-loop-terminalized` when an execute-node self-requeue loop reaches `MAX_EXECUTE_REQUEUE_LOOP_CYCLES` with an unchanged progress signature; metadata stays ids/counts/outcomes-only (`taskId`, `cycleCount`, `maxCycles`, `progressSignature`, `failureValue`) and the task is visibly failed with `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` while preserving worktree/branch/step progress.
|
||||
- FN-7011: self-healing emits `task:reconcile-engine-downtime-active-timing` when startup recovery shifts active task segment anchors to exclude proven engine-process downtime, and `task:reconcile-engine-downtime-active-timing-no-action` when no active task qualifies.
|
||||
- FN-5419: git run-audit now includes `pull:fast-forward` and `stash:pop-conflict`; dashboard git surfaces now include the extended `POST /api/git/pull` integration-worktree path plus companion `POST /api/git/stash-resolve`, `POST /api/git/stash-drop`, and `POST /api/git/stash-apply` routes.
|
||||
- FN-6292: self-healing emits `task:reconcile-dependency-blocking-lease` when it rebounds an in-progress holder whose stale file-scope lease blocks an unmet dependency, and `task:reconcile-dependency-blocking-lease-no-action` when triple-proof blocks that backward move.
|
||||
|
||||
@@ -687,6 +687,7 @@ Runtime action-gate flow (v1):
|
||||
- `recoverGhostReviewTasks()` is a fallback only for idle, non-terminal `in-review` states. Terminal/actionable states (notably `status: "failed"`) are preserved and **not** auto-kicked back to `todo`.
|
||||
- `recoverPausedAbortFailures()` clears executor pause/resume abort parks only when the durable row is safe to recover. `todo`/`in-progress` rows are requeued for normal scheduling, while clean `in-review` rows (completed steps, not paused/user-paused/executing, auto-merge eligible, no confirmed or terminal merge evidence) have `status`/`error` cleared in place so review progression can continue. FN-7749 adds the manual-hold exception to the prior `autoMerge:false` guard: a benign hard-cancel pause/resume abort at a merge-region/manual-hold node is the healthy Merge & Close resting state, so already-parked rows of that exact shape are cleared in place without moving backward (FN-5147-compliant). User hard-cancel, global/user pause, terminal merge, live-execution, and other `autoMerge:false` guards remain operator-actionable. Successful recovery emits `task:auto-recover-paused-abort-park` with `preservedInReview` metadata.
|
||||
- Workflow graph pause/resume is node-reentrant for typed engine-internal interruptions. When `WorkflowGraphExecutor` sees the graph abort signal or a node returns `value: "aborted"`, it stamps the interrupted node and `engine-pause` abort kind into graph context. `TaskExecutor` then uses the existing bounded `graphResumeRetryCount` budget to clear the transient abort, suppress failure notification with an `Auto-recovered:` task log, and re-enter the graph/task only under the same safety guards: no user/active global pause, no merge/finalize provenance, no genuine node failure, no terminal merge value, no `autoMerge:false` protected review row, and no active execution owner. Global-pause provenance from the graph-controller abort is re-entrant once the global pause has been lifted because it represents the same in-flight node interruption. Generic legacy pause-abort parks without the typed node marker remain operator-action failures except for the narrow `in-review`/`plan` stale-replay shape: hard-cancel pause provenance, `node:plan:value === "aborted"`, no typed interrupted node, no active task/user/global pause, no terminal merge value, no confirmed merge, auto-merge eligibility, and only a clean row or the exact stale plan pause-abort failure. That path logs `stale replay ignored`, clears only the stale failure state when present, preserves `in-review`, and never re-enters planning or moves the task to `todo`.
|
||||
- FN-7863 adds a progress-anchored guard at the execute-node self-requeue funnel (`failedNode === "execute"` and either the live row is `todo` or the in-process self-requeue marker proves a stale `in-progress` read). The guard persists `executeRequeueLoopCount` plus `executeRequeueLoopSignature` (`currentStep` + step statuses), warns at `EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD = 3`, and terminalizes at `MAX_EXECUTE_REQUEUE_LOOP_CYCLES = 6` with `status:"failed"` and an `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` error. It preserves worktree/branch/step progress, skips paused/user-paused/done/archived terminalization, resets on signature progress, manual retry, successful completion, forward moves, and unpause, and complements rather than replaces the scheduler's wall-clock `dispatchStormCount` guard (fast flapping stays scheduler-owned; slow no-progress flapping is caught here).
|
||||
- `reattach-orphaned-assigned-executions` is a forward-resume safety net for durable-agent assignments. During startup recovery and periodic maintenance, after orphaned-agent and stale-heartbeat-run repairs, self-healing finds `in-progress` tasks with an `assignedAgentId` whose agent has no active heartbeat run and no active executor session after the orphan grace window. It re-dispatches in place via `executor.resumeTaskForAgent(agentId)` (the same seam used by clean `HeartbeatMonitor.onRunCompleted` and guarded by executor double-execution checks), emits `task:reattach-orphaned-execution`, and never moves the task backward. This complements engine-start `executor.resumeOrphaned()` and leaves unassigned/role-based execution recovery to the existing startup/limbo/stuck-task paths.
|
||||
- Durable `Agent.taskId` is a running assignment for parked `todo`/`triage` task rows only when the agent has live proof: a fresh active heartbeat run or an executor-active/tracked heartbeat signal. Scheduler overlap requeues, task move sync, self-healing, and Reports Health Check share this invariant: stale durable links are cleared or rendered as stale while `status: "queued"` and `overlapBlockedBy` remain on the task row so file-scope lease blocking is not weakened. `fn_list_agents` and `fn_agent_show` render the linked task column next to `Current Task` (for example `Current Task: FN-1234 (triage)` or `Current Task: FN-1234 (not active — done)`) so parked-column planning ownership is not misread as in-progress execution drift.
|
||||
- Mission validation has a dedicated stale-run reaper: startup recovery and Batch 2 maintenance call `reapStaleMissionValidatorRuns()` when wired by the runtime, using `VALIDATOR_RUN_STALE_MAX_AGE_MS` (currently 6 hours). The sweep terminates ownerless `mission_validator_runs.status='running'` rows as `error`, writes the reap reason into `summary`, leaves `lastValidatorRunId` pointing at the now-terminal run, and emits run-audit telemetry with `mutationType: "mission:validator-run-reaped"` plus `runId`/`featureId`/`missionId`/`triggerType`/`elapsedMs` metadata. Active mission features move to `loopState="needs_fix"` + `lastValidatorStatus="error"` unless their parent mission is already `complete`/`archived`.
|
||||
@@ -749,6 +750,7 @@ Guardrails: this routine does **not** retry merges, does **not** apply to mixed/
|
||||
- FN-6782/FN-6796: `task:auto-recover-paused-abort-park` records self-healing recovery of pause-abort operator parks. Metadata includes the source column and whether recovery preserved a clean `in-review` row instead of requeueing to `todo`.
|
||||
- FN-7069: `task:reconcile-phantom-committed-reservation` records task-store startup or self-healing cleanup of committed-reservation-without-task phantoms. Metadata includes `reservationStatus: "committed"` plus pruned `activityLog` and `agents` counts; `runAuditEvents` and the committed reservation are intentionally retained for auditability and ID permanence.
|
||||
- FN-7074: `task:reservation-commit-rolled-back` records preventive create-path rollback when a distributed reservation was committed with the task-row insert but a later create materialization step failed. Metadata includes `{ reservationId, nodeId, reason: "failed-create", error }`; the task row/partial directory are removed and the reservation is moved to `aborted` so FN-7069 should not need to clean up a new phantom.
|
||||
- FN-7863: `task:execution-dispatch-loop-terminalized` records executor terminalization of a no-progress execute-node self-requeue loop. Metadata includes `{ taskId, cycleCount, maxCycles, progressSignature, failureValue }`; the task row carries `EXECUTION_DISPATCH_LOOP_EXHAUSTED:` and remains in its visible failed state with committed/step progress preserved.
|
||||
- FN-4956: Layer 3 merge-conflict arbitration now scope-partitions conflicted files before AI resolution. Out-of-scope conflicts are deterministically resolved to the integration branch (`git checkout --ours`) and unstaged, while only in-scope conflicts flow to AI. Integration branch defaults are resolved via `resolveIntegrationBranch(rootDir, settings)`. Audit events: `merge:layer3:foreign-file-skipped` and `merge:layer3:scope-override-bypass`.
|
||||
- FN-5655 goal anchoring observability adds `database`-domain mutation types `goal:injection-applied`, `goal:injection-skipped`, and `goal:retrieval-invoked` so Slice 2 cite-rate tracking has a prompt-independent signal. Metadata uses counts/IDs only (`count`, `lane`, `toolName`, optional `truncated`/`reason`/`notFound`) and never stores prompt bodies or goal titles/descriptions. These events surface through `GET /api/agents/:id/runs/:runId/audit` and support the existing `startTime`/`endTime` filters.
|
||||
|
||||
|
||||
@@ -59,6 +59,51 @@ describe("TaskStore", () => {
|
||||
});
|
||||
});
|
||||
|
||||
/*
|
||||
* FNXC:WorkflowLifecycle 2026-07-12-00:00:
|
||||
* FN-7863's execute self-requeue loop guard is progress-anchored, so both the no-progress streak
|
||||
* and the compact step-status signature must round-trip through SQLite and list surfaces.
|
||||
*/
|
||||
describe("executeRequeueLoop persistence", () => {
|
||||
it("round-trips execute loop counters through updateTask, getTask, and listTasks", async () => {
|
||||
const task = await harness.store().createTask({ description: "Execute requeue loop task" });
|
||||
const updated = await harness.store().updateTask(task.id, {
|
||||
executeRequeueLoopCount: 3,
|
||||
executeRequeueLoopSignature: JSON.stringify({ currentStep: 1, steps: ["done", "pending"] }),
|
||||
});
|
||||
|
||||
expect(updated.executeRequeueLoopCount).toBe(3);
|
||||
expect(updated.executeRequeueLoopSignature).toBe('{"currentStep":1,"steps":["done","pending"]}');
|
||||
|
||||
const detail = await harness.store().getTask(task.id);
|
||||
expect(detail.executeRequeueLoopCount).toBe(3);
|
||||
expect(detail.executeRequeueLoopSignature).toBe('{"currentStep":1,"steps":["done","pending"]}');
|
||||
|
||||
const listed = (await harness.store().listTasks()).find((candidate) => candidate.id === task.id);
|
||||
expect(listed?.executeRequeueLoopCount).toBe(3);
|
||||
expect(listed?.executeRequeueLoopSignature).toBe('{"currentStep":1,"steps":["done","pending"]}');
|
||||
});
|
||||
|
||||
it("clears execute loop state with explicit null updates", async () => {
|
||||
const task = await harness.store().createTask({ description: "Execute requeue loop clear task" });
|
||||
await harness.store().updateTask(task.id, {
|
||||
executeRequeueLoopCount: 2,
|
||||
executeRequeueLoopSignature: "sig",
|
||||
});
|
||||
|
||||
const cleared = await harness.store().updateTask(task.id, {
|
||||
executeRequeueLoopCount: null,
|
||||
executeRequeueLoopSignature: null,
|
||||
});
|
||||
expect(cleared.executeRequeueLoopCount).toBeUndefined();
|
||||
expect(cleared.executeRequeueLoopSignature).toBeUndefined();
|
||||
|
||||
const detail = await harness.store().getTask(task.id);
|
||||
expect(detail.executeRequeueLoopCount).toBe(0);
|
||||
expect(detail.executeRequeueLoopSignature).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
/*
|
||||
* FNXC:PlanApproval 2026-07-04-22:41:
|
||||
* FN-7569 — approvedPlanFingerprint must survive create/update/null-clear round trips through
|
||||
|
||||
@@ -184,7 +184,7 @@ export function isFts5CorruptionError(error: unknown): boolean {
|
||||
|
||||
// ── Schema Definition ────────────────────────────────────────────────
|
||||
|
||||
const SCHEMA_VERSION = 141;
|
||||
const SCHEMA_VERSION = 142;
|
||||
|
||||
const TASKS_FTS_AUTOMERGE = 8;
|
||||
const TASKS_FTS_CRISISMERGE = 16;
|
||||
@@ -285,9 +285,11 @@ CREATE TABLE IF NOT EXISTS tasks (
|
||||
mergeRetries INTEGER,
|
||||
workflowStepRetries INTEGER,
|
||||
resumeLimboCount INTEGER DEFAULT 0,
|
||||
executeRequeueLoopCount INTEGER DEFAULT 0,
|
||||
graphResumeRetryCount INTEGER DEFAULT 0,
|
||||
resumeLimboTipSha TEXT,
|
||||
resumeLimboStepSignature TEXT,
|
||||
executeRequeueLoopSignature TEXT,
|
||||
recoveryRetryCount INTEGER,
|
||||
taskDoneRetryCount INTEGER DEFAULT 0,
|
||||
worktreeSessionRetryCount INTEGER DEFAULT 0,
|
||||
@@ -5673,6 +5675,19 @@ export class Database {
|
||||
});
|
||||
}
|
||||
|
||||
if (version < 142) {
|
||||
/*
|
||||
* FNXC:WorkflowLifecycle 2026-07-12-00:00:
|
||||
* FN-7863 stores the progress-anchored execute self-requeue streak so slow
|
||||
* execute→pause-abort→todo loops survive scheduler cadence gaps and terminalize
|
||||
* visibly instead of burning executor slots indefinitely.
|
||||
*/
|
||||
this.applyMigration(142, () => {
|
||||
this.addColumnIfMissing("tasks", "executeRequeueLoopCount", "INTEGER DEFAULT 0");
|
||||
this.addColumnIfMissing("tasks", "executeRequeueLoopSignature", "TEXT");
|
||||
});
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -5,6 +5,7 @@ export const IN_REVIEW_STALL_DEADLOCK_PAUSE_REASON = "in-review-stall-deadlock";
|
||||
export const MANUAL_RETRY_RESET_COUNTER_KEYS = [
|
||||
"stuckKillCount",
|
||||
"resumeLimboCount",
|
||||
"executeRequeueLoopCount",
|
||||
"graphResumeRetryCount",
|
||||
"recoveryRetryCount",
|
||||
"taskDoneRetryCount",
|
||||
|
||||
@@ -251,9 +251,11 @@ interface TaskRow {
|
||||
workflowStepRetries: number | null;
|
||||
stuckKillCount: number | null;
|
||||
resumeLimboCount: number | null;
|
||||
executeRequeueLoopCount: number | null;
|
||||
graphResumeRetryCount: number | null;
|
||||
resumeLimboTipSha: string | null;
|
||||
resumeLimboStepSignature: string | null;
|
||||
executeRequeueLoopSignature: string | null;
|
||||
postReviewFixCount: number | null;
|
||||
recoveryRetryCount: number | null;
|
||||
taskDoneRetryCount: number | null;
|
||||
@@ -419,9 +421,11 @@ const TASK_COLUMN_DESCRIPTORS: TaskColumnDescriptor[] = [
|
||||
defineTaskColumn("workflowStepRetries", (task) => task.workflowStepRetries ?? null),
|
||||
defineTaskColumn("stuckKillCount", (task) => task.stuckKillCount ?? 0),
|
||||
defineTaskColumn("resumeLimboCount", (task) => task.resumeLimboCount ?? 0),
|
||||
defineTaskColumn("executeRequeueLoopCount", (task) => task.executeRequeueLoopCount ?? 0),
|
||||
defineTaskColumn("graphResumeRetryCount", (task) => task.graphResumeRetryCount === undefined ? 0 : task.graphResumeRetryCount),
|
||||
defineTaskColumn("resumeLimboTipSha", (task) => task.resumeLimboTipSha ?? null),
|
||||
defineTaskColumn("resumeLimboStepSignature", (task) => task.resumeLimboStepSignature ?? null),
|
||||
defineTaskColumn("executeRequeueLoopSignature", (task) => task.executeRequeueLoopSignature ?? null),
|
||||
defineTaskColumn("postReviewFixCount", (task) => task.postReviewFixCount ?? 0),
|
||||
defineTaskColumn("recoveryRetryCount", (task) => task.recoveryRetryCount ?? null),
|
||||
defineTaskColumn("taskDoneRetryCount", (task) => task.taskDoneRetryCount ?? 0),
|
||||
@@ -2152,9 +2156,11 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
workflowStepRetries: row.workflowStepRetries ?? undefined,
|
||||
stuckKillCount: row.stuckKillCount ?? undefined,
|
||||
resumeLimboCount: row.resumeLimboCount ?? undefined,
|
||||
executeRequeueLoopCount: row.executeRequeueLoopCount ?? undefined,
|
||||
graphResumeRetryCount: row.graphResumeRetryCount ?? undefined,
|
||||
resumeLimboTipSha: row.resumeLimboTipSha || undefined,
|
||||
resumeLimboStepSignature: row.resumeLimboStepSignature || undefined,
|
||||
executeRequeueLoopSignature: row.executeRequeueLoopSignature || undefined,
|
||||
postReviewFixCount: row.postReviewFixCount ?? undefined,
|
||||
recoveryRetryCount: row.recoveryRetryCount ?? undefined,
|
||||
taskDoneRetryCount: row.taskDoneRetryCount ?? undefined,
|
||||
@@ -2738,7 +2744,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
"modelPresetId", "modelProvider", "modelId",
|
||||
"validatorModelProvider", "validatorModelId",
|
||||
"planningModelProvider", "planningModelId",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "executeRequeueLoopSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"error", "summary", "thinkingLevel", "executionMode", "plannerOversightLevel", "awaitingApprovalReason", "approvedPlanFingerprint",
|
||||
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenUsageModelProvider", "tokenUsageModelId", "tokenUsagePerModel", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
|
||||
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "columnDwellMs", "executionStartedAt", "executionCompletedAt",
|
||||
@@ -2834,7 +2840,7 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
"modelPresetId", "modelProvider", "modelId",
|
||||
"validatorModelProvider", "validatorModelId",
|
||||
"planningModelProvider", "planningModelId",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"mergeRetries", "workflowStepRetries", "stuckKillCount", "resumeLimboCount", "executeRequeueLoopCount", "graphResumeRetryCount", "resumeLimboTipSha", "resumeLimboStepSignature", "executeRequeueLoopSignature", "postReviewFixCount", "recoveryRetryCount", "taskDoneRetryCount", "worktreeSessionRetryCount", "completionHandoffLimboRecoveryCount", "verificationFailureCount", "mergeConflictBounceCount", "mergeAuditBounceCount", "mergeTransientRetryCount", "branchConflictRecoveryCount", "reviewerContextRetryCount", "reviewerFallbackRetryCount", "nextRecoveryAt",
|
||||
"error", "summary", "thinkingLevel", "executionMode", "plannerOversightLevel", "awaitingApprovalReason", "approvedPlanFingerprint",
|
||||
"tokenUsageInputTokens", "tokenUsageOutputTokens", "tokenUsageCachedTokens", "tokenUsageCacheWriteTokens", "tokenUsageTotalTokens", "tokenUsageFirstUsedAt", "tokenUsageLastUsedAt", "tokenUsageModelProvider", "tokenUsageModelId", "tokenUsagePerModel", "tokenBudgetSoftAlertedAt", "tokenBudgetHardAlertedAt", "tokenBudgetOverride",
|
||||
"createdAt", "updatedAt", "columnMovedAt", "firstExecutionAt", "cumulativeActiveMs", "columnDwellMs", "executionStartedAt", "executionCompletedAt",
|
||||
@@ -8597,7 +8603,7 @@ ${TASK_UPSERT_SQL_ASSIGNMENTS}
|
||||
|
||||
async updateTask(
|
||||
id: string,
|
||||
updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; workspaceWorktrees?: import("./types.js").Task["workspaceWorktrees"]; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; customFields?: Record<string, unknown>; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; plannerOversightLevel?: import("./types.js").PlannerOversightLevel | null; awaitingApprovalReason?: import("./types.js").Task["awaitingApprovalReason"] | null; approvedPlanFingerprint?: string | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; graphResumeRetryCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record<string, unknown> | null; githubTracking?: import("./types.js").TaskGithubTracking | null; gitlabTracking?: (Omit<import("./types.js").TaskGitLabTracking, "item"> & { item?: import("./types.js").TaskGitLabTrackedItem | null }) | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; workflowTransitionNotification?: import("./types.js").Task["workflowTransitionNotification"] | null; missionId?: string | null; sliceId?: string | null },
|
||||
updates: { title?: string; description?: string; priority?: TaskPriority | null; prompt?: string; worktree?: string | null; workspaceWorktrees?: import("./types.js").Task["workspaceWorktrees"]; status?: string | null; dependencies?: string[]; steps?: import("./types.js").TaskStep[]; customFields?: Record<string, unknown>; currentStep?: number; blockedBy?: string | null; overlapBlockedBy?: string | null; assignedAgentId?: string | null; pausedByAgentId?: string | null; pausedReason?: string | null; tokenBudgetSoftAlertedAt?: string | null; worktrunkFallbackAlertedAt?: string | null; worktrunkFailure?: import("./types.js").Task["worktrunkFailure"] | null; tokenBudgetHardAlertedAt?: string | null; tokenBudgetOverride?: import("./types.js").TaskTokenBudgetOverride | null; dispatchStormCount?: number | null; lastDispatchAt?: string | null; assigneeUserId?: string | null; scopeOverride?: boolean | null; scopeOverrideReason?: string | null; scopeAutoWiden?: string[] | null; nodeId?: string | null; effectiveNodeId?: string | null; effectiveNodeSource?: string | null; checkedOutBy?: string | null; checkedOutAt?: string | null; checkoutNodeId?: string | null; checkoutRunId?: string | null; checkoutLeaseRenewedAt?: string | null; checkoutLeaseEpoch?: number | null; paused?: boolean; baseBranch?: string | null; autoMerge?: boolean | null; branch?: string | null; executionStartBranch?: string | null; baseCommitSha?: string | null; size?: "S" | "M" | "L"; reviewLevel?: number; executionMode?: import("./types.js").ExecutionMode | null; plannerOversightLevel?: import("./types.js").PlannerOversightLevel | null; awaitingApprovalReason?: import("./types.js").Task["awaitingApprovalReason"] | null; approvedPlanFingerprint?: string | null; mergeRetries?: number; workflowStepRetries?: number; stuckKillCount?: number | null; resumeLimboCount?: number | null; executeRequeueLoopCount?: number | null; graphResumeRetryCount?: number | null; resumeLimboTipSha?: string | null; resumeLimboStepSignature?: string | null; executeRequeueLoopSignature?: string | null; postReviewFixCount?: number | null; recoveryRetryCount?: number | null; taskDoneRetryCount?: number | null; worktreeSessionRetryCount?: number | null; completionHandoffLimboRecoveryCount?: number | null; verificationFailureCount?: number | null; mergeConflictBounceCount?: number | null; mergeAuditBounceCount?: number | null; mergeTransientRetryCount?: number | null; branchConflictRecoveryCount?: number | null; reviewerContextRetryCount?: number | null; reviewerFallbackRetryCount?: number | null; nextRecoveryAt?: string | null; enabledWorkflowSteps?: string[]; noCommitsExpected?: boolean | null; modelProvider?: string | null; modelId?: string | null; validatorModelProvider?: string | null; validatorModelId?: string | null; planningModelProvider?: string | null; planningModelId?: string | null; thinkingLevel?: string | null; error?: string | null; summary?: string | null; sessionFile?: string | null; firstExecutionAt?: string | null; cumulativeActiveMs?: number | null; executionStartedAt?: string | null; executionCompletedAt?: string | null; review?: import("./types.js").TaskReview | null; reviewState?: import("./types.js").TaskReviewState | null; workflowStepResults?: import("./types.js").WorkflowStepResult[] | null; mergeDetails?: import("./types.js").MergeDetails | null; sourceIssue?: import("./types.js").TaskSourceIssue | null; sourceMetadataPatch?: Record<string, unknown> | null; githubTracking?: import("./types.js").TaskGithubTracking | null; gitlabTracking?: (Omit<import("./types.js").TaskGitLabTracking, "item"> & { item?: import("./types.js").TaskGitLabTrackedItem | null }) | null; tokenUsage?: import("./types.js").TaskTokenUsage | null; modifiedFiles?: string[] | null; workflowTransitionNotification?: import("./types.js").Task["workflowTransitionNotification"] | null; missionId?: string | null; sliceId?: string | null },
|
||||
runContext?: RunMutationContext,
|
||||
): Promise<Task> {
|
||||
/*
|
||||
@@ -9343,6 +9349,11 @@ ${TASK_UPSERT_SQL_ASSIGNMENTS}
|
||||
} else if (updates.resumeLimboCount !== undefined) {
|
||||
task.resumeLimboCount = updates.resumeLimboCount;
|
||||
}
|
||||
if (updates.executeRequeueLoopCount === null) {
|
||||
task.executeRequeueLoopCount = undefined;
|
||||
} else if (updates.executeRequeueLoopCount !== undefined) {
|
||||
task.executeRequeueLoopCount = updates.executeRequeueLoopCount;
|
||||
}
|
||||
if (updates.graphResumeRetryCount === null) {
|
||||
task.graphResumeRetryCount = null;
|
||||
} else if (updates.graphResumeRetryCount !== undefined) {
|
||||
@@ -9358,6 +9369,11 @@ ${TASK_UPSERT_SQL_ASSIGNMENTS}
|
||||
} else if (updates.resumeLimboStepSignature !== undefined) {
|
||||
task.resumeLimboStepSignature = updates.resumeLimboStepSignature;
|
||||
}
|
||||
if (updates.executeRequeueLoopSignature === null) {
|
||||
task.executeRequeueLoopSignature = undefined;
|
||||
} else if (updates.executeRequeueLoopSignature !== undefined) {
|
||||
task.executeRequeueLoopSignature = updates.executeRequeueLoopSignature;
|
||||
}
|
||||
if (updates.postReviewFixCount === null) {
|
||||
task.postReviewFixCount = undefined;
|
||||
} else if (updates.postReviewFixCount !== undefined) {
|
||||
|
||||
@@ -2452,6 +2452,14 @@ export interface Task {
|
||||
* Incremented by self-healing for resume-limbo detection and reset when
|
||||
* progress is observed or recovery escalates to a fresh todo dispatch. */
|
||||
resumeLimboCount?: number;
|
||||
/**
|
||||
* FNXC:WorkflowLifecycle 2026-07-12-00:00:
|
||||
* FN-7863 bounds execute-node self-requeue loops by counting consecutive requeues
|
||||
* that preserve the same execution-progress signature. Reset this counter on real
|
||||
* progress, forward moves, and manual retry; the executor caps it before writing
|
||||
* terminal status:"failed" so committed work and step progress remain visible.
|
||||
*/
|
||||
executeRequeueLoopCount?: number;
|
||||
/** Bounded auto-retry attempts for transient workflow-graph failures observed
|
||||
* immediately after engine-restart or unpause resume. Reset by manual retry
|
||||
* and by successful forward progress; capped by the executor before terminal
|
||||
@@ -2463,6 +2471,9 @@ export interface Task {
|
||||
/** Compact execution-progress snapshot captured at the last reclaim/unpause
|
||||
* attempt (current step + step statuses) for resume-limbo detection. */
|
||||
resumeLimboStepSignature?: string;
|
||||
/** Compact execution-progress snapshot captured at the last execute-node
|
||||
* self-requeue (current step + step statuses) for FN-7863 loop detection. */
|
||||
executeRequeueLoopSignature?: string;
|
||||
/** Number of times workflow remediation has auto-revived this task after
|
||||
* failed pre-merge review feedback. Incremented each time the engine sends the
|
||||
* task back with failure feedback injected. Capped only when the workflow step
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { TaskDetail } from "@fusion/core";
|
||||
import "../executor-test-helpers.js";
|
||||
import {
|
||||
EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD,
|
||||
MAX_EXECUTE_REQUEUE_LOOP_CYCLES,
|
||||
TaskExecutor,
|
||||
} from "../../executor.js";
|
||||
import { createMockStore, resetExecutorMocks } from "../executor-test-helpers.js";
|
||||
|
||||
const now = "2026-07-12T00:00:00.000Z";
|
||||
|
||||
function task(overrides: Partial<TaskDetail> = {}): TaskDetail {
|
||||
return {
|
||||
id: "FN-7863-T",
|
||||
title: "Execute requeue loop",
|
||||
description: "Bound execute self-requeue loops",
|
||||
column: "todo",
|
||||
dependencies: [],
|
||||
steps: [{ name: "Implement", status: "pending" }],
|
||||
currentStep: 0,
|
||||
log: [],
|
||||
branch: "fusion/fn-7863",
|
||||
baseBranch: "main",
|
||||
worktree: "/tmp/fusion-fn-7863",
|
||||
status: null,
|
||||
error: null,
|
||||
paused: false,
|
||||
userPaused: false,
|
||||
autoMerge: true,
|
||||
mergeRetries: 0,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
...overrides,
|
||||
} as TaskDetail;
|
||||
}
|
||||
|
||||
function harness(initial: TaskDetail) {
|
||||
resetExecutorMocks();
|
||||
const store = createMockStore();
|
||||
let live = { ...initial } as TaskDetail;
|
||||
store.getTask.mockImplementation(async () => live);
|
||||
store.updateTask.mockImplementation(async (_id: string, updates: Partial<TaskDetail>) => {
|
||||
live = { ...live, ...updates } as TaskDetail;
|
||||
return live;
|
||||
});
|
||||
store.recordRunAuditEvent = vi.fn().mockResolvedValue(undefined);
|
||||
const executor = new TaskExecutor(store, "/tmp/test");
|
||||
return {
|
||||
store,
|
||||
executor,
|
||||
get live() {
|
||||
return live;
|
||||
},
|
||||
setLive(patch: Partial<TaskDetail>) {
|
||||
live = { ...live, ...patch } as TaskDetail;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function failAtExecute(executor: TaskExecutor, taskSnapshot: TaskDetail) {
|
||||
await (executor as any).handleGraphFailure(taskSnapshot, {
|
||||
disposition: "failed",
|
||||
outcome: "failure",
|
||||
visitedNodeIds: ["execute"],
|
||||
context: { "node:execute:value": "implementation-incomplete" },
|
||||
});
|
||||
}
|
||||
|
||||
describe("execute requeue loop guard", () => {
|
||||
it("terminalizes unchanged todo execute self-requeues and preserves progress", async () => {
|
||||
const h = harness(task({
|
||||
id: "FN-7863-TODO",
|
||||
column: "todo",
|
||||
steps: [
|
||||
{ name: "Preflight", status: "done" },
|
||||
{ name: "Implement", status: "in-progress" },
|
||||
],
|
||||
currentStep: 1,
|
||||
}));
|
||||
|
||||
for (let i = 0; i < MAX_EXECUTE_REQUEUE_LOOP_CYCLES; i += 1) {
|
||||
await failAtExecute(h.executor, h.live);
|
||||
}
|
||||
|
||||
expect(h.store.updateTask).toHaveBeenCalledWith(
|
||||
"FN-7863-TODO",
|
||||
expect.objectContaining({
|
||||
status: "failed",
|
||||
error: expect.stringMatching(/^EXECUTION_DISPATCH_LOOP_EXHAUSTED:/),
|
||||
executeRequeueLoopCount: MAX_EXECUTE_REQUEUE_LOOP_CYCLES,
|
||||
}),
|
||||
undefined,
|
||||
);
|
||||
expect(h.store.recordRunAuditEvent).toHaveBeenCalledWith(expect.objectContaining({
|
||||
mutationType: "task:execution-dispatch-loop-terminalized",
|
||||
metadata: expect.objectContaining({
|
||||
taskId: "FN-7863-TODO",
|
||||
cycleCount: MAX_EXECUTE_REQUEUE_LOOP_CYCLES,
|
||||
maxCycles: MAX_EXECUTE_REQUEUE_LOOP_CYCLES,
|
||||
failureValue: "implementation-incomplete",
|
||||
}),
|
||||
}));
|
||||
expect(h.store.moveTask).not.toHaveBeenCalled();
|
||||
const terminalUpdate = h.store.updateTask.mock.calls.find((call: any[]) => call[1]?.status === "failed")?.[1];
|
||||
expect(terminalUpdate).not.toHaveProperty("worktree");
|
||||
expect(terminalUpdate).not.toHaveProperty("branch");
|
||||
expect(terminalUpdate).not.toHaveProperty("steps");
|
||||
const lastLog = h.store.logEntry.mock.calls.at(-1)?.[1] as string;
|
||||
expect(lastLog).toMatch(/^EXECUTION_DISPATCH_LOOP_EXHAUSTED:/);
|
||||
expect(lastLog).not.toContain("executor recovery preserved");
|
||||
});
|
||||
|
||||
it("terminalizes the stale in-progress self-requeue marker path", async () => {
|
||||
const h = harness(task({ id: "FN-7863-STALE", column: "in-progress" }));
|
||||
(h.executor as any).graphRouting.add("FN-7863-STALE");
|
||||
(h.executor as any).markGraphExecuteSelfRequeued("FN-7863-STALE");
|
||||
|
||||
for (let i = 0; i < MAX_EXECUTE_REQUEUE_LOOP_CYCLES; i += 1) {
|
||||
await failAtExecute(h.executor, h.live);
|
||||
}
|
||||
|
||||
expect(h.store.updateTask).toHaveBeenCalledWith(
|
||||
"FN-7863-STALE",
|
||||
expect.objectContaining({ status: "failed", error: expect.stringMatching(/^EXECUTION_DISPATCH_LOOP_EXHAUSTED:/) }),
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
|
||||
it("resets the streak on real step progress and never terminalizes", async () => {
|
||||
const h = harness(task({ id: "FN-7863-PROGRESS", column: "todo" }));
|
||||
|
||||
for (let i = 0; i < MAX_EXECUTE_REQUEUE_LOOP_CYCLES + 3; i += 1) {
|
||||
h.setLive({ currentStep: i, steps: [{ name: `Step ${i}`, status: i % 2 === 0 ? "pending" : "in-progress" }] as any });
|
||||
await failAtExecute(h.executor, h.live);
|
||||
}
|
||||
|
||||
expect(h.store.updateTask).not.toHaveBeenCalledWith(
|
||||
"FN-7863-PROGRESS",
|
||||
expect.objectContaining({ status: "failed" }),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(h.live.executeRequeueLoopCount).toBe(1);
|
||||
});
|
||||
|
||||
it("emits a visible warning at the threshold without terminalizing", async () => {
|
||||
const h = harness(task({ id: "FN-7863-WARN", column: "todo" }));
|
||||
|
||||
for (let i = 0; i < EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD; i += 1) {
|
||||
await failAtExecute(h.executor, h.live);
|
||||
}
|
||||
|
||||
expect(h.store.logEntry).toHaveBeenCalledWith(
|
||||
"FN-7863-WARN",
|
||||
expect.stringContaining(`Execution dispatch loop building: ${EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD}/${MAX_EXECUTE_REQUEUE_LOOP_CYCLES}`),
|
||||
undefined,
|
||||
undefined,
|
||||
);
|
||||
expect(h.store.updateTask).not.toHaveBeenCalledWith(
|
||||
"FN-7863-WARN",
|
||||
expect.objectContaining({ status: "failed" }),
|
||||
expect.anything(),
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["userPaused", { userPaused: true }],
|
||||
["paused", { paused: true }],
|
||||
])("does not terminalize %s tasks from the benign branch", async (_label, patch) => {
|
||||
const h = harness(task({ id: `FN-7863-${_label}`, column: "todo", ...patch }));
|
||||
|
||||
for (let i = 0; i < MAX_EXECUTE_REQUEUE_LOOP_CYCLES; i += 1) {
|
||||
await failAtExecute(h.executor, h.live);
|
||||
}
|
||||
|
||||
expect(h.store.updateTask).not.toHaveBeenCalledWith(
|
||||
h.live.id,
|
||||
expect.objectContaining({ status: "failed" }),
|
||||
expect.anything(),
|
||||
);
|
||||
expect(h.store.logEntry).toHaveBeenCalledWith(
|
||||
h.live.id,
|
||||
expect.stringContaining("paused awaiting explicit unpause"),
|
||||
undefined,
|
||||
undefined,
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -490,6 +490,10 @@ const MAX_WORKFLOW_STEP_RETRIES = 3;
|
||||
const MAX_TASK_DONE_SESSION_RETRIES = 3;
|
||||
/** Maximum todo requeues after exhausting in-session fn_task_done retries. */
|
||||
const MAX_TASK_DONE_REQUEUE_RETRIES = 3;
|
||||
/** Maximum no-progress execute-node self-requeues before terminalizing the loop. */
|
||||
export const MAX_EXECUTE_REQUEUE_LOOP_CYCLES = 6;
|
||||
/** Low-water mark for surfacing a visible warning before loop terminalization. */
|
||||
export const EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD = 3;
|
||||
/**
|
||||
* Maximum bounded retries for the narrow resume-after-restart graph transient.
|
||||
* Budget exhaustion falls through to terminal status:"failed" so FN-5704's
|
||||
@@ -506,6 +510,13 @@ const LOOP_COMPACTION_TIMEOUT_MS = 60_000;
|
||||
|
||||
const TASK_DONE_REFUSAL_SUFFIX = "Either finish the work and resubmit, or do not call fn_task_done — exit the session and the engine will requeue.";
|
||||
|
||||
export function buildExecuteRequeueLoopSignature(live: TaskDetail): string {
|
||||
return JSON.stringify({
|
||||
currentStep: live.currentStep ?? null,
|
||||
steps: live.steps?.map((step) => step.status) ?? [],
|
||||
});
|
||||
}
|
||||
|
||||
const TRANSIENT_WORKTREE_TASK_JSON_ENOENT_PATTERN = /ENOENT:\s+no such file or directory,\s+open\s+'([^']+\/\.fusion\/tasks\/([^/]+)\/task\.json)'/;
|
||||
|
||||
export function isTransientMissingTaskJsonError(error: unknown, task: Pick<Task, "id" | "worktree">): boolean {
|
||||
@@ -8935,7 +8946,57 @@ export class TaskExecutor {
|
||||
|
||||
FNXC:WorkflowLifecycle 2026-06-23-21:19:
|
||||
Also honor the in-process self-requeue marker. Upgrade/restart races and minimal stores can return a stale `in-progress` live row even after the inner executor already moved the task to `todo`; stale reads must not strand progressing tasks in review.
|
||||
|
||||
FNXC:WorkflowLifecycle 2026-07-12-00:00:
|
||||
FN-7863: the scheduler's wall-clock dispatchStormCount guard only increments when re-dispatches happen inside its short window; slow execute→pause-abort→todo loops reset that counter every cycle. Count this funnel by execution-progress signature instead, warn early for board-visible monitoring, and terminalize only non-paused live tasks after the bounded no-progress cap while preserving worktree/branch/step progress.
|
||||
*/
|
||||
const signature = buildExecuteRequeueLoopSignature(live);
|
||||
const nextCount = live.executeRequeueLoopSignature === signature
|
||||
? (live.executeRequeueLoopCount ?? 0) + 1
|
||||
: 1;
|
||||
if (live.executeRequeueLoopCount !== nextCount || live.executeRequeueLoopSignature !== signature) {
|
||||
await this.store.updateTask(task.id, {
|
||||
executeRequeueLoopCount: nextCount,
|
||||
executeRequeueLoopSignature: signature,
|
||||
}, this.getRunContextFor(task.id));
|
||||
}
|
||||
if (nextCount === EXECUTE_REQUEUE_LOOP_VISIBLE_THRESHOLD) {
|
||||
const warningMessage = `Execution dispatch loop building: ${nextCount}/${MAX_EXECUTE_REQUEUE_LOOP_CYCLES} no-progress execute re-queues`;
|
||||
executorLog.warn(`${task.id}: ${warningMessage}`);
|
||||
await this.store.logEntry(task.id, warningMessage, undefined, this.getRunContextFor(task.id));
|
||||
}
|
||||
const canTerminalizeExecuteLoop = live.userPaused !== true
|
||||
&& live.paused !== true
|
||||
&& live.column !== "done"
|
||||
&& live.column !== "archived";
|
||||
if (nextCount >= MAX_EXECUTE_REQUEUE_LOOP_CYCLES && canTerminalizeExecuteLoop) {
|
||||
const terminalError = `EXECUTION_DISPATCH_LOOP_EXHAUSTED: execute node re-queued task to todo ${nextCount} times with no forward progress (last value=${failureValue ?? "no-value"}). No further automatic retries will run. Manually retry, decompose, or rescope the task.`;
|
||||
await this.store.updateTask(task.id, {
|
||||
status: "failed",
|
||||
error: terminalError,
|
||||
executeRequeueLoopCount: nextCount,
|
||||
executeRequeueLoopSignature: signature,
|
||||
}, this.getRunContextFor(task.id));
|
||||
await this.store.recordRunAuditEvent?.({
|
||||
taskId: task.id,
|
||||
agentId: "executor",
|
||||
runId: generateSyntheticRunId("execution-dispatch-loop", task.id),
|
||||
domain: "database",
|
||||
mutationType: "task:execution-dispatch-loop-terminalized",
|
||||
target: task.id,
|
||||
metadata: {
|
||||
taskId: task.id,
|
||||
cycleCount: nextCount,
|
||||
maxCycles: MAX_EXECUTE_REQUEUE_LOOP_CYCLES,
|
||||
progressSignature: signature,
|
||||
failureValue: failureValue ?? null,
|
||||
},
|
||||
});
|
||||
executorLog.warn(`${task.id}: ${terminalError}`);
|
||||
await this.store.logEntry(task.id, terminalError, undefined, this.getRunContextFor(task.id));
|
||||
await this.persistTokenUsage(task.id);
|
||||
return;
|
||||
}
|
||||
const benignMessage = `Workflow graph execute node ended after executor re-queued task to todo (${failureValue ?? "no-value"}) — executor recovery preserved`;
|
||||
executorLog.log(`${task.id}: ${benignMessage}`);
|
||||
await this.store.logEntry(task.id, benignMessage, undefined, this.getRunContextFor(task.id));
|
||||
@@ -10253,7 +10314,7 @@ export class TaskExecutor {
|
||||
}
|
||||
|
||||
// Reset retry counters on success
|
||||
await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null });
|
||||
await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null, executeRequeueLoopCount: null, executeRequeueLoopSignature: null });
|
||||
if (await this.shouldDeferCompletionForGlobalPause(task.id, "before in-review transition after step-session completion")) {
|
||||
return;
|
||||
}
|
||||
@@ -11085,7 +11146,7 @@ export class TaskExecutor {
|
||||
}
|
||||
|
||||
// Reset retry counters on success
|
||||
await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null });
|
||||
await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null, executeRequeueLoopCount: null, executeRequeueLoopSignature: null });
|
||||
if (await this.shouldDeferCompletionForGlobalPause(task.id, "before in-review transition after task completion (post-reset)")) {
|
||||
return;
|
||||
}
|
||||
@@ -11369,7 +11430,7 @@ export class TaskExecutor {
|
||||
await this.store.logEntry(task.id, "Fast mode — pre-merge workflow steps skipped", undefined, this.getRunContextFor(task.id));
|
||||
}
|
||||
|
||||
await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null });
|
||||
await this.store.updateTask(task.id, { workflowStepRetries: undefined, taskDoneRetryCount: null, executeRequeueLoopCount: null, executeRequeueLoopSignature: null });
|
||||
if (await this.shouldDeferCompletionForGlobalPause(task.id, "before in-review transition after task completion retry")) {
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -551,6 +551,8 @@ export type DatabaseMutationType =
|
||||
| "task:stuck-no-progress-churn-terminalized"
|
||||
/** Metadata: { taskId, cycleCount, windowMs, lastMoveSource } */
|
||||
| "task:dispatch-oscillation-terminalized"
|
||||
/** Metadata: { taskId, cycleCount, maxCycles, progressSignature, failureValue } */
|
||||
| "task:execution-dispatch-loop-terminalized"
|
||||
| "task:auto-recover-starved-refinement"
|
||||
/** Metadata: { rawDiffFileCount: number; attributedFileCount: number; foreignCommitCount: number; foreignCommitShas: string[]; source: string } */
|
||||
| "task:worktree-contamination-detected"
|
||||
|
||||
@@ -766,10 +766,12 @@ export class Scheduler {
|
||||
}
|
||||
} else if (to === "in-review" || to === "done" || to === "archived") {
|
||||
this.recentEngineTodoRequeues.delete(task.id);
|
||||
if (task.dispatchStormCount != null || task.lastDispatchAt != null) {
|
||||
if (task.dispatchStormCount != null || task.lastDispatchAt != null || task.executeRequeueLoopCount != null || task.executeRequeueLoopSignature != null) {
|
||||
void this.store.updateTask(task.id, {
|
||||
dispatchStormCount: null,
|
||||
lastDispatchAt: null,
|
||||
executeRequeueLoopCount: null,
|
||||
executeRequeueLoopSignature: null,
|
||||
}).catch((error) => {
|
||||
schedulerLog.warn(`Failed to reset dispatch oscillation state for ${task.id} on move to ${to}: ${error instanceof Error ? error.message : String(error)}`);
|
||||
});
|
||||
@@ -812,10 +814,12 @@ export class Scheduler {
|
||||
} else if (this.pausedTaskIds.has(task.id)) {
|
||||
// Task was paused, now unpaused — trigger scheduling
|
||||
this.pausedTaskIds.delete(task.id);
|
||||
if (task.userPaused === false && (task.dispatchStormCount != null || task.lastDispatchAt != null)) {
|
||||
if (task.userPaused === false && (task.dispatchStormCount != null || task.lastDispatchAt != null || task.executeRequeueLoopCount != null || task.executeRequeueLoopSignature != null)) {
|
||||
void this.store.updateTask(task.id, {
|
||||
dispatchStormCount: null,
|
||||
lastDispatchAt: null,
|
||||
executeRequeueLoopCount: null,
|
||||
executeRequeueLoopSignature: null,
|
||||
}).catch((error) => {
|
||||
schedulerLog.warn(`Failed to reset dispatch oscillation state for ${task.id} on unpause: ${error instanceof Error ? error.message : String(error)}`);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user