root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

FN-7743: detect and recover stalled in-progress executor tasks in overseer

Browse Source 
Fix the planner overseer's executor-stage stall detection so hung in-progress tasks get unstuck instead of being reported as progressing forever.

- Add configurable stuck-detection: the executor-stage overseer observation now emits signal: "stuck" once an in-progress task has been inactive past a new plannerOverseerExecutorStuckAfterMs threshold, feeding the existing decidePlannerRecovery -> bounded inject_guidance recovery path.
- Register the new plannerOverseerExecutorStuckAfterMs setting in builtin-workflow-settings.ts and export it via core index.ts/index.gate.ts.
- Preserve human-control withholds (user-paused / approval-blocked / autoMerge-off) taking precedence over stuck detection.
- Add/extend tests covering planner-overseer, planner-recovery-controller, planner-recovery, and builtin-workflow-settings-triage.
- Document the new setting in docs/architecture.md and docs/settings-reference.md.
- Add changeset fn-7743-overseer-executor-stall.md (patch).

Files changed:
 .changeset/fn-7743-overseer-executor-stall.md      |   7 +
 docs/architecture.md                               |  31 +++++
 docs/settings-reference.md                         |   3 +-
 .../builtin-workflow-settings-triage.test.ts       |  22 ++++
 .../core/src/__tests__/planner-recovery.test.ts    |  12 ++
 packages/core/src/builtin-workflow-settings.ts     |  26 ++++
 packages/core/src/index.gate.ts                    |   1 +
 packages/core/src/index.ts                         |   1 +
 .../engine/src/__tests__/planner-overseer.test.ts  | 146 +++++++++++++++++++++
 .../__tests__/planner-recovery-controller.test.ts  |  52 +++++++-
 packages/engine/src/planner-overseer.ts            |  88 ++++++++++++-
 packages/engine/src/project-engine.ts              |  11 +-
 12 files changed, 391 insertions(+), 9 deletions(-)

Fusion-Task-Id: FN-7743

Fusion-Task-Lineage: 791852a2-5b77-48de-955a-1b3916616dfa

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
2026-07-09 13:32:21 -07:00
parent 9d7b087188
commit 2ff8e2e13e
12 changed files with 391 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/settings-reference.md
View File

@@ -327,7 +327,7 @@ These groups moved out of project settings and into workflow settings (built-in
|---|---|
| **Step execution** | `workflowStepTimeoutMs`, `runStepsInNewSessions`, `maxParallelSteps`, `workflowStepScopeEnforcement`, `strictScopeEnforcement`, `verificationFixRetries`, `maxPostReviewFixes`, `buildRetryCount` |
| **Review / approval** | Workflow values: `requirePrApproval`, `requirePlanApproval`, `reviewHandoffPolicy`, `maxReviewerContextRetries`, `maxReviewerFallbackRetries`, `planReviewMaxRevisions`, `codeReviewMaxRevisions`; project override: `planApprovalMode` |
| **Planner oversight** | `plannerOversightLevel` (workflow-native; values: `off`, `observe`, `steer`, `autonomous`); `plannerOversightNotificationLevel` (workflow-native; values: `silent`, `errors`, `important`, `all`) |
| **Planner oversight** | `plannerOversightLevel` (workflow-native; values: `off`, `observe`, `steer`, `autonomous`); `plannerOversightNotificationLevel` (workflow-native; values: `silent`, `errors`, `important`, `all`); `plannerOverseerExecutorStuckAfterMs` (workflow-native; number, default `7200000` = 2h) |
| **Per-phase model lanes** | `executionProvider`/`executionModelId`, `planningProvider`/`planningModelId` (+ fallbacks), `validatorProvider`/`validatorModelId` (+ fallbacks) |
### Workflow-native triage policy settings
@@ -363,6 +363,7 @@ The built-in workflows also declare triage/spec policy settings that were **not*
| `codeReviewMaxRevisions` | unset | Workflow-native Code Review remediation cap. Unset/empty means unbounded automatic code-fix passes; a non-negative integer caps attempts; `0` disables automatic Code Review remediation. |
| `plannerOversightLevel` | `autonomous` | Workflow-native planner oversight mode. `off` disables oversight; `observe` watches only; `steer` injects guidance or suggests revisions; `autonomous` enables bounded retry and targeted-fix recovery — but merge/PR progression and any destructive or external-service side effect ALWAYS require an explicit, recorded human confirmation before they run, even at `autonomous` (FN-7513's confirmation gate; see `docs/architecture.md` → "Planner overseer confirmation gate"). Tasks may set a nullable `Task.plannerOversightLevel` override (same four values) that wins over this workflow value when present; `null`/unset means "inherit the workflow value". `resolveEffectivePlannerOversightLevel` in `@fusion/core` computes the effective level (task override → workflow effective → `autonomous`). The per-task override is exposed in the dashboard as a "Planner oversight" selector (Inherit from workflow / Off / Observe / Steer / Autonomous recovery) in both the New Task dialog and Task Detail edit form, threaded through `createTask`/`updateTask` (FN-7515); the project/global default is set via the **Workflow Editor → Values** tab on the default workflow's `plannerOversightLevel` value, not in Project Settings. FN-7517 additionally exposes a quick inline oversight-level select in the Task Detail modal's meta-controls cluster (same `updateTask` override plumbing, no parallel path) plus manual nudge/stop-oversight/explain-current-action controls that call the overseer runtime directly — see `docs/dashboard-guide.md`. Engine read-site behavior beyond the FN-7513 confirmation gate remains follow-up work (FN-7510+). |
| `plannerOversightNotificationLevel` | `important` | Workflow-native planner-overseer notification verbosity (FN-7518). `silent` suppresses overseer notifications; `errors` notifies only on failures/escalations; `important` (the default) notifies on interventions/recovery actions and errors; `all` notifies on every observation. Resolves through the generic `resolveEffectiveSettings` default path with no special-casing, alongside `plannerOversightLevel`. This is a declaration-only setting: the notification-emission gating that reads it lands downstream in FN-7519 (intervention timeline) and FN-7520 (run-audit/activity events). |
| `plannerOverseerExecutorStuckAfterMs` | `7200000` (2h) | Workflow-native executor-stage stall threshold (FN-7743). Milliseconds of executor-stage inactivity — no execution activity since the task's last column move/update (`columnMovedAt ?? updatedAt`) — before a non-paused `in-progress` task is reported `signal: "stuck"` instead of `"progressing"`, feeding the existing `decidePlannerRecovery` → bounded `inject_guidance` recovery path at the `autonomous` oversight level (no effect at `off`/`observe`/`steer`). Fixes the class of bug where a genuinely hung/idle executor (dead session, silent agent) was indistinguishable from a healthy one and was never nudged, retried, or escalated. A missing/malformed activity timestamp degrades to `"progressing"` (fail-safe — never fabricates a stall), and a user-paused/approval-blocked/`autoMerge:false` task is still fully withheld from any autonomous action regardless of this threshold. Resolves through the generic `resolveEffectiveSettings` default path alongside `plannerOversightLevel`. See `docs/architecture.md` → "Executor-stage stall detection (FN-7743)". |
When `triageProactiveSubtaskSplittingEnabled` is `true` (the default), triage may proactively replace a large task with 2-5 child tasks when the size, step-count, package breadth, file-scope, or remediation-batch signals justify the coordination overhead. When it is `false`, those automatic oversized-task signals are advisory only for writing a realistic single-task spec; triage must not split solely because the task is large. The per-task `breakIntoSubtasks: true` flag is separate and remains mandatory: if a user explicitly asks for subtask breakdown, triage still evaluates and creates child tasks when the work is meaningfully decomposable.