root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(engine): unstick agents in running and clean up ephemeral worker pile-up

Browse Source 
Two related leaks in the agent lifecycle plus a refactor:

- Governance-skip paths in executeHeartbeat (budget/global-pause/engine-paused)
  were leaving agents permanently stuck in `running` because they ran startRun
  first and then short-circuited with skipStateTransition: true. Removed the
  flag from those four paths so they flow through running → active. Added
  HeartbeatMonitor.reconcileOrphanedRunningAgents() on start to recover any
  rows already trapped in this state.
- Ephemeral task-workers piled up across runtime restarts because taskAgentMap
  was in-memory only and the startup sweep ignored ephemerals with no taskId.
  Now: spawn dedup via findAgentByName before create, on-disk fallback in
  finalize when the in-memory map is empty, and the sweep deletes any
  ephemeral not bound to an in-progress task.
- Extracted the lifecycle into EphemeralWorkerManager
  (packages/engine/src/ephemeral-worker-manager.ts). InProcessRuntime drops
  ~140 lines and delegates via onTaskStart/onTaskComplete/onTaskError/
  attachStateChangeListener/reconcileOrphaned. ChildProcessRuntime and
  RemoteNodeRuntime inherit the fix because they delegate execution to a
  worker that runs InProcessRuntime.

Durable assigned agents now return to `active` after task completion (was
`terminated` in the old contract).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
gsxdsm
2026-05-05 19:11:49 -07:00
parent 9d8589a149
commit d47501feeb
5 changed files with 450 additions and 249 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
.changeset/ephemeral-worker-lifecycle.md Normal file
View File

@@ -0,0 +1,11 @@
---
"@runfusion/fusion": patch
---
Fix two related agent-lifecycle leaks and extract the coordinator into a reusable class.
**Stuck-in-running bug.** `executeHeartbeat`'s governance-skip paths (budget exhausted, budget threshold, global pause, engine paused) called `startRun` first — flipping the agent to `running` — then short-circuited with `skipStateTransition: true`, leaving the agent permanently stuck at `running` with no active run. Removed the `skipStateTransition` flag from those four paths so they flow through the normal `running → active` transition. Added `HeartbeatMonitor.reconcileOrphanedRunningAgents()` on startup to recover any agents already trapped in this state from older versions.
**Ephemeral task-worker pile-up.** Runtime-spawned `executor-FN-XXXX` workers leaked across runtime restarts because the in-memory `taskAgentMap` reset every process and there was no on-disk fallback. A task started in one session and completed in another would orphan its worker; over time hundreds piled up. The startup sweep also only deleted ephemerals in halt states, ignoring the no-`taskId` case that accounted for nearly every zombie. Now: spawn dedup via `findAgentByName` lookup before create, on-disk fallback in completion/error paths, and the startup sweep deletes any ephemeral not bound to an in-progress task.
**`EphemeralWorkerManager` extraction.** The lifecycle logic is now a single class (`packages/engine/src/ephemeral-worker-manager.ts`) owning `taskAgentMap`, `pendingDeletions`, the halt-state listener, and the startup sweep. `InProcessRuntime` shrinks by ~140 lines and delegates via `workerManager.onTaskStart` / `.onTaskComplete` / `.onTaskError` / `.attachStateChangeListener` / `.reconcileOrphaned`. Future runtimes that drive `TaskExecutor` directly inherit the same lifecycle. Durable assigned agents now return to `active` after task completion (was `terminated` in the old contract).