root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: auto-recover orphaned heartbeat runs from crashed processes

Browse Source 
When the dashboard crashes mid-heartbeat, the agentRuns row is left in
status='active' forever. HeartbeatTriggerScheduler.onTimerTick treats
any active run as "still running" and skips every subsequent tick, so
agents go silent indefinitely (observed: 6+ hours). The existing
in-memory missed-heartbeat watchdog can't help — its trackedAgents map
is wiped on process restart.

SelfHealingManager.recoverStaleHeartbeatRuns now reconciles these on
startup and during periodic maintenance: terminates active runs whose
processPid does not match the current process, has no recorded pid, or
has been active for more than 6 hours.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
gsxdsm
2026-05-05 14:13:07 -07:00
parent c67e786f48
commit 9b01c0a369
4 changed files with 229 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.changeset/auto-recover-orphan-heartbeat-runs.md Normal file
View File

@@ -0,0 +1,5 @@
---
"@runfusion/fusion": patch
---
Self-heal orphaned `agentRuns` rows left in `status='active'` when the dashboard process crashes mid-heartbeat. The trigger scheduler treats any active run as "still running" and silently skips every subsequent tick, so a single crashed run could leave an agent without heartbeats for hours. SelfHealingManager now reconciles these on startup and during periodic maintenance, terminating runs whose `processPid` does not match the current process or whose age exceeds 6 hours.