## Problem
Permanent (durable) agent heartbeats went silent while the rest of the
engine kept running. Investigation of the live DB showed **every**
permanent agent's `heartbeatTimerRepair` metadata carrying
`nonAdvancingEscalated: true` with **3–63 consecutive** zombie re-arms —
the audit re-arming a timer every 60s for hours while emitting
`heartbeat-rearm-nonadvancing-escalated` warnings that never recovered
anything.
## Root cause
The heartbeat trigger audit classified a "zombie" (dead) timer **solely
from a stale `lastHeartbeatAt`**. But that column advances *only* on a
successful `"ok"` delivery (`agent-store.ts` `recordHeartbeat`). It
stays frozen whenever a heartbeat is intentionally skipped or no-op'd:
- agent over budget / over budget threshold
- `globalPause` / `enginePaused`
- `skipHeartbeatWhenIdle` on an idle agent
- idle "org" agents whose runs complete as `no_assignment_identity_run`
In all of these the interval keeps firing perfectly — the timer is
alive, delivery is just (correctly) skipped. Keying zombie detection off
`lastHeartbeatAt` misread those healthy timers as dead, re-armed them
every 60s, and escalated forever. Re-arming a live timer is a no-op, so
the loop could never recover — it only produced churn and phantom
warnings.
## Fix (the invariant)
Key zombie detection off **whether the interval physically fired**, not
whether delivery advanced.
- New `lastTimerFireAtMs` map, stamped at the top of `onTimerTick`
**before any gate** — a fired-but-skipped tick still counts as proof of
liveness.
- In the audit: a present + stale timer that fired within its stale
window is **left untouched** (no re-arm, no escalation, non-advancing
counter reset). Only a timer with **no recent fire** (a genuinely dead
interval) falls through to the existing re-arm/escalation path.
- Map cleaned up in `unregisterAgent()` / `stop()`.
This preserves the FN-7645 zombie repair (a timer that stops firing goes
stale in lockstep on both clocks and is still re-armed) and the FN-7939
watchdog, while eliminating the phantom churn for live-but-skipping
timers.
Why not "force a heartbeat" or "park the agent": forcing delivery would
bypass budget/pause governance, and parking a healthy idle agent would
be wrong. The correct action for a live-but-skipping timer is to leave
it alone — its next real tick delivers once the skip condition clears.
## Tests
- Rewrote the old `skipHeartbeatWhenIdle` test that codified the buggy
escalation → now asserts a **live** idle-skipping timer is left
untouched (no zombie re-arm, no escalation).
- Added a budget/no-assignment surface: a live timer that dispatches but
leaves `lastHeartbeatAt` frozen must not be misclassified.
`heartbeat-scheduler.test.ts` 120/120; broader heartbeat + concurrency
suites 349/349; `@fusion/engine` typecheck 0 errors.
## Review
Self-reviewed at medium effort. Two acknowledged, bounded trade-offs
(kept intentionally): a genuinely-dead-but-recently-fired timer's repair
latency is bounded at ~2× interval (same as the original FN-7645
latency), and the escalation warning is suppressed for live timers (it
only ever fired because of the churn this removes; per-tick error logs +
a new "left live-but-skipping timer" log retain visibility). One trivial
cleanup applied (single `Date.now()` sample).
## Notes
- Engine is a private package → no changeset.
- Complementary to a separate in-flight fix for the
agentStore/scheduler-not-constructed bug (why heartbeats stopped
*entirely*); this PR ensures that once the scheduler runs again, the
audit stops the phantom churn/escalation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved heartbeat timer monitoring to distinguish healthy timers from
genuinely stopped timers.
* Prevented unnecessary timer re-registration and warning escalation
when heartbeats are intentionally skipped due to idle, paused,
budget-limited, or unassigned states.
* Improved recovery when a replacement timer stops firing, ensuring it
is detected and repaired reliably.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
a7ab3ccb49