feat(FN-3452): document mesh lease recovery semantics
Documents mesh lease recovery semantics across the agents, architecture, and multi-project reference files, adding 32 lines of clarifying documentation to explain how mesh leases are recovered in the system. Fusion-Task-Id: FN-3452
This commit is contained in:
@@ -850,6 +850,8 @@ Implementation tasks require an agent with `role: "executor"`.
|
||||
|
||||
## Heartbeat Monitoring and Trigger Scheduling
|
||||
|
||||
Heartbeat/executor ownership now actively renews persisted task lease metadata while work is running (`checkoutLeaseRenewedAt` plus owner node/run context). Abandonment recovery is fenced by `checkoutLeaseEpoch` and executed only through `MeshLeaseManager.recoverAbandonedLease(...)`, so stale owners cannot reclaim tasks after recovery.
|
||||
|
||||
Fusion's `HeartbeatTriggerScheduler` supports five trigger types:
|
||||
|
||||
- `timer` — periodic wake based on heartbeat interval
|
||||
|
||||
@@ -591,6 +591,27 @@ Implemented in `agent-heartbeat.ts`:
|
||||
### Node/mesh runtime services
|
||||
- `NodeHealthMonitor` (`node-health-monitor.ts`) — remote node liveness/metrics checks
|
||||
- `PeerExchangeService` (`peer-exchange-service.ts`) — peer sync orchestration
|
||||
- `MeshLeaseManager` (`mesh-lease-manager.ts`) — canonical abandoned-lease detection + recovery path
|
||||
|
||||
### Mesh task lease ownership and recovery
|
||||
|
||||
Task ownership is persisted in shared task metadata so all nodes agree on one canonical lease view. The persisted lease fields are:
|
||||
|
||||
- `checkedOutBy` — owning agent id (compatibility field)
|
||||
- `checkedOutAt` — lease acquisition timestamp (compatibility field)
|
||||
- `checkoutNodeId` — owning node id
|
||||
- `checkoutRunId` — active owning heartbeat/executor run id when known
|
||||
- `checkoutLeaseRenewedAt` — last successful lease renewal timestamp
|
||||
- `checkoutLeaseEpoch` — monotonic fencing generation used to reject stale owners after recovery
|
||||
|
||||
`AgentStore.checkoutTask()` remains the compatibility entrypoint for ownership claims, but lease replacement is fenced by epoch semantics: only the same live owner can renew idempotently, and stale owner replacement is performed only through the recovery path.
|
||||
|
||||
`MeshLeaseManager.recoverAbandonedLease(taskId, reason, context)` is the single canonical abandoned-work path used by scheduler/self-healing/runtime orchestration. Recovery validates staleness, bumps `checkoutLeaseEpoch`, clears active-owner fields, logs the reason, and re-queues work for scheduler visibility.
|
||||
|
||||
A lease is recoverable only when there is **no active local executor session for that task** and either:
|
||||
|
||||
1. the owning node is `offline` or `error`, or
|
||||
2. the owner heartbeat/run age exceeds `max(agentHeartbeatTimeoutMs * 2, 120_000)` measured against the most recent lease renewal timestamp.
|
||||
- Canonical replication/write-coordination contract: [`docs/shared-mesh-protocol.md`](./shared-mesh-protocol.md)
|
||||
- Defines protocol versioning, write classes, quorum/ack semantics, lease epochs/fencing, offline queue/replay, reconciliation outcomes, restart recovery hooks, and degraded-read staleness metadata.
|
||||
- Existing `/api/mesh/sync` and settings-sync payloads remain the active exchange primitives while follow-on runtime tasks implement full v1 coordinator/quorum behavior.
|
||||
|
||||
@@ -34,6 +34,7 @@ Per-project task data remains in each repo’s `.fusion/fusion.db`.
|
||||
Peer/mesh coordination spans core + engine, with startup ownership in CLI process entrypoints:
|
||||
- `NodeDiscovery` and `NodeConnection` in `@fusion/core` handle discovery and remote node connectivity/auth primitives.
|
||||
- `PeerExchangeService` in `@fusion/engine` coordinates node-to-node sync/exchange workflows.
|
||||
- `MeshLeaseManager` in `@fusion/engine` is the single authority for stale lease detection and abandoned-work recovery across nodes.
|
||||
- Canonical replication semantics live in [`docs/shared-mesh-protocol.md`](./shared-mesh-protocol.md). That protocol separates strongly coordinated shared state from append-only streams, queued replay classes, and node-local runtime state.
|
||||
- Distributed task-ID allocation is one strongly coordinated shared-state path: reserve/commit/abort are coordinator-mediated writes, and cluster-wide committed task totals come from allocator `committedClusterTaskCount` state (not per-node local task counts).
|
||||
- `runServe()` and `runDashboard()` (CLI) own process-level mesh service lifecycle:
|
||||
@@ -42,6 +43,14 @@ Peer/mesh coordination spans core + engine, with startup ownership in CLI proces
|
||||
- stop peer exchange + discovery on shutdown
|
||||
- `InProcessRuntime` remains project-scoped (scheduler/executor/heartbeat/missions) and does **not** start mesh services, which avoids one peer-exchange instance per project.
|
||||
|
||||
## Mesh lease recovery in multi-node execution
|
||||
|
||||
Task ownership is shared as persisted lease metadata (`checkedOutBy`, `checkedOutAt`, `checkoutNodeId`, `checkoutRunId`, `checkoutLeaseRenewedAt`, `checkoutLeaseEpoch`) through the canonical mesh sync payloads.
|
||||
|
||||
When a node disappears or stops renewing ownership, recovery is routed only through `MeshLeaseManager.recoverAbandonedLease(...)`. The manager releases ownership only after staleness checks pass and no active local executor session exists for the task. Recovery then bumps `checkoutLeaseEpoch`, clears owner fields, logs the abandonment reason, and returns the task to scheduler-visible work.
|
||||
|
||||
This fencing prevents double-claims: a restarted or delayed stale owner cannot reclaim work using older epoch state once recovery has advanced the lease generation.
|
||||
|
||||
## Registering and Managing Projects
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user