feat(FN-3452): document mesh lease recovery semantics

Documents mesh lease recovery semantics across the agents, architecture, and multi-project reference files, adding 32 lines of clarifying documentation to explain how mesh leases are recovered in the system.

Fusion-Task-Id: FN-3452
This commit is contained in:
Fusion
2026-05-09 11:24:24 -07:00
committed by gsxdsm
parent 0bd460bb7f
commit e0a84b27fb
16 changed files with 486 additions and 22 deletions

View File

@@ -591,6 +591,27 @@ Implemented in `agent-heartbeat.ts`:
### Node/mesh runtime services
- `NodeHealthMonitor` (`node-health-monitor.ts`) — remote node liveness/metrics checks
- `PeerExchangeService` (`peer-exchange-service.ts`) — peer sync orchestration
- `MeshLeaseManager` (`mesh-lease-manager.ts`) — canonical abandoned-lease detection + recovery path
### Mesh task lease ownership and recovery
Task ownership is persisted in shared task metadata so all nodes agree on one canonical lease view. The persisted lease fields are:
- `checkedOutBy` — owning agent id (compatibility field)
- `checkedOutAt` — lease acquisition timestamp (compatibility field)
- `checkoutNodeId` — owning node id
- `checkoutRunId` — active owning heartbeat/executor run id when known
- `checkoutLeaseRenewedAt` — last successful lease renewal timestamp
- `checkoutLeaseEpoch` — monotonic fencing generation used to reject stale owners after recovery
`AgentStore.checkoutTask()` remains the compatibility entrypoint for ownership claims, but lease replacement is fenced by epoch semantics: only the same live owner can renew idempotently, and stale owner replacement is performed only through the recovery path.
`MeshLeaseManager.recoverAbandonedLease(taskId, reason, context)` is the single canonical abandoned-work path used by scheduler/self-healing/runtime orchestration. Recovery validates staleness, bumps `checkoutLeaseEpoch`, clears active-owner fields, logs the reason, and re-queues work for scheduler visibility.
A lease is recoverable only when there is **no active local executor session for that task** and either:
1. the owning node is `offline` or `error`, or
2. the owner heartbeat/run age exceeds `max(agentHeartbeatTimeoutMs * 2, 120_000)` measured against the most recent lease renewal timestamp.
- Canonical replication/write-coordination contract: [`docs/shared-mesh-protocol.md`](./shared-mesh-protocol.md)
- Defines protocol versioning, write classes, quorum/ack semantics, lease epochs/fencing, offline queue/replay, reconciliation outcomes, restart recovery hooks, and degraded-read staleness metadata.
- Existing `/api/mesh/sync` and settings-sync payloads remain the active exchange primitives while follow-on runtime tasks implement full v1 coordinator/quorum behavior.