feat(FN-3455): document outage tolerance contracts in mesh protocol docs

Documents outage tolerance contracts across three architecture docs, with the bulk of changes in `docs/shared-mesh-protocol.md` plus minor additions to the main architecture and multi-project guides.

Fusion-Task-Id: FN-3455
This commit is contained in:
Fusion
2026-05-10 18:19:29 -07:00
committed by gsxdsm
parent 1e76f247e4
commit 294f98c0e0
3 changed files with 36 additions and 15 deletions

View File

@@ -620,6 +620,12 @@ Implemented in `agent-heartbeat.ts`:
- `PeerExchangeService` (`peer-exchange-service.ts`) — peer sync orchestration
- `MeshLeaseManager` (`mesh-lease-manager.ts`) — canonical abandoned-lease detection + recovery path
### Outage ownership boundaries (degraded reads + queued write replay)
- `CentralCore` owns durable outage state in central persistence (`meshSharedSnapshots` + `meshWriteQueue`) and computes canonical degraded-read metadata (`MeshDegradedReadState`) for API consumers.
- `PeerExchangeService` owns retryability classification for sync/apply failures, queue insertion for retryable failures, and replay execution after successful peer sync using queue ordering `(createdAt ASC, id ASC)`.
- `NodeHealthMonitor` provides liveness transitions as replay hints only; `online` is a trigger to attempt replay, not proof that replay succeeded.
- Dashboard mesh routes (`register-mesh-routes.ts`) preserve `GET /api/mesh/state` array shape and attach per-node degraded `readState` metadata so stale fallback data is explicit during partitions.
### Mesh task lease ownership and recovery
Task ownership is persisted in shared task metadata so all nodes agree on one canonical lease view. The persisted lease fields are:

View File

@@ -34,6 +34,7 @@ Per-project task data remains in each repos `.fusion/fusion.db`.
Peer/mesh coordination spans core + engine, with startup ownership in CLI process entrypoints:
- Topology visibility is now cluster-wide from any connected node: dashboard mesh reads aggregate remote local snapshots and dedupe by `nodeId`, with fallback to last-known local mesh state when a peer is temporarily unreachable.
- Outage tolerance persistence is central and project-scoped: degraded mesh snapshots and queued write replay rows are stored with `projectId` keys so partitions in one project do not blur reconciliation state across other registered projects.
- `NodeDiscovery` and `NodeConnection` in `@fusion/core` handle discovery and remote node connectivity/auth primitives.
- `PeerExchangeService` in `@fusion/engine` coordinates node-to-node sync/exchange workflows.
- `MeshLeaseManager` in `@fusion/engine` is the single authority for stale lease detection and abandoned-work recovery across nodes.

View File

@@ -110,16 +110,15 @@ For `strong` writes:
## 9. Offline queueing and replay
When a strong/queued write cannot reach quorum:
- Persist queue entry durably with:
- `intentId`, `entityType`, `entityId`, `writeClass`
- `originNodeId`, `originSeq`, `leaseEpoch`, `fenceToken`
- retry counters, first/last attempt timestamps, next attempt time
- Local node may expose optimistic local result as `queued` only (not globally committed).
- Persist queue entry durably in `meshWriteQueue` (`status`: `pending | replaying | applied | failed`).
- Retryable queueing is limited to transport/outage failures: HTTP `502/503/504`, timeout/abort, and transport errors (`TypeError` fetch/network rejections, or Node-style `ECONNREFUSED`, `ENOTFOUND`, `ETIMEDOUT`, `ECONNRESET`).
- Non-retryable HTTP failures (`400/401/403/404/409/422`) are recorded as immediate failures and are not queued for replay.
- `applied` and `failed` rows are retained as durable reconciliation history (no auto-cleanup in v1).
Replay ordering:
1. Sort by `(leaseEpoch asc, originSeq asc, createdAt asc, intentId asc)`.
2. Re-validate preconditions and fence tokens.
3. Commit, reject, or reconcile with deterministic outcome.
1. Sort by `(createdAt asc, id asc)`.
2. Transition idempotently on the same row (`pending → replaying → applied|failed`) while incrementing `attemptCount` on each attempt.
3. Re-validate preconditions/fencing and record deterministic outcome.
## 10. Reconciliation
@@ -142,14 +141,29 @@ On node startup:
## 12. Degraded reads and staleness
Read responses for shared entities include staleness metadata:
- `source`: `local-committed` | `local-queued` | `replica`
- `lastGlobalCommitAt`
- `replicationLagMs`
- `queueDepth`
- `isStale`
Mesh state reads expose a canonical `MeshDegradedReadState` contract:
In degraded mode, clients may read last-known global state plus queued-local overlays, but must be able to distinguish them.
```ts
type MeshDegradedReadState = {
mode: "fresh" | "degraded";
asOf: string;
sourceNodeId: string | null;
snapshotVersion: string | null;
stalenessMs: number;
queueDepth: number;
pendingWriteCount: number;
failedWriteCount: number;
};
```
Rules:
- `mode="fresh"` only when data came from the live mesh read path.
- `mode="degraded"` when read falls back to `meshSharedSnapshots`.
- `asOf` comes from snapshot `capturedAt`; `stalenessMs = Date.now() - new Date(asOf).getTime()`.
- `snapshotVersion` is the stored 64-char SHA-256 hex digest of snapshot payload.
- `queueDepth` is computed from queue rows where `status IN ('pending','replaying','failed')`.
API behavior must never hide fallback mode: degraded reads are explicit so clients can distinguish stale last-known state from fresh cluster state.
## 13. End-to-end v1 write path