fusion/docs/shared-mesh-protocol.md

# Shared Mesh Replication Protocol (v1)

[← Docs index](./README.md)

This document is the canonical contract for Fusion multi-leader mesh replication.

## 1. Goals and non-goals

### Goals
- Preserve one shared durable project state across multiple nodes.
- Keep task and planning state strongly coordinated by default.
- Allow local progress during peer outages via durable queues.
- Support deterministic replay/reconciliation after recovery.
- Expose read staleness so clients can decide whether to trust last-known global state.

### Non-goals (for v1)
- Full runtime scheduler failover.
- Full live-process state migration.
- Immediate global consistency for every data class.

## 2. Terms

- **Node**: A Fusion runtime instance participating in mesh sync.
- **Coordinator**: Node currently responsible for committing a write intent.
- **Intent**: Durable write proposal before global ack quorum completes.
- **Envelope**: Wire record carrying replication metadata + payload.
- **Epoch**: Monotonic lease/fencing generation for coordinator authority.
- **Fence token**: `epoch + coordinatorNodeId + sequence` token that invalidates stale coordinators.
- **Queue entry**: Durable locally-accepted write waiting for replay.

## 3. Versioning

- Protocol id: `fusion.shared-mesh`
- Initial version: `1.0`
- All envelopes must include `{ protocol, version }`.
- Minor versions (`1.x`) are backward-compatible additive.
- Major versions (`2.0+`) may change semantics and require explicit compatibility checks.

## 4. Data-class coordination matrix

| Data class | Mode | Notes |
|---|---|---|
| Tasks (core fields, deps, steps, column transitions) | Strongly coordinated | Quorum-acked intent/commit path; replayable with fencing |
| Task metadata (priority, model overrides, docs metadata refs) | Strongly coordinated | Same write path as tasks |
| Missions/milestones/slices/features | Strongly coordinated | Ordered writes preserve hierarchy invariants |
| Agent definitions/configuration | Strongly coordinated | Durable config replicated; runtime process handles excluded |
| Agent runtime state (heartbeat ticks, local process internals, worktree paths) | Node-local only | Exposed as local telemetry, not global truth |
| Project settings | Strongly coordinated | Existing settings payloads remain canonical payload shape |
| Auth material / provider credentials | Queued-for-later (secured transport only) | Explicit auth channel; never merged as ordinary settings data |
| Execution runs / live activity streams | Node-local + queued summary | Live events local; durable run outcomes appended later |
| Audit / event streams (`activityLog`, `runAuditEvents`) | Append-only replicated | Immutable event replication with origin metadata |
| Filesystem blobs (`.fusion/tasks/*` prompts/logs/attachments) | Queued-for-later | Metadata in replicated records, blob transfer out-of-band |

## 5. Write classes

- **`strong`**: Requires coordinator fence + quorum ack before `committed`.
- **`append-only`**: Event-style immutable replication; dedupe by event id.
- **`queued`**: Accept locally when peers unavailable; replay later.
- **`local`**: Never replicated globally.

## 6. Replication envelope

Every replicated record uses:
- `protocol`, `version`
- `recordId`, `entityType`, `entityId`
- `originNodeId`, `originSeq`
- `writeClass`
- `leaseEpoch`, `fenceToken`
- `intentId` and `state` (`intent` | `committed` | `rejected` | `queued` | `reconciled`)
- `createdAt`, `committedAt?`
- `payload`
- `precondition?` (base revision / expected epoch)

`PeerSyncRequest` / `PeerSyncResponse` remain mesh exchange carriers. v1 envelopes are payloads exchanged through current mesh sync infrastructure and follow-on sync endpoints.

## 7. Quorum and acknowledgements

For `strong` writes:
1. Coordinator accepts intent locally.
2. Coordinator requests acknowledgements from peers in current membership view.
3. Commit requires `quorum = floor(eligibleVoters / 2) + 1` including coordinator.
4. If quorum fails before timeout, intent becomes `queued` with retry metadata.

`append-only` writes can be accepted locally and replicated asynchronously, but must preserve origin ordering `(originNodeId, originSeq)`.

## 8. Lease epochs and fencing

- Coordinator authority is leased with a monotonic `leaseEpoch`.
- Any write with stale epoch/fence must be rejected (`fenced`).
- Restarted nodes must reacquire lease and increment epoch before coordinating strong writes.
- Replay workers must carry original fence metadata; reconciler can reject stale queued entries after epoch advancement.

## 9. Offline queueing and replay

When a strong/queued write cannot reach quorum:
- Persist queue entry durably with:
  - `intentId`, `entityType`, `entityId`, `writeClass`
  - `originNodeId`, `originSeq`, `leaseEpoch`, `fenceToken`
  - retry counters, first/last attempt timestamps, next attempt time
- Local node may expose optimistic local result as `queued` only (not globally committed).

Replay ordering:
1. Sort by `(leaseEpoch asc, originSeq asc, createdAt asc, intentId asc)`.
2. Re-validate preconditions and fence tokens.
3. Commit, reject, or reconcile with deterministic outcome.

## 10. Reconciliation

Reconciliation outcomes are explicit:
- `applied` — replayed successfully.
- `noop_already_applied` — idempotent duplicate.
- `superseded` — newer committed revision already exists.
- `conflict_requires_merge` — semantic conflict; requires policy/agent/manual resolution.
- `rejected_fenced` — stale epoch/fence.

Conflict policy must never silently downgrade strong writes to local-only updates.

## 11. Restart recovery hooks

On node startup:
1. Load durable queue.
2. Rebuild last known lease epoch / origin sequence.
3. Mark in-flight intents without terminal state as `queued` recovery candidates.
4. Start replay loop only after mesh membership snapshot and lease status are known.

## 12. Degraded reads and staleness

Read responses for shared entities include staleness metadata:
- `source`: `local-committed` | `local-queued` | `replica`
- `lastGlobalCommitAt`
- `replicationLagMs`
- `queueDepth`
- `isStale`

In degraded mode, clients may read last-known global state plus queued-local overlays, but must be able to distinguish them.

## 13. End-to-end v1 write path

1. **Intent creation**: Node creates write intent + envelope.
2. **Coordinator selection**: Node routes to current coordinator lease holder for the entity scope.
3. **Commit/ack**:
   - strong: quorum commit
   - append-only: local append + async replication
4. **Fallback**: if unreachable/quorum-fail, persist queue entry (`queued`).
5. **Replay**: on recovery, replay durable queue in canonical order with fencing checks.
6. **Reconciliation**: produce explicit outcome and update entity revision state.

## 14. Contract for FN-3449 through FN-3456

Follow-on tasks must implement against this contract and not redefine it:
- **FN-3449**: distributed ids/origin sequence allocation + monotonic ordering.
- **FN-3450**: coordinator selection and lease management runtime.
- **FN-3451**: strong-write commit path + quorum ack handling.
- **FN-3452**: durable offline queue persistence and replay engine.
- **FN-3453**: reconciliation executor + conflict outcome handling.
- **FN-3454**: restart recovery bootstrap and in-flight intent recovery.
- **FN-3455**: degraded-read APIs exposing staleness metadata.
- **FN-3456**: partition behavior policy, observability, and operator controls.

## 15. Security boundary

- Mesh transport authentication (node API keys / trust) is mandatory for replication traffic.
- Auth credential replication is explicit and separately controlled from ordinary settings replication.
- Sensitive payloads must be redacted from non-secure logs and diagnostics.