root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
Files
96b1f21707442afff7b071fbd9562dadbf165ac9
fusion/docs/shared-mesh-protocol.md
gsxdsm 85f8b1f909 feat: shared Postgres multi-node — retire mesh data-plane replication (#2130) 
## Summary

- Treat **shared PostgreSQL** (`DATABASE_URL`) as the multi-node durable
data plane; mesh HTTP is membership + optional auth, not task/settings
replication.
- **Peer exchange**: under Postgres backend mode, write queue is
**topology/auth-only**; non-topology pending rows fail rather than
replaying multi-leader task/settings payloads.
- **Mesh routes**: task-ID reserve/commit/abort always hit local shared
allocator rows (ignore remote `coordinatorNodeId`); mesh sync ignores
settings and only exchanges `authMaterial`.
- **Docs**: rewrite multi-project runbook, shared cluster protocol, and
architecture mesh sections for shared-Postgres + claims/leases.

## Context

Follows the SQLite→Postgres cutover. Multiple Fusion nodes can share one
external Postgres while keeping **per-node execution** (worktrees,
processes, claims via `central.task_claims`). Explicit non-goals remain:
scheduler failover and live process migration.

Plan:
`docs/plans/2026-07-15-001-refactor-mesh-shared-postgres-multinode-plan.md`

## Test plan

- [x] `pnpm --filter @fusion/engine exec vitest run
src/__tests__/peer-exchange-service.test.ts`
- [x] `pnpm --filter @fusion/dashboard exec vitest run
src/__tests__/mesh-routes.test.ts`
- [x] `pnpm --filter @fusion/core exec vitest run
src/__tests__/shared-mesh-state.test.ts`
- [ ] CI gate (lint/typecheck/build/gate)
- [ ] Manual (optional): two processes, same `DATABASE_URL`, create task
on A visible on B; settings change without mesh settings sync; claim
exclusivity

## Operator note

Multi-node shared board requires **external** `DATABASE_URL` on every
node. Default embedded Postgres is still single-host.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Improved multi-node deployments using shared PostgreSQL as the durable
source of execution state.
* Task ID reservation/commit/abort now run locally (no remote
coordinator forwarding).
* Mesh syncing now prioritizes topology visibility and authentication
material; settings replication is disabled in shared-Postgres mode.
* **Bug Fixes**
* Prevented task/settings replication over mesh HTTP in shared-Postgres
deployments.
* Refined lease ownership, recovery, and reconciliation to converge via
shared-database primitives.
* **Documentation**
* Updated architecture and shared-mesh protocol guidance, including
multi-node setup and lease/task-ID allocation behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-15 13:32:33 -07:00

136 lines
7.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Shared Cluster Protocol (Postgres multi-node)
[← Docs index](./README.md)
<!--
FNXC:SharedPostgresMultiNode 2026-07-14-23:45:
This document supersedes the multi-leader SQLite mesh replication contract. Durable project state is shared PostgreSQL; mesh HTTP is membership, optional auth material, and execution ownership — not a second database.
-->
This document is the canonical contract for Fusion **multi-node operation on shared PostgreSQL**.
The historical multi-leader SQLite mesh (HTTP task replication, settings gossip, strong-write quorum, offline task write queues) is **retired**. Nodes that share `DATABASE_URL` already share durable state at the database layer.
## 1. Goals and non-goals
### Goals
- One shared durable project + central state across multiple Fusion nodes.
- Exclusive execution ownership per task via central claims + lease epochs.
- Per-node worktrees, processes, and path mappings without live process migration.
- Explicit degraded topology reads when peer **HTTP** health probes fail (membership visibility), without inventing divergent local task truth.
### Non-goals
- Scheduler failover (a peer does not take over another node’s live scheduler tick loop).
- Live-process / in-memory session migration mid-task.
- Multi-leader task writes when Postgres is unavailable (if the DB is down, nodes do not queue alternate task realities over HTTP).
- Treating embedded Postgres as a multi-host shared backend (embedded is per-machine only).
Supported recovery model: **lease handoff** under `OwningNodeHandoffPolicy` (`park`, `reassign-to-local`, `reassign-any-healthy`) so a healthy node resumes from **durable** task state.
## 2. Terms
- **Node**: A Fusion runtime/API process with a registered `central.nodes` row and local execution capacity.
- **Shared database**: One Postgres cluster (schemas `project`, `central`, `archive`) reached via the same `DATABASE_URL` on every participating node.
- **Claim**: Authoritative ownership row in `central.task_claims` keyed by `(projectId, taskId)`.
- **Lease epoch**: Monotonic fencing generation on the task row that invalidates stale owners after recovery.
- **Membership gossip**: Optional peer HTTP exchange of known peers / metrics; does not carry task or settings payloads under Postgres.
- **Auth material**: Provider credentials in per-machine `auth.json` (not in the shared DB by default); optional secure HTTP sync remains.
## 3. Data-class matrix (current truth)
| Data class | Mode | Notes |
|---|---|---|
| Tasks, deps, steps, columns | **Shared Postgres** | Commit is cluster-visible; no HTTP task replication |
| Missions / agents config / workflows / audit | **Shared Postgres** | Same |
| Project + global settings | **Shared Postgres** | Settings HTTP push/pull between nodes is disabled (`409`) |
| Distributed task IDs | **Shared Postgres** | `distributed_task_id_state` / `_reservations`; always local allocator against shared rows |
| Checkout ownership | **Central claim + task mirror** | `task_claims` then task lease columns |
| Agent runtime / worktrees / live sessions | **Node-local** | Paths may differ via `project_node_path_mappings` |
| Auth credentials (`auth.json`) | **Node-local + optional sync** | `sharedState.authMaterial` / auth routes only |
| FS blobs (`.fusion/tasks/*`) | **Node-local** | Metadata may be in PG; bytes on the materializing host |
| Topology / peer metrics | **Registry + probes** | `central.nodes` / peers; optional gossip + health HTTP |
## 4. Execution ownership
### Claim path
1. `AgentStore.checkoutTask` → `CentralClaimStore.tryClaimTask` (`central.task_claims`).
2. Mirror winner onto the task row (`tryClaimCheckout`: `checkedOutBy`, `checkoutNodeId`, `checkoutRunId`, `checkoutLeaseRenewedAt`, `checkoutLeaseEpoch`).
3. Scheduler/executor on the winning node run locally; other nodes must not start a second exclusive execution lane for the same claim.
### Recovery path
Only `MeshLeaseManager.recoverAbandonedLease(...)`:
1. Prove recoverable (owner offline/error, or lease/heartbeat stale; not active local execution).
2. Apply handoff policy when configured.
3. Release **central claim first**, then clear task lease fields and **bump epoch**.
4. Requeue to `todo` (preserve progress when appropriate).
5. Partial split-brain → `reconcileLeaseRow` on a later tick.
Run-audit: `task:auto-recover-lease-*`, `node:lease:*`, `node:handoff:*` as applicable.
## 5. Membership and HTTP mesh surfaces
Still useful under shared Postgres:
| Surface | Role |
|---|---|
| `GET /api/mesh/state` | Topology snapshot for dashboard Nodes UI |
| `POST /api/mesh/sync` | Peer gossip: `knownPeers` (+ optional `authMaterial` only) |
| `POST/GET /api/mesh/task-ids/*` | Local allocator against shared ID tables (no remote coordinator hop) |
| Auth sync routes | Optional credential fan-out for file-local auth |
| mDNS discovery | Join convenience, not task SoT |
| Docker mesh config generator | Provision managed peers |
Removed / disabled:
| Surface | Status |
|---|---|
| `POST /api/mesh/tasks/create` | Removed — DB is the replication plane |
| Task/agent/mission/audit shared-state domains | Removed |
| Settings gossip / node settings push-pull | Disabled on Postgres (`409`) |
| Remote task-ID coordinator forwarding | Disabled on Postgres |
## 6. Write queue and degraded topology (narrowed)
Historical multi-leader design used `meshWriteQueue` for offline **task** write replay and `meshSharedSnapshots` for last-known global task state.
Under shared Postgres:
- **Do not** invent local task commits when Postgres is unavailable.
- `meshWriteQueue` is limited to **topology / auth** retry classes (membership sync / auth material), not task or settings payloads.
- `meshSharedSnapshots` support **degraded membership/topology** reads only; they are not a substitute board store.
- `PeerExchangeService.replayPendingWritesForNode` replays only those narrow scopes.
If Postgres is down, operators fix the database; nodes do not multi-master task rows over HTTP.
## 7. Process lifecycle
- `fn serve` / `fn dashboard` start one process-wide `PeerExchangeService` and call `CentralCore.startDiscovery()` after the HTTP server binds the real port.
- `InProcessRuntime` is project-scoped (scheduler/executor/heartbeat) and does **not** start mesh services.
- `HybridExecutor` remains the multi-project / multi-node orchestration path when the hybrid gate enables it.
## 8. Security boundary
- Peer HTTP (sync, auth, remote isolation runtime) requires node API-key authentication when configured.
- Never log raw secrets from auth snapshots.
- Database credentials in `DATABASE_URL` must not appear in logs (redaction helpers in the Postgres connection layer).
## 9. Operator checklist
See the **Shared Postgres multi-node runbook** in [`docs/multi-project.md`](./multi-project.md).
Short form:
1. Same external `DATABASE_URL` on every node.
2. Register nodes/projects + path mappings per host.
3. Run engines; claims enforce exclusive execution.
4. Expect worktrees/auth/blobs to remain node-local unless you opt into auth-sync or a future blob store.
## 10. Historical note
Earlier revisions of this file described protocol id `fusion.shared-mesh` v1 with strong/queued/append-only write classes and quorum acks for multi-leader SQLite. That contract is archived by this rewrite. Implementation remnants that still mention multi-leader envelopes are compatibility shims and must not reintroduce HTTP task replication.
Reference in New Issue View Git Blame Copy Permalink