fusion/docs/findings/fn-4820-multi-node-coordination-validation.md

# FN-4820 — Multi-Node Coordination Validation vs FN-4819

## 1. Scope & Method

### Scope

This audit validates the **current implementation on `main`** against FN-4819 across exactly three areas:

1. Scheduler ownership / checkout semantics across nodes
2. Wake / assignment propagation across nodes
3. Conflict-handling telemetry

### Explicit non-goals (inherited from FN-4819 §5)

This audit does **not** propose implementation work for:

- Scheduler failover
- Live-process state migration
- Cross-node settings consensus
- Multi-master writes to one per-project `.fusion/fusion.db`
- Automatic node promotion

### Method

- Evidence source: repository files on commit `2595b4111fefe510d2cbc556b83d8b3c50067655`
- Method: direct `file:line` code citation from current code paths
- Design baseline: `docs/design/fn-4819-distributed-multi-node-coordination-gap.md` (§1–§5)
- Classification policy: conservative (`partial` unless requirements are fully met)

## 2. FN-4819 Concern Coverage Matrix

| FN-4819 concern | Status | Primary evidence (file:line) | Notes |
|---|---|---|---|
| §1 Single-node WAL baseline and cross-node gap | **works today** (baseline only) | `packages/core/src/store.ts:155-160`; `packages/core/src/store.ts:3822-3831`; `packages/core/src/agent-store.ts:1371-1460` | Single-project SQLite row-level lease fields + CAS-style update exist. This satisfies the baseline model but does not close distributed arbitration by itself. |
| §2 Distributed checkout mutex (central claim authority) | **missing** | `packages/core/src/central-db.ts:36-286` (no `taskClaims` table); repo search: `taskClaims` no hits; `packages/core/src/agent-store.ts:1371-1460` | Checkout logic is task-row-local. No authoritative central claim row keyed by `(projectId, taskId)` as required by FN-4819 §2.2/§2.3. |
| §3 Unavailable-node handoff policy | **partial** | `packages/engine/src/node-routing-policy.ts:4-75`; `packages/engine/src/scheduler.ts:1004-1044`; `packages/engine/src/mesh-lease-manager.ts:84-140`; `packages/engine/src/node-health-monitor.ts:103-148` | Policy seams and stale lease recovery exist, but durable run-audit event `task:auto-recover-node-unreachable` from FN-4819 §3.3 is not implemented. |
| §4 Isolation/ownership transition path | **partial** | `packages/core/src/central-core.ts:521-547`; `packages/engine/src/hybrid-executor-gate.ts:20-36`; `packages/engine/src/project-manager.ts:168-196`; `packages/core/src/central-db.ts:36-286` | Runtime mode selection + isolation transition activity logging exists, but no central `taskClaims` bootstrap and no visible rollback guard tied to sole-node condition from FN-4819 §4.4/§4.5. |

## 3. Findings by Audit Area

### 3.1 Ownership / checkout semantics across nodes

#### Finding A — Checkout CAS is local-row atomic, not central-distributed

- **Severity:** P0
- **Summary:** Current checkout conflict protection is atomic against one task row, but no cross-node central authority claim exists.
- **Current behavior:**
  - `AgentStore.checkoutTask()` uses holder checks and `tryClaimCheckout` when available (`packages/core/src/agent-store.ts:1371-1460`).
  - `TaskStore.tryClaimCheckout()` performs compare-and-set over task fields (`checkedOutBy`, `checkoutNodeId`, `checkoutLeaseEpoch`) (`packages/core/src/store.ts:3822-3831`).
  - Lease fields are persisted on task rows (`packages/core/src/store.ts:155-160`).
- **Expected per FN-4819:** FN-4819 §2.2/§2.3 requires an authoritative central claim row (`~/.fusion/fusion-central.db`) keyed by `(projectId, taskId)` for atomic lease arbitration.
- **Delta:** No `taskClaims` authority table or central claim mutation path found (`packages/core/src/central-db.ts:36-286`; repo search no `taskClaims` symbol).
- **Recommended follow-up:** Add central claim-table mutex path and wire checkout through it before task-row sync.

#### Finding B — Lease recovery writes only task row and task log entries

- **Severity:** P1
- **Summary:** Abandoned lease recovery can clear lease fields and requeue tasks, but reconciliation remains task-row-local and not central-claim-driven.
- **Current behavior:**
  - `recoverAbandonedLease()` checks staleness and node health, then clears checkout fields and moves task to `todo` (`packages/engine/src/mesh-lease-manager.ts:84-140`).
  - Scheduler invokes lease recovery during todo dispatch (`packages/engine/src/scheduler.ts:829-836`).
- **Expected per FN-4819:** FN-4819 §2.5 and §3.3 require central claim release/sync authority and bounded handoff semantics tied to that authority.
- **Delta:** Recovery does not interact with any central lease claim primitive; no central/local reconciliation worker path evidenced.
- **Recommended follow-up:** Introduce central-claim-aware recovery + reconciliation semantics for split-write scenarios.

#### Finding C — Scheduler dispatch guards reduce local double-starts but do not prove multi-node exclusivity

- **Severity:** P1
- **Summary:** Scheduler re-read CAS and handoff checks help local correctness, but no evidence of distributed mutual exclusion beyond local data path.
- **Current behavior:**
  - Scheduler re-reads the task to verify still `todo` before dispatch (`packages/engine/src/scheduler.ts:958-965`).
  - Owner health and handoff policy are applied before dispatch (`packages/engine/src/scheduler.ts:1004-1044`; `packages/engine/src/node-routing-policy.ts:4-75`).
- **Expected per FN-4819:** FN-4819 §2 requires centralized ownership arbitration independent of scheduler timing.
- **Delta:** Dispatch policy gates are additive, but no global claim winner/loser contract is present in cited code.
- **Recommended follow-up:** Add race tests and scheduler integration assertions against central claim winner semantics.

### 3.2 Wake / assignment propagation across nodes

#### Finding D — Assignment wake is event-push local; cross-node wake guarantee is implicit/indirect

- **Severity:** P1
- **Summary:** Local assignment wake hook is explicit; cross-node propagation path appears indirect via remote runtime event stream, not a dedicated assignment-delivery contract.
- **Current behavior:**
  - Local assignment trigger subscribes to `agent:assigned` and invokes callback source `assignment` (`packages/engine/src/agent-heartbeat.ts:3663-3740`).
  - In-process runtime forwards local `task:*` events (`packages/engine/src/runtimes/in-process-runtime.ts:1186-1194`).
  - Remote runtime consumes `/api/events/stream` and forwards remote `task:*` events (`packages/engine/src/runtimes/remote-node-runtime.ts:153-213,241-254`; `packages/engine/src/runtimes/remote-node-client.ts:98-120`).
- **Expected per FN-4819:** FN-4819 §3 + audit scope expects explicit understanding of owner/peer wake propagation and missed-wake behavior.
- **Delta:** No dedicated “task assigned on node A must wake owner on node B” contract or acceptance test seam was found in these modules.
- **Recommended follow-up:** Define and test explicit cross-node assignment wake semantics (push/pull fallback, bounded latency, missed-wake recovery).

#### Finding E — Peer exchange loop provides periodic mesh sync, not assignment wake routing

- **Severity:** P2
- **Summary:** Peer exchange cadence is periodic gossip and settings/shared-state sync; it is not an assignment wake transport.
- **Current behavior:**
  - Peer exchange default sync interval is 120s and syncs online peers sequentially (`packages/engine/src/peer-exchange-service.ts:116-206`).
- **Expected per FN-4819:** FN-4819 requires clear operational behavior when wake signals are missed or delayed.
- **Delta:** No assignment-specific fallback in peer exchange indicates uncertain wake SLO under event-stream outages.
- **Recommended follow-up:** Document wake fallback behavior and add measurable latency/error telemetry for cross-node assignment delivery.

### 3.3 Conflict-handling telemetry

#### Finding F — Required `task:auto-recover-node-unreachable` event is absent

- **Severity:** P1
- **Summary:** FN-4819 names a required run-audit event for node-unreachable recovery, but the event type is not present.
- **Current behavior:**
  - Run-audit database event union contains several `task:auto-recover-*` events (`packages/engine/src/run-audit.ts:126-136`).
  - `task:auto-recover-node-unreachable` is not listed there and not found in engine source search.
- **Expected per FN-4819:** FN-4819 §3.3 requires durable run-audit row `task:auto-recover-node-unreachable` on unreachable-owner recovery.
- **Delta:** Recovery path currently relies on task log entries / logger messages instead of typed run-audit event.
- **Recommended follow-up:** Add typed run-audit event emission for unreachable-owner recovery path and test coverage.

#### Finding G — Handoff/recovery decisions can complete without structured run-audit classification

- **Severity:** P2
- **Summary:** Handoff decisions are persisted mainly as `taskStore.logEntry` text.
- **Current behavior:**
  - Scheduler logs handoff decisions via `logEntry` (`packages/engine/src/scheduler.ts:1018-1025`).
  - Mesh lease manager logs and task log entries on recovery (`packages/engine/src/mesh-lease-manager.ts:108-133`).
- **Expected per FN-4819:** Durable conflict telemetry should support querying by event class.
- **Delta:** Text log entries are less queryable than typed run-audit records.
- **Recommended follow-up:** Standardize node-handoff/recovery telemetry into run-audit event taxonomy.

## 4. Prioritized Follow-Up Backlog

1. **[FN-4822] Add authoritative central task-claim mutex for cross-node checkout arbitration**  
   **Severity:** P0  
   Implement FN-4819 §2.2/§2.3 by introducing a central `(projectId, taskId)` claim authority and routing checkout acquisition through it before task-row sync.

   **Acceptance criteria:**
   - Introduce central claim persistence keyed by `(projectId, taskId)` with owner tuple and epoch semantics from FN-4819 §2.3.
   - `checkoutTask` winner/loser behavior follows FN-4819 §2.4 (exactly one winner; loser gets `CheckoutConflictError`).
   - Renewal keeps epoch stable; owner change bumps epoch.
   - Add race integration test asserting one winner across concurrent claim attempts (FN-4819 §2.8).

2. **[FN-4823] Make lease recovery central-claim-aware with reconciliation guarantees**  
   **Severity:** P1  
   Align abandoned-lease recovery and release semantics with FN-4819 §2.5 and §3.3 so central authority and task-row views cannot diverge silently.

   **Acceptance criteria:**
   - Recovery/release path updates or reconciles central claim + task row according to FN-4819 §2.5.
   - Split-write failure mode has deterministic repair path before next dispatch.
   - Scheduler + mesh lease recovery tests cover offline/error owner-node handoff behavior (FN-4819 §3.6).

3. **[FN-4824] Specify and test cross-node assignment wake propagation contract**  
   **Severity:** P1  
   Define explicit push/pull/missed-wake behavior for assignment propagation between nodes, with bounded-latency and recovery expectations.

   **Acceptance criteria:**
   - Document assignment wake contract across node A→node B transitions, including missed-stream fallback.
   - Add integration tests proving assigned owner receives actionable wake under normal and degraded transport conditions.
   - Define and expose measurable worst-case wake latency behavior consistent with FN-4819 §3 operational intent.

4. **[FN-4825] Emit typed run-audit telemetry for node-unreachable auto-recovery**  
   **Severity:** P1  
   Add structured telemetry for unreachable-owner recovery path so conflict handling is queryable and auditable.

   **Acceptance criteria:**
   - Add `task:auto-recover-node-unreachable` run-audit event support and emit on applicable recovery paths (FN-4819 §3.3).
   - Add tests validating event emission metadata for owner node, policy decision, and resulting task transition.
   - Ensure no recovery completion path is silent (must emit either success/failure classified event).

5. **[FN-4826] Normalize node handoff telemetry taxonomy beyond free-text task logs**  
   **Severity:** P2  
   Reduce ambiguity by converting key scheduler/lease-manager handoff text logs into typed event classes.

   **Acceptance criteria:**
   - Define typed telemetry names for handoff parked/reassign/recovered outcomes.
   - Keep backward-compatible human-readable task log entries where useful.
   - Add tests ensuring each decision path emits one structured telemetry event.

## 5. Non-Goals Reaffirmation

The following FN-4819 non-goals were respected and **no follow-up task below targets them**:

- Scheduler failover
- Live-process state migration
- Cross-node settings consensus
- Multi-master concurrent writes to one per-project `.fusion/fusion.db`
- Automatic node promotion

## 6. References

### Primary design reference

- `docs/design/fn-4819-distributed-multi-node-coordination-gap.md`

### Ownership / checkout evidence

- `packages/core/src/store.ts:155-160` — lease-related task fields exist on row model
- `packages/core/src/store.ts:3822-3831` — `tryClaimCheckout` CAS precondition update
- `packages/core/src/agent-store.ts:1371-1460` — checkout acquisition path and conflict behavior
- `packages/core/src/agent-store.ts:1417-1420` — renewal/epoch behavior in claim payload
- `packages/core/src/store.ts:3900-3917` — task selection filters against foreign checkout holder
- `packages/core/src/central-db.ts:36-286` — central schema inventory (no `taskClaims` table)
- `packages/core/src/central-core.ts:7` — central DB location contract (`~/.fusion/fusion-central.db`)
- `packages/engine/src/scheduler.ts:829-836` — scheduler invokes lease recovery for checked-out todo
- `packages/engine/src/scheduler.ts:958-965` — re-read guard for todo dispatch
- `packages/engine/src/scheduler.ts:1004-1044` — handoff + unavailable-node policy gate during dispatch
- `packages/engine/src/node-routing-policy.ts:4-75` — handoff and unavailable-node decision logic
- `packages/engine/src/mesh-lease-manager.ts:84-140` — abandoned-lease recovery path
- `packages/engine/src/mesh-lease-manager.ts:93-115` — owner offline/error handoff policy branch
- `packages/engine/src/mesh-lease-manager.ts:117-127` — lease clear + epoch increment + row mutation
- `packages/engine/src/mesh-lease-manager.ts:129-140` — task log + todo rebound behavior

### Wake / assignment propagation evidence

- `packages/engine/src/agent-heartbeat.ts:3663-3740` — local `agent:assigned` wake trigger
- `packages/engine/src/agent-heartbeat.ts:3731-3734` — assignment callback payload fields
- `packages/engine/src/runtimes/in-process-runtime.ts:1186-1194` — local task event forwarding
- `packages/engine/src/runtimes/remote-node-runtime.ts:153-213` — remote stream loop and reconnect behavior
- `packages/engine/src/runtimes/remote-node-runtime.ts:241-254` — forwarding remote `task:*` events
- `packages/engine/src/runtimes/remote-node-client.ts:98-120` — stream endpoint parsing (SSE/JSON fallback)
- `packages/engine/src/runtimes/remote-node-client.ts:114-120` — long-polling JSON fallback handling
- `packages/engine/src/runtimes/remote-node-client.ts:406-416` — retry behavior for failed requests
- `packages/engine/src/peer-exchange-service.ts:116-206` — periodic peer sync loop (default interval behavior)
- `packages/engine/src/peer-exchange-service.ts:262-380` — per-peer sync request flow

### Conflict telemetry evidence

- `packages/engine/src/run-audit.ts:97-111` — existing `branch:*` structured audit taxonomy
- `packages/engine/src/run-audit.ts:126-136` — existing `task:auto-recover-*` taxonomy
- `packages/engine/src/run-audit.ts` search result — no `task:auto-recover-node-unreachable` entry
- `packages/engine/src/scheduler.ts:1018-1025` — handoff decisions persisted via `taskStore.logEntry`
- `packages/engine/src/mesh-lease-manager.ts:108-133` — recovery outcomes persisted via logs/task entries

### Isolation transition evidence

- `packages/core/src/central-core.ts:521-547` — `transitionProjectIsolation` + activity log write
- `packages/core/src/central-core.ts:531-535` — noop guard for unchanged isolation mode
- `packages/engine/src/hybrid-executor-gate.ts:20-36` — runtime-mode gate based on node/project topology
- `packages/engine/src/project-manager.ts:168-196` — runtime selection by isolation mode + assigned node type
- `packages/core/src/central-db.ts:36-286` — absence of migration/bootstrap for central `taskClaims`

### Additional context references consulted

- `AGENTS.md` (Storage Model, Multi-Project Support, Checkout Leasing, Architecture)
- `docs/storage.md`
- `docs/multi-project.md`
- `docs/shared-mesh-protocol.md`
- `docs/architecture.md`
- `packages/engine/src/project-engine-manager.ts:398-409` — runtime config construction includes isolation mode input
- `packages/engine/src/project-engine-manager.ts:360-389` — runtime creation/start lifecycle context
- `packages/engine/src/agent-heartbeat.ts:2016-2028` — checkout validation preflight in heartbeat execution
- `packages/engine/src/peer-exchange-service.ts:175-206` — triggerSync/single-flight synchronization behavior
- `packages/core/src/central-core.ts:252-302` — project registration includes isolation mode defaulting
- `packages/core/src/central-core.ts:449-485` — project update path persists isolation mode changes
- `packages/engine/src/runtimes/remote-node-client.ts:179-183` — retryability classification for remote failures
- `packages/engine/src/runtimes/remote-node-runtime.ts:181-203` — bounded reconnect attempt behavior