fusion/docs/design/fn-4819-distributed-multi-node-coordination-gap.md

# FN-4819 — Distributed Multi-Node Task Coordination Gap Specification

## 1. Baseline: Single-Node SQLite WAL Correctness Model

Fusion's current correctness baseline is single-node.
Within one project runtime, structured task metadata is stored in one per-project SQLite database at `.fusion/fusion.db`.
That database runs in WAL mode.
Large blobs stay on disk under `.fusion/tasks/{ID}/`.

From `AGENTS.md` (Storage Model):

> Fusion uses a hybrid storage architecture: structured metadata lives in SQLite (`.fusion/fusion.db`) while large blob files (PROMPT.md, attachments) remain on the filesystem under `.fusion/tasks/{ID}/`. The database runs in WAL mode for concurrent access.

From `AGENTS.md` (Checkout Leasing):

> Checkout conflicts return **409 Conflict** when another agent already holds the lease.
> Clients **must not retry 409 automatically** — this is ownership contention, not a transient failure.

From `docs/storage.md`:

- Project authoritative task row storage is `.fusion/fusion.db` (`docs/storage.md:70`).
- SQLite connections enforce `journal_mode = WAL` and `busy_timeout` (`docs/storage.md:92-101`).

Within one process, ownership is enforced by `AgentStore.checkoutTask(...)`.
The in-process path checks `task.checkedOutBy`, throws `CheckoutConflictError` if held by another agent, and writes lease fields on success (`packages/core/src/agent-store.ts:1378-1426`).

Durable lease-related fields already present on task rows are:

- `checkedOutBy`
- `checkoutNodeId`
- `checkoutRunId`
- `checkoutLeaseEpoch`
- `checkoutLeaseRenewedAt`

These are defined on task records in `packages/core/src/store.ts:155-160` and persisted through the task row write path (`packages/core/src/store.ts:1571-1576`, `1599-1624`).

### Gap statement

Under multi-node operation, each node can maintain a locally consistent read/write path for the same logical task row, but there is no documented atomic cross-node compare-and-set ownership contract.
Two nodes can each evaluate the local single-row precondition as true before either write is globally fenced, causing double-checkout risk.
SQLite WAL guarantees concurrency semantics for a database file, not distributed consensus across multiple node runtimes.
Therefore, the single-node WAL safety model does not, by itself, define cross-node lease correctness.

## 2. Distributed Checkout Mutex (Independent, Testable)

This section is intentionally independent from §3.
It is testable without invoking unavailable-node handoff logic.

### 2.1 Contract

A lease claim attempt by claimant tuple `(nodeId, agentId, runId)` against task `(projectId, taskId)` MUST:

1. Atomically read ownership precondition over:
   - `checkedOutBy`
   - `checkoutNodeId`
   - `checkoutLeaseEpoch`
2. Either:
   - Acquire ownership and bump epoch (`checkoutLeaseEpoch = priorEpoch + 1`) for owner changes, or
   - Return `CheckoutConflictError` populated with current owner details.
3. Renewal by the existing owner MUST update renewal timestamp but MUST NOT bump epoch.

### 2.2 Authority placement decision

**Decision:** Use an authoritative central claim row in `~/.fusion/fusion-central.db`, keyed by `(projectId, taskId)`.

Why this option:

- Least new infrastructure relative to existing architecture.
- Reuses existing central registry database already used for cross-project/node coordination (`docs/multi-project.md:19`, `packages/core/src/central-core.ts:7`, `2626`).
- Avoids introducing external consensus services.
- Keeps per-project `.fusion/fusion.db` semantics intact for local task state while adding one cross-node authority point.

Rejected options (for FN-4819 scope):

- Pure per-project DB mutex only:
  - insufficient when ownership arbitration spans nodes and runtimes.
- Full distributed consensus layer:
  - larger infra jump than required for the defined gap.

### 2.3 Data model (minimum)

Authoritative central table (conceptual): `taskClaims`

Primary key:

- `(projectId, taskId)`

Required columns:

- `projectId TEXT NOT NULL`
- `taskId TEXT NOT NULL`
- `ownerNodeId TEXT NOT NULL`
- `ownerAgentId TEXT NOT NULL`
- `ownerRunId TEXT NULL`
- `leaseEpoch INTEGER NOT NULL`
- `leaseRenewedAt TEXT NOT NULL`
- `createdAt TEXT NOT NULL`
- `updatedAt TEXT NOT NULL`

Optional but recommended columns:

- `ownerSessionId TEXT NULL`
- `metadataJson TEXT NULL`

Task row remains source for local executor checks and backward compatibility:

- `checkedOutBy`
- `checkoutNodeId`
- `checkoutRunId`
- `checkoutLeaseEpoch`
- `checkoutLeaseRenewedAt`

### 2.4 Acquisition semantics

Acquisition is one atomic write transaction in central DB.
Use single-writer semantics with `INSERT ... ON CONFLICT(projectId, taskId) DO UPDATE ... WHERE <precondition>`.

Precondition for claimant change:

- Existing claim is absent, OR
- Existing claim matches same owner tuple for renewal path, OR
- Existing claim explicitly released.

Owner-change claim success path:

- `leaseEpoch = existingEpoch + 1` (or `1` on first claim)
- Set owner tuple to claimant.
- Set `leaseRenewedAt = now`.

Renewal success path (same owner tuple):

- Keep `leaseEpoch` unchanged.
- Update `leaseRenewedAt = now`.

Loser path:

- No mutation.
- Return `CheckoutConflictError` with current holder.
- Caller MUST NOT auto-retry immediately (matches existing 409 semantics from `AGENTS.md` Checkout Leasing).

### 2.5 Per-project row synchronization

On successful central claim transaction, update per-project task row lease fields in the same logical operation boundary:

- Write `checkedOutBy = ownerAgentId`
- Write `checkoutNodeId = ownerNodeId`
- Write `checkoutRunId = ownerRunId`
- Write `checkoutLeaseEpoch = central.leaseEpoch`
- Write `checkoutLeaseRenewedAt = central.leaseRenewedAt`

On release:

- Clear owner fields on both central row and task row.
- Preserve monotonic epoch progression in central row history (or tombstone).

If a split-write occurs (central success, task-row write fail), reconciliation policy is:

- Treat central row as authority.
- Repair task row via reconciliation worker before next dispatch.

### 2.6 Heartbeat semantics

Heartbeat renewal from active owner:

- Re-assert same owner tuple.
- Update `leaseRenewedAt`.
- Do not bump epoch.

Heartbeat from stale owner (epoch mismatch or owner mismatch):

- Must fail as conflict.
- Worker aborts run before any write/merge path.

### 2.7 API-level behavior contract

`AgentStore.checkoutTask(...)` remains the integration point.
Behavior evolves from single-row local check to:

1. central authoritative claim mutation
2. task-row synchronization write
3. return claimed task

Conflict mapping remains `CheckoutConflictError` / 409 for external callers.
No automatic retry loops in caller.

### 2.8 Testability surface

Required unit-level contract test:

- Given pre-seeded central claim row with owner A
- When owner B attempts claim
- Then claim fails with `CheckoutConflictError`
- And error contains `currentHolder = owner A`

Required race integration test:

- Simulate two nodes racing same `(projectId, taskId)` claim.
- Use barrier/latch to release both attempts simultaneously.
- Assert exactly one winner.
- Assert loser receives `CheckoutConflictError`.
- Assert post-state owner fields match winner tuple `(nodeId, agentId)`.
- Assert epoch bumped exactly once from initial value.

## 3. Unavailable-Node Handoff Policy (Independent, Testable)

This section is intentionally independent from §2.
It is testable without invoking mutex acquisition by pre-seeding task ownership rows and node health transitions.

### 3.1 Scope and intent

Define deterministic handling for node availability transitions affecting existing lease owners.
No live-process migration is introduced.
Scheduler failover is out of scope.

### 3.2 Transition A: pre-dispatch unavailable

Keep current dispatch semantics from `applyUnavailableNodePolicy(...)`:

- `fallback-local` allows local fallback (`packages/engine/src/node-routing-policy.ts:11-37`).
- `block` refuses dispatch with reason (`packages/engine/src/node-routing-policy.ts:39-42`).

Current scheduler call site remains dispatch-time policy gate (`packages/engine/src/scheduler.ts:1005`).

### 3.3 Transition B: owner unreachable mid-task

Behavior contract:

1. In-flight task remains owned until lease expiry window is reached.
2. After lease expiry, if `NodeHealthMonitor` reports owner status `offline` or `error` (`packages/engine/src/node-health-monitor.ts:132-148`), owner is considered unreachable.
3. Recovery action:
   - release claim
   - move task to `todo`
   - preserve step progress (`preserveProgress = true` when progress exists)
   - do not preserve old worktree on picking node (`preserveWorktree = false`)
4. Emit run-audit event: `task:auto-recover-node-unreachable`.

`MeshLeaseManager.isLeaseRecoverable(...)` and `recoverAbandonedLease(...)` provide current recovery seam (`packages/engine/src/mesh-lease-manager.ts:23-107`), extended by policy checks above.

No executor/session process handoff occurs.
Restarted work is fresh execution from preserved progress metadata.

### 3.4 Transition C: owner returns after handoff

A previously owning node may come back online after another node advanced epoch.
Returning node behavior:

- On next heartbeat, validate lease ownership.
- If central claim owner/epoch no longer matches local run, abort cleanly.
- Do not continue writes, merges, or `fn_task_done`.

Existing detection seam:

- `HeartbeatMonitor.executeHeartbeat()` already validates `task.checkedOutBy` and exits with checkout-conflict semantics (`AGENTS.md` Checkout Leasing section).

Minimal extension needed:

- include central epoch/owner validation (or synchronized local epoch validation) in the same heartbeat preflight so stale owner runs fence themselves before side effects.

### 3.5 Policy enum

Define policy enum for unavailable-owner reassignment behavior:

- `block`
- `reassign-to-local`
- `reassign-any-healthy`

Default recommendation:

- Keep default behavior equivalent to today's fallback/local-block pattern unless explicitly configured.
- For compatibility, map existing `fallback-local` to `reassign-to-local` in transition layer.

### 3.6 Testability surface

Unit policy tests:

- For each enum value, assert expected decision for `{owner online, owner offline, owner error, owner unknown}`.

Scheduler integration tests:

1. Online-owner protection:
   - pre-seed task owned by online node A
   - scheduler on node B must not dispatch it
2. `reassign-to-local` behavior:
   - pre-seed task owned by offline node A
   - after lease expiry and policy evaluation, row is released
   - task becomes dispatch-eligible for local node B

Independence harness note:

- These tests can bypass §2 by directly seeding ownership row + lease timestamps and toggling health monitor outputs.

## 4. Single-Project → Multi-Project / Multi-Node Isolation & Ownership Transition Path

### 4.1 Supported state machine

Supported operational states:

- `single-project/in-process`
- `multi-project/in-process`
- `multi-project/child-process`
- `multi-project/in-process + remote runtime placement`

Transition trigger is runtime reconfiguration requiring restart-bound re-evaluation.

### 4.2 Preconditions

Transition allowed when:

- Node is idle (`0` tasks in `in-progress`), OR
- Caller passes explicit `force` flag.

If `force=true`:

- In-flight tasks are paused through engine-initiated rebound path.
- Do not mark tasks as `userPaused`.
- Resume under new runtime after restart and ownership re-validation.

### 4.3 Runtime selection behavior on transition

On startup after transition:

- Re-evaluate `shouldUseHybridExecutor(...)` gate (`packages/engine/src/hybrid-executor-gate.ts:20`).
- Re-evaluate runtime type selection in `ProjectManager.addProject(...)` based on `isolationMode` (`packages/engine/src/project-manager.ts:168-170`).
- Continue to use `CentralCore` registration as source for project placement and mode (`packages/core/src/central-core.ts:252`, `449`).

Per-project SQLite DBs are not migrated or rewritten by this transition.
They remain at `.fusion/fusion.db` per project.

### 4.4 Central claim bootstrap

On first startup in multi-node-capable mode:

- Apply idempotent migration to create central `taskClaims` table (from §2).
- Migration must be safe to rerun.
- No data loss on repeated startups.

### 4.5 Rollback rules

Rollback to single-project mode is only supported when node is sole registered node in `CentralCore`.
If multiple nodes are registered, reject rollback with actionable error.

Rationale:

- prevents silently orphaning cross-node claim ownership semantics.

### 4.6 Durable trail requirements

Every transition writes durable activity/audit records:

- Activity entry: `project:isolation-transition`
- Event entry: `project:runtime-restarted` with payload:
  - `projectId`
  - `isolationMode`
  - `reason`

### 4.7 Explicit non-hot-swap scope

Hot-swap isolation transitions without restart are out of scope.
All supported transitions are restart-bound.

## 5. Non-Goals (Explicit Exclusions)

The following are explicitly out of scope for FN-4819 and MUST NOT be pulled into this brief's implementation follow-up:

1. **Scheduler failover**
   - Exclusion: a peer taking over the live scheduler tick loop for another node's project runtime.
   - Rationale: requires runtime leadership election and scheduler state transfer not needed to close lease-ownership gap.
2. **Live-process state migration**
   - Exclusion: transferring in-memory executor/session process state mid-task across nodes.
   - Rationale: requires session serialization/resume semantics beyond lease fencing and handoff policy.
3. **Cross-node consensus for engine settings mutations**
   - Exclusion: globally coordinated settings write quorum.
   - Rationale: orthogonal control-plane consistency problem.
4. **Multi-master concurrent writes to the same per-project `.fusion/fusion.db`**
   - Exclusion: active simultaneous writers from multiple nodes to one project DB file.
   - Rationale: storage topology constraint outside this task's lease contract.
5. **Automatic node promotion**
   - Exclusion: autonomous promotion/demotion workflows for node roles.
   - Rationale: operational governance feature, not checkout correctness.

No follow-up task is to be filed under this brief for these exclusions.

## 6. Acceptance Mapping

- **Acceptance: "four concerns called out as separate sections"**
  - Satisfied by:
    - §1 Baseline correctness model
    - §2 Distributed checkout mutex
    - §3 Unavailable-node handoff policy
    - §4 Isolation/ownership transition path
- **Acceptance: "distributed checkout mutex and unavailable-node handoff policy explicitly independent and testable"**
  - Satisfied by:
    - §2 opening independence statement + §2.8 testability surface
    - §3 opening independence statement + §3.6 testability surface
- **Acceptance: "WAL-based single-node baseline referenced in the protocol/docs section"**
  - Satisfied by:
    - §1 references to `AGENTS.md` Storage Model and `docs/storage.md` WAL details
- **Acceptance: "non-goal section explicitly excludes scheduler failover/live-process migration"**
  - Satisfied by:
    - §5 items 1 and 2

## 7. Code & Doc References

All references below were path-verified during FN-4819 preflight.

- `AGENTS.md:81-85` — Storage Model canonical wording
- `AGENTS.md:472-486` — Checkout Leasing conflict semantics and heartbeat enforcement notes
- `docs/storage.md:70` — per-project DB row source (`.fusion/fusion.db`)
- `docs/storage.md:92-101` — WAL/busy-timeout and concurrent write notes
- `docs/multi-project.md:19` — central DB path `~/.fusion/fusion-central.db`
- `docs/multi-project.md:108-113` — isolation mode definitions
- `docs/multi-project.md:158-166` — ProjectManager runtime-mode mapping narrative
- `docs/shared-mesh-protocol.md:16-19` — existing protocol non-goals including scheduler failover/live-process migration
- `docs/architecture.md` — engine composition and run-audit reference baseline
- `packages/core/src/store.ts:155-160` — task lease field definitions
- `packages/core/src/store.ts:1571-1576` — persisted lease field write mapping
- `packages/core/src/store.ts:1599-1624` — insert/upsert columns include lease fields
- `packages/core/src/agent-store.ts:1378-1426` — `checkoutTask` current single-process lease behavior
- `packages/core/src/agent-store.ts:47` — `CheckoutConflictError` usage import site
- `packages/core/src/central-core.ts:7` — central DB location note
- `packages/core/src/central-core.ts:252-255` — `registerProject` with `isolationMode`
- `packages/core/src/central-core.ts:449` — `updateProject` mutation path
- `packages/core/src/central-core.ts:2626` — central DB path getter
- `packages/core/src/types.ts:3369` — `IsolationMode = "in-process" | "child-process"`
- `packages/engine/src/project-manager.ts:168-170` — `isolationMode` runtime branching
- `packages/engine/src/node-routing-policy.ts:4-42` — `PolicyDecision` and `applyUnavailableNodePolicy`
- `packages/engine/src/mesh-lease-manager.ts:23-107` — lease recoverability and recovery path
- `packages/engine/src/node-health-monitor.ts:132-148` — offline/error transition handling
- `packages/engine/src/scheduler.ts:1005` — dispatch-time unavailable-node policy call site
- `packages/engine/src/hybrid-executor-gate.ts:20` — `shouldUseHybridExecutor` decision seam

## 8. Related Work (Optional Footnote)

No sibling FN design brief is required to interpret this document.
If adjacent work (for example another FN-48xx mesh coordination brief) exists, treat it as supplemental context only; FN-4819 remains self-contained and normative for this specific coordination-gap specification.