feat(FN-4819): merge fusion/fn-4819

2026-05-16 18:06:12 -07:00
parent a1b1f9aa05
commit 591e0b9e90
1 changed files with 435 additions and 0 deletions
--- a/docs/design/fn-4819-distributed-multi-node-coordination-gap.md
+++ b/docs/design/fn-4819-distributed-multi-node-coordination-gap.md
@@ -0,0 +1,435 @@
+# FN-4819 — Distributed Multi-Node Task Coordination Gap Specification
+
+## 1. Baseline: Single-Node SQLite WAL Correctness Model
+
+Fusion's current correctness baseline is single-node.
+Within one project runtime, structured task metadata is stored in one per-project SQLite database at `.fusion/fusion.db`.
+That database runs in WAL mode.
+Large blobs stay on disk under `.fusion/tasks/{ID}/`.
+
+From `AGENTS.md` (Storage Model):
+
+> Fusion uses a hybrid storage architecture: structured metadata lives in SQLite (`.fusion/fusion.db`) while large blob files (PROMPT.md, attachments) remain on the filesystem under `.fusion/tasks/{ID}/`. The database runs in WAL mode for concurrent access.
+
+From `AGENTS.md` (Checkout Leasing):
+
+> Checkout conflicts return **409 Conflict** when another agent already holds the lease.
+> Clients **must not retry 409 automatically** — this is ownership contention, not a transient failure.
+
+From `docs/storage.md`:
+
+- Project authoritative task row storage is `.fusion/fusion.db` (`docs/storage.md:70`).
+- SQLite connections enforce `journal_mode = WAL` and `busy_timeout` (`docs/storage.md:92-101`).
+
+Within one process, ownership is enforced by `AgentStore.checkoutTask(...)`.
+The in-process path checks `task.checkedOutBy`, throws `CheckoutConflictError` if held by another agent, and writes lease fields on success (`packages/core/src/agent-store.ts:1378-1426`).
+
+Durable lease-related fields already present on task rows are:
+
+- `checkedOutBy`
+- `checkoutNodeId`
+- `checkoutRunId`
+- `checkoutLeaseEpoch`
+- `checkoutLeaseRenewedAt`
+
+These are defined on task records in `packages/core/src/store.ts:155-160` and persisted through the task row write path (`packages/core/src/store.ts:1571-1576`, `1599-1624`).
+
+### Gap statement
+
+Under multi-node operation, each node can maintain a locally consistent read/write path for the same logical task row, but there is no documented atomic cross-node compare-and-set ownership contract.
+Two nodes can each evaluate the local single-row precondition as true before either write is globally fenced, causing double-checkout risk.
+SQLite WAL guarantees concurrency semantics for a database file, not distributed consensus across multiple node runtimes.
+Therefore, the single-node WAL safety model does not, by itself, define cross-node lease correctness.
+
+## 2. Distributed Checkout Mutex (Independent, Testable)
+
+This section is intentionally independent from §3.
+It is testable without invoking unavailable-node handoff logic.
+
+### 2.1 Contract
+
+A lease claim attempt by claimant tuple `(nodeId, agentId, runId)` against task `(projectId, taskId)` MUST:
+
+1. Atomically read ownership precondition over:
+   - `checkedOutBy`
+   - `checkoutNodeId`
+   - `checkoutLeaseEpoch`
+2. Either:
+   - Acquire ownership and bump epoch (`checkoutLeaseEpoch = priorEpoch + 1`) for owner changes, or
+   - Return `CheckoutConflictError` populated with current owner details.
+3. Renewal by the existing owner MUST update renewal timestamp but MUST NOT bump epoch.
+
+### 2.2 Authority placement decision
+
+**Decision:** Use an authoritative central claim row in `~/.fusion/fusion-central.db`, keyed by `(projectId, taskId)`.
+
+Why this option:
+
+- Least new infrastructure relative to existing architecture.
+- Reuses existing central registry database already used for cross-project/node coordination (`docs/multi-project.md:19`, `packages/core/src/central-core.ts:7`, `2626`).
+- Avoids introducing external consensus services.
+- Keeps per-project `.fusion/fusion.db` semantics intact for local task state while adding one cross-node authority point.
+
+Rejected options (for FN-4819 scope):
+
+- Pure per-project DB mutex only:
+  - insufficient when ownership arbitration spans nodes and runtimes.
+- Full distributed consensus layer:
+  - larger infra jump than required for the defined gap.
+
+### 2.3 Data model (minimum)
+
+Authoritative central table (conceptual): `taskClaims`
+
+Primary key:
+
+- `(projectId, taskId)`
+
+Required columns:
+
+- `projectId TEXT NOT NULL`
+- `taskId TEXT NOT NULL`
+- `ownerNodeId TEXT NOT NULL`
+- `ownerAgentId TEXT NOT NULL`
+- `ownerRunId TEXT NULL`
+- `leaseEpoch INTEGER NOT NULL`
+- `leaseRenewedAt TEXT NOT NULL`
+- `createdAt TEXT NOT NULL`
+- `updatedAt TEXT NOT NULL`
+
+Optional but recommended columns:
+
+- `ownerSessionId TEXT NULL`
+- `metadataJson TEXT NULL`
+
+Task row remains source for local executor checks and backward compatibility:
+
+- `checkedOutBy`
+- `checkoutNodeId`
+- `checkoutRunId`
+- `checkoutLeaseEpoch`
+- `checkoutLeaseRenewedAt`
+
+### 2.4 Acquisition semantics
+
+Acquisition is one atomic write transaction in central DB.
+Use single-writer semantics with `INSERT ... ON CONFLICT(projectId, taskId) DO UPDATE ... WHERE <precondition>`.
+
+Precondition for claimant change:
+
+- Existing claim is absent, OR
+- Existing claim matches same owner tuple for renewal path, OR
+- Existing claim explicitly released.
+
+Owner-change claim success path:
+
+- `leaseEpoch = existingEpoch + 1` (or `1` on first claim)
+- Set owner tuple to claimant.
+- Set `leaseRenewedAt = now`.
+
+Renewal success path (same owner tuple):
+
+- Keep `leaseEpoch` unchanged.
+- Update `leaseRenewedAt = now`.
+
+Loser path:
+
+- No mutation.
+- Return `CheckoutConflictError` with current holder.
+- Caller MUST NOT auto-retry immediately (matches existing 409 semantics from `AGENTS.md` Checkout Leasing).
+
+### 2.5 Per-project row synchronization
+
+On successful central claim transaction, update per-project task row lease fields in the same logical operation boundary:
+
+- Write `checkedOutBy = ownerAgentId`
+- Write `checkoutNodeId = ownerNodeId`
+- Write `checkoutRunId = ownerRunId`
+- Write `checkoutLeaseEpoch = central.leaseEpoch`
+- Write `checkoutLeaseRenewedAt = central.leaseRenewedAt`
+
+On release:
+
+- Clear owner fields on both central row and task row.
+- Preserve monotonic epoch progression in central row history (or tombstone).
+
+If a split-write occurs (central success, task-row write fail), reconciliation policy is:
+
+- Treat central row as authority.
+- Repair task row via reconciliation worker before next dispatch.
+
+### 2.6 Heartbeat semantics
+
+Heartbeat renewal from active owner:
+
+- Re-assert same owner tuple.
+- Update `leaseRenewedAt`.
+- Do not bump epoch.
+
+Heartbeat from stale owner (epoch mismatch or owner mismatch):
+
+- Must fail as conflict.
+- Worker aborts run before any write/merge path.
+
+### 2.7 API-level behavior contract
+
+`AgentStore.checkoutTask(...)` remains the integration point.
+Behavior evolves from single-row local check to:
+
+1. central authoritative claim mutation
+2. task-row synchronization write
+3. return claimed task
+
+Conflict mapping remains `CheckoutConflictError` / 409 for external callers.
+No automatic retry loops in caller.
+
+### 2.8 Testability surface
+
+Required unit-level contract test:
+
+- Given pre-seeded central claim row with owner A
+- When owner B attempts claim
+- Then claim fails with `CheckoutConflictError`
+- And error contains `currentHolder = owner A`
+
+Required race integration test:
+
+- Simulate two nodes racing same `(projectId, taskId)` claim.
+- Use barrier/latch to release both attempts simultaneously.
+- Assert exactly one winner.
+- Assert loser receives `CheckoutConflictError`.
+- Assert post-state owner fields match winner tuple `(nodeId, agentId)`.
+- Assert epoch bumped exactly once from initial value.
+
+## 3. Unavailable-Node Handoff Policy (Independent, Testable)
+
+This section is intentionally independent from §2.
+It is testable without invoking mutex acquisition by pre-seeding task ownership rows and node health transitions.
+
+### 3.1 Scope and intent
+
+Define deterministic handling for node availability transitions affecting existing lease owners.
+No live-process migration is introduced.
+Scheduler failover is out of scope.
+
+### 3.2 Transition A: pre-dispatch unavailable
+
+Keep current dispatch semantics from `applyUnavailableNodePolicy(...)`:
+
+- `fallback-local` allows local fallback (`packages/engine/src/node-routing-policy.ts:11-37`).
+- `block` refuses dispatch with reason (`packages/engine/src/node-routing-policy.ts:39-42`).
+
+Current scheduler call site remains dispatch-time policy gate (`packages/engine/src/scheduler.ts:1005`).
+
+### 3.3 Transition B: owner unreachable mid-task
+
+Behavior contract:
+
+1. In-flight task remains owned until lease expiry window is reached.
+2. After lease expiry, if `NodeHealthMonitor` reports owner status `offline` or `error` (`packages/engine/src/node-health-monitor.ts:132-148`), owner is considered unreachable.
+3. Recovery action:
+   - release claim
+   - move task to `todo`
+   - preserve step progress (`preserveProgress = true` when progress exists)
+   - do not preserve old worktree on picking node (`preserveWorktree = false`)
+4. Emit run-audit event: `task:auto-recover-node-unreachable`.
+
+`MeshLeaseManager.isLeaseRecoverable(...)` and `recoverAbandonedLease(...)` provide current recovery seam (`packages/engine/src/mesh-lease-manager.ts:23-107`), extended by policy checks above.
+
+No executor/session process handoff occurs.
+Restarted work is fresh execution from preserved progress metadata.
+
+### 3.4 Transition C: owner returns after handoff
+
+A previously owning node may come back online after another node advanced epoch.
+Returning node behavior:
+
+- On next heartbeat, validate lease ownership.
+- If central claim owner/epoch no longer matches local run, abort cleanly.
+- Do not continue writes, merges, or `fn_task_done`.
+
+Existing detection seam:
+
+- `HeartbeatMonitor.executeHeartbeat()` already validates `task.checkedOutBy` and exits with checkout-conflict semantics (`AGENTS.md` Checkout Leasing section).
+
+Minimal extension needed:
+
+- include central epoch/owner validation (or synchronized local epoch validation) in the same heartbeat preflight so stale owner runs fence themselves before side effects.
+
+### 3.5 Policy enum
+
+Define policy enum for unavailable-owner reassignment behavior:
+
+- `block`
+- `reassign-to-local`
+- `reassign-any-healthy`
+
+Default recommendation:
+
+- Keep default behavior equivalent to today's fallback/local-block pattern unless explicitly configured.
+- For compatibility, map existing `fallback-local` to `reassign-to-local` in transition layer.
+
+### 3.6 Testability surface
+
+Unit policy tests:
+
+- For each enum value, assert expected decision for `{owner online, owner offline, owner error, owner unknown}`.
+
+Scheduler integration tests:
+
+1. Online-owner protection:
+   - pre-seed task owned by online node A
+   - scheduler on node B must not dispatch it
+2. `reassign-to-local` behavior:
+   - pre-seed task owned by offline node A
+   - after lease expiry and policy evaluation, row is released
+   - task becomes dispatch-eligible for local node B
+
+Independence harness note:
+
+- These tests can bypass §2 by directly seeding ownership row + lease timestamps and toggling health monitor outputs.
+
+## 4. Single-Project → Multi-Project / Multi-Node Isolation & Ownership Transition Path
+
+### 4.1 Supported state machine
+
+Supported operational states:
+
+- `single-project/in-process`
+- `multi-project/in-process`
+- `multi-project/child-process`
+- `multi-project/in-process + remote runtime placement`
+
+Transition trigger is runtime reconfiguration requiring restart-bound re-evaluation.
+
+### 4.2 Preconditions
+
+Transition allowed when:
+
+- Node is idle (`0` tasks in `in-progress`), OR
+- Caller passes explicit `force` flag.
+
+If `force=true`:
+
+- In-flight tasks are paused through engine-initiated rebound path.
+- Do not mark tasks as `userPaused`.
+- Resume under new runtime after restart and ownership re-validation.
+
+### 4.3 Runtime selection behavior on transition
+
+On startup after transition:
+
+- Re-evaluate `shouldUseHybridExecutor(...)` gate (`packages/engine/src/hybrid-executor-gate.ts:20`).
+- Re-evaluate runtime type selection in `ProjectManager.addProject(...)` based on `isolationMode` (`packages/engine/src/project-manager.ts:168-170`).
+- Continue to use `CentralCore` registration as source for project placement and mode (`packages/core/src/central-core.ts:252`, `449`).
+
+Per-project SQLite DBs are not migrated or rewritten by this transition.
+They remain at `.fusion/fusion.db` per project.
+
+### 4.4 Central claim bootstrap
+
+On first startup in multi-node-capable mode:
+
+- Apply idempotent migration to create central `taskClaims` table (from §2).
+- Migration must be safe to rerun.
+- No data loss on repeated startups.
+
+### 4.5 Rollback rules
+
+Rollback to single-project mode is only supported when node is sole registered node in `CentralCore`.
+If multiple nodes are registered, reject rollback with actionable error.
+
+Rationale:
+
+- prevents silently orphaning cross-node claim ownership semantics.
+
+### 4.6 Durable trail requirements
+
+Every transition writes durable activity/audit records:
+
+- Activity entry: `project:isolation-transition`
+- Event entry: `project:runtime-restarted` with payload:
+  - `projectId`
+  - `isolationMode`
+  - `reason`
+
+### 4.7 Explicit non-hot-swap scope
+
+Hot-swap isolation transitions without restart are out of scope.
+All supported transitions are restart-bound.
+
+## 5. Non-Goals (Explicit Exclusions)
+
+The following are explicitly out of scope for FN-4819 and MUST NOT be pulled into this brief's implementation follow-up:
+
+1. **Scheduler failover**
+   - Exclusion: a peer taking over the live scheduler tick loop for another node's project runtime.
+   - Rationale: requires runtime leadership election and scheduler state transfer not needed to close lease-ownership gap.
+2. **Live-process state migration**
+   - Exclusion: transferring in-memory executor/session process state mid-task across nodes.
+   - Rationale: requires session serialization/resume semantics beyond lease fencing and handoff policy.
+3. **Cross-node consensus for engine settings mutations**
+   - Exclusion: globally coordinated settings write quorum.
+   - Rationale: orthogonal control-plane consistency problem.
+4. **Multi-master concurrent writes to the same per-project `.fusion/fusion.db`**
+   - Exclusion: active simultaneous writers from multiple nodes to one project DB file.
+   - Rationale: storage topology constraint outside this task's lease contract.
+5. **Automatic node promotion**
+   - Exclusion: autonomous promotion/demotion workflows for node roles.
+   - Rationale: operational governance feature, not checkout correctness.
+
+No follow-up task is to be filed under this brief for these exclusions.
+
+## 6. Acceptance Mapping
+
+- **Acceptance: "four concerns called out as separate sections"**
+  - Satisfied by:
+    - §1 Baseline correctness model
+    - §2 Distributed checkout mutex
+    - §3 Unavailable-node handoff policy
+    - §4 Isolation/ownership transition path
+- **Acceptance: "distributed checkout mutex and unavailable-node handoff policy explicitly independent and testable"**
+  - Satisfied by:
+    - §2 opening independence statement + §2.8 testability surface
+    - §3 opening independence statement + §3.6 testability surface
+- **Acceptance: "WAL-based single-node baseline referenced in the protocol/docs section"**
+  - Satisfied by:
+    - §1 references to `AGENTS.md` Storage Model and `docs/storage.md` WAL details
+- **Acceptance: "non-goal section explicitly excludes scheduler failover/live-process migration"**
+  - Satisfied by:
+    - §5 items 1 and 2
+
+## 7. Code & Doc References
+
+All references below were path-verified during FN-4819 preflight.
+
+- `AGENTS.md:81-85` — Storage Model canonical wording
+- `AGENTS.md:472-486` — Checkout Leasing conflict semantics and heartbeat enforcement notes
+- `docs/storage.md:70` — per-project DB row source (`.fusion/fusion.db`)
+- `docs/storage.md:92-101` — WAL/busy-timeout and concurrent write notes
+- `docs/multi-project.md:19` — central DB path `~/.fusion/fusion-central.db`
+- `docs/multi-project.md:108-113` — isolation mode definitions
+- `docs/multi-project.md:158-166` — ProjectManager runtime-mode mapping narrative
+- `docs/shared-mesh-protocol.md:16-19` — existing protocol non-goals including scheduler failover/live-process migration
+- `docs/architecture.md` — engine composition and run-audit reference baseline
+- `packages/core/src/store.ts:155-160` — task lease field definitions
+- `packages/core/src/store.ts:1571-1576` — persisted lease field write mapping
+- `packages/core/src/store.ts:1599-1624` — insert/upsert columns include lease fields
+- `packages/core/src/agent-store.ts:1378-1426` — `checkoutTask` current single-process lease behavior
+- `packages/core/src/agent-store.ts:47` — `CheckoutConflictError` usage import site
+- `packages/core/src/central-core.ts:7` — central DB location note
+- `packages/core/src/central-core.ts:252-255` — `registerProject` with `isolationMode`
+- `packages/core/src/central-core.ts:449` — `updateProject` mutation path
+- `packages/core/src/central-core.ts:2626` — central DB path getter
+- `packages/core/src/types.ts:3369` — `IsolationMode = "in-process" | "child-process"`
+- `packages/engine/src/project-manager.ts:168-170` — `isolationMode` runtime branching
+- `packages/engine/src/node-routing-policy.ts:4-42` — `PolicyDecision` and `applyUnavailableNodePolicy`
+- `packages/engine/src/mesh-lease-manager.ts:23-107` — lease recoverability and recovery path
+- `packages/engine/src/node-health-monitor.ts:132-148` — offline/error transition handling
+- `packages/engine/src/scheduler.ts:1005` — dispatch-time unavailable-node policy call site
+- `packages/engine/src/hybrid-executor-gate.ts:20` — `shouldUseHybridExecutor` decision seam
+
+## 8. Related Work (Optional Footnote)
+
+No sibling FN design brief is required to interpret this document.
+If adjacent work (for example another FN-48xx mesh coordination brief) exists, treat it as supplemental context only; FN-4819 remains self-contained and normative for this specific coordination-gap specification.