docs(FN-4820): complete Step 2 — author findings document
Fusion-Task-Id: FN-4820 Fusion-Task-Lineage: 8f15714c-4733-4733-8f47-a9b8738b9bcd
This commit is contained in:
Fusion (runfusion.ai)
committed by
gsxdsm
gsxdsm
parent
8405e1fce6
commit
5b2b50e916
204
docs/findings/fn-4820-multi-node-coordination-validation.md
Normal file
204
docs/findings/fn-4820-multi-node-coordination-validation.md
Normal file
@@ -0,0 +1,204 @@
|
||||
# FN-4820 — Multi-Node Coordination Validation vs FN-4819
|
||||
|
||||
## 1. Scope & Method
|
||||
|
||||
### Scope
|
||||
|
||||
This audit validates the **current implementation on `main`** against FN-4819 across exactly three areas:
|
||||
|
||||
1. Scheduler ownership / checkout semantics across nodes
|
||||
2. Wake / assignment propagation across nodes
|
||||
3. Conflict-handling telemetry
|
||||
|
||||
### Explicit non-goals (inherited from FN-4819 §5)
|
||||
|
||||
This audit does **not** propose implementation work for:
|
||||
|
||||
- Scheduler failover
|
||||
- Live-process state migration
|
||||
- Cross-node settings consensus
|
||||
- Multi-master writes to one per-project `.fusion/fusion.db`
|
||||
- Automatic node promotion
|
||||
|
||||
### Method
|
||||
|
||||
- Evidence source: repository files on commit `2595b4111fefe510d2cbc556b83d8b3c50067655`
|
||||
- Method: direct `file:line` code citation from current code paths
|
||||
- Design baseline: `docs/design/fn-4819-distributed-multi-node-coordination-gap.md` (§1–§5)
|
||||
- Classification policy: conservative (`partial` unless requirements are fully met)
|
||||
|
||||
## 2. FN-4819 Concern Coverage Matrix
|
||||
|
||||
| FN-4819 concern | Status | Primary evidence (file:line) | Notes |
|
||||
|---|---|---|---|
|
||||
| §1 Single-node WAL baseline and cross-node gap | **works today** (baseline only) | `packages/core/src/store.ts:155-160`; `packages/core/src/store.ts:3822-3831`; `packages/core/src/agent-store.ts:1371-1460` | Single-project SQLite row-level lease fields + CAS-style update exist. This satisfies the baseline model but does not close distributed arbitration by itself. |
|
||||
| §2 Distributed checkout mutex (central claim authority) | **missing** | `packages/core/src/central-db.ts:36-286` (no `taskClaims` table); repo search: `taskClaims` no hits; `packages/core/src/agent-store.ts:1371-1460` | Checkout logic is task-row-local. No authoritative central claim row keyed by `(projectId, taskId)` as required by FN-4819 §2.2/§2.3. |
|
||||
| §3 Unavailable-node handoff policy | **partial** | `packages/engine/src/node-routing-policy.ts:4-75`; `packages/engine/src/scheduler.ts:1004-1044`; `packages/engine/src/mesh-lease-manager.ts:84-140`; `packages/engine/src/node-health-monitor.ts:103-148` | Policy seams and stale lease recovery exist, but durable run-audit event `task:auto-recover-node-unreachable` from FN-4819 §3.3 is not implemented. |
|
||||
| §4 Isolation/ownership transition path | **partial** | `packages/core/src/central-core.ts:521-547`; `packages/engine/src/hybrid-executor-gate.ts:20-36`; `packages/engine/src/project-manager.ts:168-196`; `packages/core/src/central-db.ts:36-286` | Runtime mode selection + isolation transition activity logging exists, but no central `taskClaims` bootstrap and no visible rollback guard tied to sole-node condition from FN-4819 §4.4/§4.5. |
|
||||
|
||||
## 3. Findings by Audit Area
|
||||
|
||||
### 3.1 Ownership / checkout semantics across nodes
|
||||
|
||||
#### Finding A — Checkout CAS is local-row atomic, not central-distributed
|
||||
|
||||
- **Severity:** P0
|
||||
- **Summary:** Current checkout conflict protection is atomic against one task row, but no cross-node central authority claim exists.
|
||||
- **Current behavior:**
|
||||
- `AgentStore.checkoutTask()` uses holder checks and `tryClaimCheckout` when available (`packages/core/src/agent-store.ts:1371-1460`).
|
||||
- `TaskStore.tryClaimCheckout()` performs compare-and-set over task fields (`checkedOutBy`, `checkoutNodeId`, `checkoutLeaseEpoch`) (`packages/core/src/store.ts:3822-3831`).
|
||||
- Lease fields are persisted on task rows (`packages/core/src/store.ts:155-160`).
|
||||
- **Expected per FN-4819:** FN-4819 §2.2/§2.3 requires an authoritative central claim row (`~/.fusion/fusion-central.db`) keyed by `(projectId, taskId)` for atomic lease arbitration.
|
||||
- **Delta:** No `taskClaims` authority table or central claim mutation path found (`packages/core/src/central-db.ts:36-286`; repo search no `taskClaims` symbol).
|
||||
- **Recommended follow-up:** Add central claim-table mutex path and wire checkout through it before task-row sync.
|
||||
|
||||
#### Finding B — Lease recovery writes only task row and task log entries
|
||||
|
||||
- **Severity:** P1
|
||||
- **Summary:** Abandoned lease recovery can clear lease fields and requeue tasks, but reconciliation remains task-row-local and not central-claim-driven.
|
||||
- **Current behavior:**
|
||||
- `recoverAbandonedLease()` checks staleness and node health, then clears checkout fields and moves task to `todo` (`packages/engine/src/mesh-lease-manager.ts:84-140`).
|
||||
- Scheduler invokes lease recovery during todo dispatch (`packages/engine/src/scheduler.ts:829-836`).
|
||||
- **Expected per FN-4819:** FN-4819 §2.5 and §3.3 require central claim release/sync authority and bounded handoff semantics tied to that authority.
|
||||
- **Delta:** Recovery does not interact with any central lease claim primitive; no central/local reconciliation worker path evidenced.
|
||||
- **Recommended follow-up:** Introduce central-claim-aware recovery + reconciliation semantics for split-write scenarios.
|
||||
|
||||
#### Finding C — Scheduler dispatch guards reduce local double-starts but do not prove multi-node exclusivity
|
||||
|
||||
- **Severity:** P1
|
||||
- **Summary:** Scheduler re-read CAS and handoff checks help local correctness, but no evidence of distributed mutual exclusion beyond local data path.
|
||||
- **Current behavior:**
|
||||
- Scheduler re-reads the task to verify still `todo` before dispatch (`packages/engine/src/scheduler.ts:958-965`).
|
||||
- Owner health and handoff policy are applied before dispatch (`packages/engine/src/scheduler.ts:1004-1044`; `packages/engine/src/node-routing-policy.ts:4-75`).
|
||||
- **Expected per FN-4819:** FN-4819 §2 requires centralized ownership arbitration independent of scheduler timing.
|
||||
- **Delta:** Dispatch policy gates are additive, but no global claim winner/loser contract is present in cited code.
|
||||
- **Recommended follow-up:** Add race tests and scheduler integration assertions against central claim winner semantics.
|
||||
|
||||
### 3.2 Wake / assignment propagation across nodes
|
||||
|
||||
#### Finding D — Assignment wake is event-push local; cross-node wake guarantee is implicit/indirect
|
||||
|
||||
- **Severity:** P1
|
||||
- **Summary:** Local assignment wake hook is explicit; cross-node propagation path appears indirect via remote runtime event stream, not a dedicated assignment-delivery contract.
|
||||
- **Current behavior:**
|
||||
- Local assignment trigger subscribes to `agent:assigned` and invokes callback source `assignment` (`packages/engine/src/agent-heartbeat.ts:3663-3740`).
|
||||
- In-process runtime forwards local `task:*` events (`packages/engine/src/runtimes/in-process-runtime.ts:1186-1194`).
|
||||
- Remote runtime consumes `/api/events/stream` and forwards remote `task:*` events (`packages/engine/src/runtimes/remote-node-runtime.ts:153-213,241-254`; `packages/engine/src/runtimes/remote-node-client.ts:98-120`).
|
||||
- **Expected per FN-4819:** FN-4819 §3 + audit scope expects explicit understanding of owner/peer wake propagation and missed-wake behavior.
|
||||
- **Delta:** No dedicated “task assigned on node A must wake owner on node B” contract or acceptance test seam was found in these modules.
|
||||
- **Recommended follow-up:** Define and test explicit cross-node assignment wake semantics (push/pull fallback, bounded latency, missed-wake recovery).
|
||||
|
||||
#### Finding E — Peer exchange loop provides periodic mesh sync, not assignment wake routing
|
||||
|
||||
- **Severity:** P2
|
||||
- **Summary:** Peer exchange cadence is periodic gossip and settings/shared-state sync; it is not an assignment wake transport.
|
||||
- **Current behavior:**
|
||||
- Peer exchange default sync interval is 120s and syncs online peers sequentially (`packages/engine/src/peer-exchange-service.ts:116-206`).
|
||||
- **Expected per FN-4819:** FN-4819 requires clear operational behavior when wake signals are missed or delayed.
|
||||
- **Delta:** No assignment-specific fallback in peer exchange indicates uncertain wake SLO under event-stream outages.
|
||||
- **Recommended follow-up:** Document wake fallback behavior and add measurable latency/error telemetry for cross-node assignment delivery.
|
||||
|
||||
### 3.3 Conflict-handling telemetry
|
||||
|
||||
#### Finding F — Required `task:auto-recover-node-unreachable` event is absent
|
||||
|
||||
- **Severity:** P1
|
||||
- **Summary:** FN-4819 names a required run-audit event for node-unreachable recovery, but the event type is not present.
|
||||
- **Current behavior:**
|
||||
- Run-audit database event union contains several `task:auto-recover-*` events (`packages/engine/src/run-audit.ts:126-136`).
|
||||
- `task:auto-recover-node-unreachable` is not listed there and not found in engine source search.
|
||||
- **Expected per FN-4819:** FN-4819 §3.3 requires durable run-audit row `task:auto-recover-node-unreachable` on unreachable-owner recovery.
|
||||
- **Delta:** Recovery path currently relies on task log entries / logger messages instead of typed run-audit event.
|
||||
- **Recommended follow-up:** Add typed run-audit event emission for unreachable-owner recovery path and test coverage.
|
||||
|
||||
#### Finding G — Handoff/recovery decisions can complete without structured run-audit classification
|
||||
|
||||
- **Severity:** P2
|
||||
- **Summary:** Handoff decisions are persisted mainly as `taskStore.logEntry` text.
|
||||
- **Current behavior:**
|
||||
- Scheduler logs handoff decisions via `logEntry` (`packages/engine/src/scheduler.ts:1018-1025`).
|
||||
- Mesh lease manager logs and task log entries on recovery (`packages/engine/src/mesh-lease-manager.ts:108-133`).
|
||||
- **Expected per FN-4819:** Durable conflict telemetry should support querying by event class.
|
||||
- **Delta:** Text log entries are less queryable than typed run-audit records.
|
||||
- **Recommended follow-up:** Standardize node-handoff/recovery telemetry into run-audit event taxonomy.
|
||||
|
||||
## 4. Prioritized Follow-Up Backlog
|
||||
|
||||
1. **[TBD] Add authoritative central task-claim mutex for cross-node checkout arbitration**
|
||||
**Severity:** P0
|
||||
Implement FN-4819 §2.2/§2.3 by introducing a central `(projectId, taskId)` claim authority and routing checkout acquisition through it before task-row sync.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- Introduce central claim persistence keyed by `(projectId, taskId)` with owner tuple and epoch semantics from FN-4819 §2.3.
|
||||
- `checkoutTask` winner/loser behavior follows FN-4819 §2.4 (exactly one winner; loser gets `CheckoutConflictError`).
|
||||
- Renewal keeps epoch stable; owner change bumps epoch.
|
||||
- Add race integration test asserting one winner across concurrent claim attempts (FN-4819 §2.8).
|
||||
|
||||
2. **[TBD] Make lease recovery central-claim-aware with reconciliation guarantees**
|
||||
**Severity:** P1
|
||||
Align abandoned-lease recovery and release semantics with FN-4819 §2.5 and §3.3 so central authority and task-row views cannot diverge silently.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- Recovery/release path updates or reconciles central claim + task row according to FN-4819 §2.5.
|
||||
- Split-write failure mode has deterministic repair path before next dispatch.
|
||||
- Scheduler + mesh lease recovery tests cover offline/error owner-node handoff behavior (FN-4819 §3.6).
|
||||
|
||||
3. **[TBD] Specify and test cross-node assignment wake propagation contract**
|
||||
**Severity:** P1
|
||||
Define explicit push/pull/missed-wake behavior for assignment propagation between nodes, with bounded-latency and recovery expectations.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- Document assignment wake contract across node A→node B transitions, including missed-stream fallback.
|
||||
- Add integration tests proving assigned owner receives actionable wake under normal and degraded transport conditions.
|
||||
- Define and expose measurable worst-case wake latency behavior consistent with FN-4819 §3 operational intent.
|
||||
|
||||
4. **[TBD] Emit typed run-audit telemetry for node-unreachable auto-recovery**
|
||||
**Severity:** P1
|
||||
Add structured telemetry for unreachable-owner recovery path so conflict handling is queryable and auditable.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- Add `task:auto-recover-node-unreachable` run-audit event support and emit on applicable recovery paths (FN-4819 §3.3).
|
||||
- Add tests validating event emission metadata for owner node, policy decision, and resulting task transition.
|
||||
- Ensure no recovery completion path is silent (must emit either success/failure classified event).
|
||||
|
||||
5. **[TBD] Normalize node handoff telemetry taxonomy beyond free-text task logs**
|
||||
**Severity:** P2
|
||||
Reduce ambiguity by converting key scheduler/lease-manager handoff text logs into typed event classes.
|
||||
|
||||
**Acceptance criteria:**
|
||||
- Define typed telemetry names for handoff parked/reassign/recovered outcomes.
|
||||
- Keep backward-compatible human-readable task log entries where useful.
|
||||
- Add tests ensuring each decision path emits one structured telemetry event.
|
||||
|
||||
## 5. Non-Goals Reaffirmation
|
||||
|
||||
The following FN-4819 non-goals were respected and **no follow-up task below targets them**:
|
||||
|
||||
- Scheduler failover
|
||||
- Live-process state migration
|
||||
- Cross-node settings consensus
|
||||
- Multi-master concurrent writes to one per-project `.fusion/fusion.db`
|
||||
- Automatic node promotion
|
||||
|
||||
## 6. References
|
||||
|
||||
- `docs/design/fn-4819-distributed-multi-node-coordination-gap.md`
|
||||
- `packages/core/src/store.ts:155-160`
|
||||
- `packages/core/src/store.ts:3822-3831`
|
||||
- `packages/core/src/agent-store.ts:1371-1460`
|
||||
- `packages/core/src/central-db.ts:36-286`
|
||||
- `packages/core/src/central-core.ts:521-547`
|
||||
- `packages/engine/src/scheduler.ts:829-836`
|
||||
- `packages/engine/src/scheduler.ts:958-965`
|
||||
- `packages/engine/src/scheduler.ts:1004-1044`
|
||||
- `packages/engine/src/node-routing-policy.ts:4-75`
|
||||
- `packages/engine/src/mesh-lease-manager.ts:84-140`
|
||||
- `packages/engine/src/node-health-monitor.ts:103-148`
|
||||
- `packages/engine/src/agent-heartbeat.ts:3663-3740`
|
||||
- `packages/engine/src/runtimes/in-process-runtime.ts:1186-1194`
|
||||
- `packages/engine/src/runtimes/remote-node-runtime.ts:153-213`
|
||||
- `packages/engine/src/runtimes/remote-node-runtime.ts:241-254`
|
||||
- `packages/engine/src/runtimes/remote-node-client.ts:98-120`
|
||||
- `packages/engine/src/peer-exchange-service.ts:116-206`
|
||||
- `packages/engine/src/run-audit.ts:97-111`
|
||||
- `packages/engine/src/run-audit.ts:126-136`
|
||||
Reference in New Issue
Block a user