369 lines
19 KiB
Markdown
369 lines
19 KiB
Markdown
# FN-4814 — Multi-node runtime readiness remediation brief
|
|
|
|
## Status & Premise
|
|
|
|
As of FN-4772 (wiring) and FN-4775 (verification), `HybridExecutor` is already the canonical runtime orchestration path when the gate resolves enabled; it is not unwired. The gate order in `packages/engine/src/hybrid-executor-gate.ts` is explicit and current: env override (`FUSION_HYBRID_EXECUTOR=1|0`) → multi-node (`nodes.length > 1`) → multi-project (`liveProjects.length > 1`) → disabled (`single-project-local-only`) → disabled on central failures (`central-unavailable`) (`packages/engine/src/hybrid-executor-gate.ts:8-41`).
|
|
|
|
## Scope Frame
|
|
|
|
| Area | In scope for this brief (input to FN-4813) | Out of scope for this brief |
|
|
| --- | --- | --- |
|
|
| Runtime truth | Confirm current architecture and gate behavior as already landed | Re-litigating whether HybridExecutor should exist |
|
|
| Ownership | Define distributed claim mutex boundary for task ownership | General-purpose distributed lock service for arbitrary state |
|
|
| Unavailable node behavior | Define pre-dispatch vs mid-task unreachable behavior and recovery contract | Live process migration of an in-flight task |
|
|
| Isolation transition | Define supported single-project → multi-project/child-process/remote-node transition model | Hot-swap transition without restart |
|
|
| Scheduler model | Clarify current routing boundary and handoff interactions | Scheduler failover across nodes |
|
|
| Session state | Define ownership-loss abort behavior | Cross-node mutation/replication of executor in-memory session internals |
|
|
| Topology control | Define rollback guard when returning to single-node mode | Automatic node promotion/election |
|
|
|
|
### Explicit non-remediation directives
|
|
|
|
The following are acknowledged and intentionally excluded from remediation under this brief:
|
|
|
|
- Scheduler failover.
|
|
- Live-process migration of an in-flight task across nodes.
|
|
- Automatic node promotion.
|
|
- Cross-node mutation of executor session state.
|
|
|
|
No follow-up tasks should be filed under FN-4814 for those items.
|
|
|
|
## Ownership Boundaries
|
|
|
|
### Boundary map when `shouldUseHybridExecutor(...).enabled === true`
|
|
|
|
```text
|
|
+-----------------------------------------------------------------------+
|
|
| CentralCore (~/.fusion/fusion-central.db) |
|
|
| - Node registry |
|
|
| - Project registry |
|
|
| - Proposed distributed task-claim mutex table (projectId, taskId key) |
|
|
+-------------------------------+---------------------------------------+
|
|
|
|
|
v
|
|
+-----------------------------------------------------------------------+
|
|
| HybridExecutor |
|
|
| - Selects/loads ProjectRuntime instances |
|
|
| - Starts/stops NodeHealthMonitor lifecycle |
|
|
+-------------------------------+---------------------------------------+
|
|
|
|
|
+-------------------+-------------------+
|
|
v v
|
|
+-------------------------------+ +-----------------------------------+
|
|
| ProjectEngineManager/Engine | | Scheduler |
|
|
| - Per-project engine lifecycle| | - Calls applyUnavailableNodePolicy|
|
|
| - Existing ownership unchanged| | - Pre-dispatch routing decisions |
|
|
+-------------------------------+ +-----------------------------------+
|
|
|
|
|
v
|
|
+-----------------------------------------------------------------------+
|
|
| Per-project DB (.fusion/fusion.db) |
|
|
| - Existing task row and local lease fields, including checkedOutBy |
|
|
| - Remains local source for task lifecycle data |
|
|
+-----------------------------------------------------------------------+
|
|
```
|
|
|
|
### Ownership statements
|
|
|
|
- `CentralCore` is the only shared visibility plane across nodes and therefore owns the distributed claim record.
|
|
- `HybridExecutor` owns orchestration composition (`ProjectRuntime` selection) plus node health monitor lifecycle; it does not become a distributed lock coordinator itself.
|
|
- `ProjectEngineManager` and `ProjectEngine` remain the per-project lifecycle owners; this brief does not alter that contract.
|
|
- `Scheduler` remains the call-site boundary for `applyUnavailableNodePolicy` pre-dispatch decisions (`packages/engine/src/scheduler.ts:1035-1039`).
|
|
- Per-project `.fusion/fusion.db` keeps local lease semantics and task state (`Task.checkedOutBy`), but cross-node authority is lifted to the central claim row.
|
|
|
|
## Distributed Checkout / Claim Mutex Design
|
|
|
|
### Goal
|
|
|
|
Provide a durable, cross-node, per-task ownership claim so two nodes cannot concurrently own the same task execution lane.
|
|
|
|
### Proposed central table (design sketch)
|
|
|
|
Table name (proposed): `task_claims`
|
|
|
|
Primary/unique key:
|
|
|
|
- `UNIQUE(projectId, taskId)`
|
|
|
|
Required columns (minimum):
|
|
|
|
- `projectId` — string/project identifier.
|
|
- `taskId` — string/task identifier.
|
|
- `nodeId` — owning node identifier.
|
|
- `ownerAgentId` — current owning agent identifier.
|
|
- `acquiredAt` — unix ms/iso timestamp of first acquisition for epoch.
|
|
- `heartbeatAt` — last successful owner heartbeat timestamp.
|
|
- `leaseTtlMs` — lease TTL assigned on acquisition.
|
|
- `epoch` — monotonically increasing ownership generation.
|
|
|
|
Recommended operational columns (optional but useful):
|
|
|
|
- `runId` — active execution run correlation.
|
|
- `updatedAt` — last write audit convenience.
|
|
- `releasedAt` — explicit release timestamp for diagnostics.
|
|
|
|
### Acquisition semantics
|
|
|
|
- Claim path uses single-writer semantics with `INSERT ... ON CONFLICT(projectId, taskId)`.
|
|
- Winner acquires claim and sets initial `acquiredAt`, `heartbeatAt`, `leaseTtlMs`, and `epoch`.
|
|
- Loser gets a conflict result that is treated as 409-equivalent ownership contention.
|
|
- Loser must **not** auto-retry in a tight loop, matching checkout conflict contract in AGENTS.md.
|
|
|
|
### Heartbeat semantics
|
|
|
|
- The owner refreshes `heartbeatAt` on each task heartbeat tick.
|
|
- Claim is live when `now - heartbeatAt < leaseTtlMs`.
|
|
- Heartbeat update must be guarded by current `(projectId, taskId, epoch, nodeId, ownerAgentId)` to prevent stale owner refresh.
|
|
|
|
### Expiry/reclaim semantics
|
|
|
|
A peer may reclaim only if **both** are true:
|
|
|
|
1. Lease has expired (`now - heartbeatAt >= leaseTtlMs`), and
|
|
2. `NodeHealthMonitor` reports owner status `offline` or `error`.
|
|
|
|
Rationale:
|
|
|
|
- Expiry-only can steal from transiently slow but healthy owners.
|
|
- Health-only can steal from temporarily delayed heartbeats.
|
|
- Conjunction reduces split-brain risk for active owners.
|
|
|
|
### Relationship to per-project `Task.checkedOutBy`
|
|
|
|
- Central claim is authoritative across nodes.
|
|
- Local `Task.checkedOutBy` remains local engine lease signal.
|
|
- Acquisition/release operations must update central claim and local row atomically from caller perspective (single success/failure outcome).
|
|
- If either side fails, operation is considered failed and must roll back/compensate before dispatch continues.
|
|
|
|
### Migration/bootstrap
|
|
|
|
- Add claim table in `fusion-central.db` via idempotent central migration.
|
|
- Migration runs on first multi-mode startup (or first startup after introducing schema).
|
|
- No historical backfill is required: prior single-node operation had no cross-node claim contention.
|
|
|
|
### Guardrail (non-goal)
|
|
|
|
This mutex governs **task ownership claims only**. It is not a distributed lock primitive for arbitrary engine mutations, scheduler internals, or settings writes.
|
|
|
|
## Unavailable-Node Handoff Behavior
|
|
|
|
This section layers on top of existing `applyUnavailableNodePolicy` behavior in `packages/engine/src/node-routing-policy.ts:16-47`.
|
|
|
|
### 1) Pre-dispatch unavailable (existing behavior preserved)
|
|
|
|
- Keep current semantics unchanged.
|
|
- `fallback-local`: dispatch allowed and rewritten to local.
|
|
- `block`: dispatch denied with policy reason.
|
|
- Call-site remains scheduler pre-dispatch boundary (`packages/engine/src/scheduler.ts:1035-1056`).
|
|
|
|
### 2) Owner becomes unreachable mid-task
|
|
|
|
Behavior contract:
|
|
|
|
- In-flight task stays owned by original central claim until lease expiry.
|
|
- No live process migration is attempted (explicit non-goal).
|
|
- When lease expires **and** owner health is `offline`/`error`:
|
|
- engine-initiated rebound moves task to `todo`;
|
|
- rebound uses `preserveProgress` and `preserveWorktree=false`;
|
|
- stale claim is released/replaced by reclaiming node;
|
|
- emit run-audit event `task:auto-recover-node-unreachable`.
|
|
|
|
Worktree treatment:
|
|
|
|
- Worktree on unreachable owner is forfeit.
|
|
- New owner recreates fresh worktree/session using existing recovery posture aligned with FN-4601 session-start recovery patterns.
|
|
|
|
### 3) Owner returns after handoff
|
|
|
|
Behavior contract:
|
|
|
|
- Returning node checks whether it still owns central claim epoch.
|
|
- If claim epoch/node ownership no longer matches:
|
|
- abort active session cleanly;
|
|
- clear local `checkedOutBy` row state;
|
|
- do not commit/finalize stale session output.
|
|
|
|
Detection point:
|
|
|
|
- Existing heartbeat guard already validates local checkout ownership in `HeartbeatMonitor.executeHeartbeat()` per AGENTS guidance.
|
|
- Extend same guard to compare local owner identity against central claim epoch.
|
|
|
|
User pause invariant:
|
|
|
|
- This is engine-initiated recovery, not a user pause.
|
|
- Do not set `userPaused`; preserve existing engine rebound semantics from AGENTS “Engine Process Rules”.
|
|
|
|
## Single-Project → Multi-Project Transition Path
|
|
|
|
### Supported state machine
|
|
|
|
```text
|
|
single-project local mode
|
|
-> (config change + restart, if preconditions met)
|
|
multi-project/hybrid orchestration mode
|
|
-> (config change + restart, rollback guard passes)
|
|
single-project local mode
|
|
```
|
|
|
|
### Preconditions for enabling multi-mode
|
|
|
|
- Node must be idle (no `in-progress` tasks).
|
|
- If non-idle, reject transition with explicit “drain tasks first” error.
|
|
|
|
### Transition execution contract
|
|
|
|
- On next startup, re-evaluate `shouldUseHybridExecutor`.
|
|
- If enabled, call `HybridExecutor.initialize()`.
|
|
- `ProjectEngineManager` ownership of per-project engines remains unchanged.
|
|
- Existing per-project SQLite databases remain untouched (consistent with `docs/multi-project.md`).
|
|
|
|
### Claim-mutex bootstrap during transition
|
|
|
|
- First multi-mode startup runs idempotent central claim table migration.
|
|
- Migration is safe to run repeatedly and no-op when already applied.
|
|
|
|
### Rollback to single-project constraints
|
|
|
|
- Rollback allowed only when local node is the sole registered node in `CentralCore`.
|
|
- If multiple nodes remain registered, reject rollback to prevent orphaned remote orchestration peers.
|
|
|
|
### Explicit transition non-goal
|
|
|
|
- Hot-swap isolation transition without restart is not supported by this brief.
|
|
|
|
## Acknowledged Non-Goals (No Remediation Required)
|
|
|
|
- **Scheduler failover across nodes**: maintaining single scheduler loops per node avoids distributed election complexity and is sufficient with explicit ownership claims.
|
|
- **Live-process migration of in-flight tasks**: process/state transfer cost and correctness risk outweigh value versus lease-expiry rebound.
|
|
- **Cross-node consensus for engine settings mutations**: settings convergence is out of the task-ownership safety path and would introduce high coordination overhead.
|
|
- **Multi-master writes to the same per-project DB simultaneously**: per-project DB remains local-owner scoped; central claim prevents concurrent execution ownership races instead of enabling shared-master semantics.
|
|
|
|
No follow-up task should be filed under this brief for these items.
|
|
|
|
## Code References Summary
|
|
|
|
- `packages/engine/src/hybrid-executor-gate.ts:8-41` — `parseEnvOverride`, `shouldUseHybridExecutor` gate order and reasons.
|
|
- `packages/engine/src/hybrid-executor.ts:168-214` — `HybridExecutor.initialize()` and `NodeHealthMonitor` lifecycle startup.
|
|
- `packages/engine/src/node-routing-policy.ts:16-47` — `applyUnavailableNodePolicy` dispatch decision contract.
|
|
- `packages/engine/src/node-routing-policy.ts:49-80` — `decideOwningNodeHandoff` helper semantics.
|
|
- `packages/engine/src/scheduler.ts:1035-1056` — scheduler call-site for `applyUnavailableNodePolicy` and local fallback rewrite.
|
|
- `packages/engine/src/runtimes/in-process-runtime.ts` — local runtime path under `ProjectRuntime` abstraction.
|
|
- `packages/engine/src/runtimes/child-process-runtime.ts` — child-process runtime path.
|
|
- `packages/engine/src/runtimes/remote-node-runtime.ts` — remote runtime path.
|
|
- `packages/engine/src/runtimes/child-process-worker.ts` — child worker process runtime implementation.
|
|
- `packages/engine/src/runtimes/remote-node-client.ts` — remote-node runtime client path.
|
|
- `packages/engine/src/project-manager.ts` — runtime orchestration internals under hybrid layer.
|
|
- `packages/engine/src/project-engine-manager.ts` — per-project engine lifecycle owner.
|
|
- `packages/engine/src/node-health-monitor.ts` — node health signal source used by routing/handoff policy.
|
|
- `packages/engine/src/__tests__/hybrid-executor-multi-node-routing.test.ts` — multi-node routing coverage.
|
|
- `packages/engine/src/__tests__/node-routing-policy.test.ts` — routing policy unit coverage.
|
|
- `packages/engine/src/__tests__/hybrid-executor-startup.integration.test.ts` — startup/wiring integration coverage.
|
|
- `packages/core/src/types.ts:1631` — `Task.checkedOutBy` task lease field.
|
|
- `packages/core/src/types.ts:283` — `UnavailableNodePolicy` type.
|
|
- `packages/core/src/types.ts:3410` — `NodeStatus` type.
|
|
- `docs/multi-project.md` — existing public runtime narrative and hybrid wiring section.
|
|
- `docs/architecture.md` — startup architecture overview context.
|
|
- `AGENTS.md` — canonical guardrails for checkout leasing and engine rebound behavior.
|
|
|
|
## Implementation Handoff
|
|
|
|
Mapping from this brief to FN-4813 implementation work items:
|
|
|
|
1. **Brief §3 Ownership Boundaries** → establish integration points and ownership contracts in implementation plan.
|
|
2. **Brief §4 Claim Mutex Design** → add central claim schema + acquire/heartbeat/reclaim/release flow.
|
|
3. **Brief §4 Relationship to `checkedOutBy`** → add atomic coupling between central claim row and per-project lease updates.
|
|
4. **Brief §5 Mid-task unreachable behavior** → implement lease-expiry + health-gated rebound and `task:auto-recover-node-unreachable` audit event.
|
|
5. **Brief §5 Owner returns after handoff** → extend heartbeat ownership checks with central claim epoch comparison and stale-owner abort path.
|
|
6. **Brief §6 Transition path** → implement isolation-mode transition precondition checks, startup migration bootstrap trigger, and rollback guard for multi-node registry presence.
|
|
7. **Brief §7 Non-goals** → enforce scope boundaries during implementation review; do not add scheduler failover/live migration/consensus features under FN-4813.
|
|
|
|
## Operational Sequences (Normative)
|
|
|
|
### Sequence A — Initial claim acquisition
|
|
|
|
1. Scheduler selects a task candidate in `todo`.
|
|
2. Executor prepares ownership acquisition intent (`projectId`, `taskId`, `nodeId`, `ownerAgentId`, `leaseTtlMs`).
|
|
3. Central claim write is attempted with single-writer conflict semantics.
|
|
4. On success:
|
|
- central row is visible with current `epoch`;
|
|
- local task lease (`checkedOutBy`) is set by same ownership flow;
|
|
- execution proceeds.
|
|
5. On conflict:
|
|
- caller receives ownership contention result (409-equivalent);
|
|
- caller does not auto-retry in a loop;
|
|
- task remains available for future scheduling pass.
|
|
|
|
### Sequence B — Normal heartbeat renewal
|
|
|
|
1. Owner heartbeat tick runs on existing cadence.
|
|
2. Renewal updates `heartbeatAt` only when `(projectId, taskId, nodeId, ownerAgentId, epoch)` match.
|
|
3. If update succeeds, ownership remains valid.
|
|
4. If update fails due to epoch mismatch, owner has been superseded and must abort its session.
|
|
|
|
### Sequence C — Mid-task owner unreachable
|
|
|
|
1. Owner becomes unreachable (network/process failure).
|
|
2. Peers observe degraded health but do not reclaim immediately.
|
|
3. Reclaim eligibility begins only after both:
|
|
- lease TTL expiry, and
|
|
- owner health `offline` or `error`.
|
|
4. Recovering node performs engine-initiated rebound:
|
|
- task moved to `todo` with `preserveProgress` and `preserveWorktree=false`;
|
|
- claim is reclaimed/replaced;
|
|
- run-audit event `task:auto-recover-node-unreachable` emitted.
|
|
5. New owner starts clean worktree/session.
|
|
|
|
### Sequence D — Returning stale owner
|
|
|
|
1. Original owner recovers after handoff.
|
|
2. Next heartbeat/ownership check compares local lease to central claim epoch.
|
|
3. Mismatch result triggers stale-owner abort path:
|
|
- stop execution;
|
|
- clear local `checkedOutBy`;
|
|
- suppress any finalize/commit from stale run.
|
|
|
|
### Sequence E — Transition to multi-mode
|
|
|
|
1. User requests isolation transition.
|
|
2. Runtime verifies node idle precondition.
|
|
3. If non-idle, transition rejected and user instructed to drain.
|
|
4. If idle, user restarts process.
|
|
5. Startup re-evaluates hybrid gate.
|
|
6. On enabled path:
|
|
- `HybridExecutor.initialize()` starts runtime orchestration;
|
|
- central claim table migration runs idempotently.
|
|
|
|
## Failure-Mode Matrix
|
|
|
|
| Failure mode | Detection source | Required behavior |
|
|
| --- | --- | --- |
|
|
| Claim row conflict at acquisition | Central DB conflict result | Treat as 409-equivalent; no tight auto-retry |
|
|
| Owner heartbeat delayed but node still online | `heartbeatAt` lag + health `online` | Do not reclaim |
|
|
| Owner offline but lease still live | Health monitor + TTL check | Do not reclaim |
|
|
| Owner offline and lease expired | Health monitor + TTL check | Reclaim eligible; rebound to `todo` |
|
|
| Returning owner after reclaim | Central epoch mismatch | Abort stale owner; clear local lease |
|
|
| Transition request with active tasks | Local runtime state | Reject transition; require drain |
|
|
| Rollback-to-single while >1 registered node | Central node registry | Reject rollback to avoid orphaning peers |
|
|
|
|
## Guardrails for FN-4813 Implementation Review
|
|
|
|
The following checks are mandatory when validating FN-4813 against this brief:
|
|
|
|
- Verify no code path introduces automatic retries for central claim conflicts.
|
|
- Verify reclaim requires TTL expiry **and** unhealthy owner status.
|
|
- Verify engine rebound path does not set `userPaused`.
|
|
- Verify stale-owner return path blocks duplicate finalize/commit.
|
|
- Verify transition guards are restart-based and reject hot-swap behavior.
|
|
- Verify no implementation extends scope into scheduler failover or live migration.
|
|
|
|
## Terminology
|
|
|
|
- **Central claim**: shared ownership row in `fusion-central.db` keyed by `(projectId, taskId)`.
|
|
- **Local lease**: per-project task-row ownership signal (e.g., `checkedOutBy`) in `.fusion/fusion.db`.
|
|
- **Epoch**: monotonically increasing ownership generation used to detect stale owners.
|
|
- **Rebound**: engine-initiated task move back to `todo` for safe re-dispatch.
|
|
- **Forfeit worktree**: unreachable-owner worktree treated as non-recoverable state.
|
|
|
|
## Closure Statement
|
|
|
|
This brief intentionally narrows FN-4813 scope to distributed claim ownership, unavailable-owner handoff behavior, and isolation transition guards while preserving existing runtime topology assumptions. Any proposal that expands into failover-election, live-process migration, or distributed consensus falls outside this design predicate and should be rejected during FN-4813 implementation review.
|