docs(FN-4491): add milestone B DAG scaffold design artifacts

Fusion-Task-Id: FN-4491 Fusion-Task-Lineage: 83356e98-5ab8-4f3e-8b46-2960c350a79e
2026-05-14 11:22:11 -07:00
parent 12d0d8d90f
commit 2b26a2003e
4 changed files with 331 additions and 1 deletions
--- a/docs/dag/milestone-b-dag-coordinator-design.md
+++ b/docs/dag/milestone-b-dag-coordinator-design.md
@@ -0,0 +1,130 @@
+# Milestone B DagCoordinator Enqueue-Only Adapter Design
+
+Related tasks: **FN-4491**, **FN-4490**, **FN-4487**, **FN-4471**, governance gate **FN-4359**.
+
+See also: [Schema migration plan](./milestone-b-schema-migration-plan.md) · [Implementation checklist](./milestone-b-implementation-checklist.md) · [ADR v1](./adr-0001-dag-orchestration.md)
+
+## Governance gate
+
+Per `AGENTS.md` reliability governance policy:
+
+> "Reliability mechanism changes are currently under freeze pending FN-4359 governance hardening; treat new reliability-layer behavior changes as blocked unless explicitly approved in task scope."
+
+This design is architecture-only. Milestone B implementation requires freeze lift or explicit carve-out.
+
+## Adapter contract
+
+Proposed file: `packages/engine/src/dag-coordinator.ts`.
+
+`DagCoordinator` is **enqueue-only**:
+- evaluates node readiness from DAG tables.
+- creates/unblocks tasks through `TaskStore` public APIs.
+- triggers wakeups through `HeartbeatTriggerScheduler` surface.
+
+`DagCoordinator` MUST NOT:
+- call scheduler internals in `packages/engine/src/scheduler.ts`.
+- mutate `AgentSemaphore` directly (`packages/engine/src/concurrency.ts`).
+- bypass checkout leasing / 409 conflict semantics.
+
+Proposed methods:
+- `startDagRun(spec: DagRunSpec): Promise<{ runId: string }>`
+- `onTaskMoved(event: { taskId: string; from: string; to: string; status?: string }): Promise<void>`
+- `onTaskUpdated(event: { taskId: string; status?: string; column?: string }): Promise<void>`
+- `cancelDagRun(runId: string, reason: string): Promise<void>`
+- `tick(runId?: string): Promise<void>` (optional bounded reconciliation pass)
+
+## Event sources
+
+Existing observable events from current engine/task store surfaces:
+- `task:created`
+- `task:moved`
+- `task:updated`
+
+Evidence in source scan:
+- `packages/engine/src/scheduler.ts` subscribes to `task:created`, `task:moved`, `task:updated`.
+- `packages/engine/src/executor.ts` subscribes to `task:moved`, `task:updated`.
+- `packages/engine/src/project-manager.ts` forwards runtime `task:created`, `task:moved`, `task:updated`.
+
+No dedicated `task:failed` event was confirmed in existing public event vocabulary; failure detection should currently derive from `task:updated` status/column state.
+
+## Required engine seams (additive only)
+
+1. Additive coordinator wiring point in project engine bootstrap to register coordinator listeners without changing scheduler dispatch semantics.
+2. Optional narrow helper on task store/runtime event payloads for explicit terminal reason mapping (if current `task:updated` payload lacks enough fidelity).
+3. Feature-flag check seam (`experimentalDagCoordinator`) at wiring boundary so default behavior is unchanged.
+
+All seams must be additive and behavior-preserving while FN-4359 freeze is active.
+
+## Concurrency and event-loop safety
+
+- Coordinator logic must be async and non-blocking.
+- No `execSync` in coordinator paths (aligns with AGENTS.md Engine Process Rules).
+- Any external command path (if ever needed) must use async `exec`/`promisify(exec)` with timeout.
+- Readiness evaluation should use bounded batches to avoid monopolizing the single Node event loop.
+
+## Merge-path integration (explicit invariant)
+
+The coordinator does **not** change merger behavior. It does not alter:
+- post-squash audit,
+- file-scope invariant,
+- gitignored-path guard,
+- one-task-one-branch execution/merge mapping.
+
+## Failure handling map (from FN-4490 contract)
+
+- **Per-node failure** → mark `dag_node.status=failed`; emit `dag:node:fail`; evaluate run abort/continue policy.
+- **Edge failure/dependency unsatisfied** → keep downstream `blocked`/`pending`; emit `dag:node:blocked` with reason.
+- **Partial-DAG abort** → set run `aborted`; mark remaining non-terminal nodes `skipped`/`blocked`; emit `dag:run:abort`.
+- **Whole-DAG cancel** → mark run `cancelled`; mark non-terminal nodes `cancelled`; emit `dag:run:abort` with cancel reason.
+- **Retry-exhausted** → defer to existing executor retry exhaustion path (incl. FN-4398 `retriesBurned` semantics); coordinator records terminal node outcome, no custom retry engine.
+- **Governance-blocked** (e.g., FN-3973/FN-4488 policy denial) → emit `dag:node:blocked` with policy reason; do not retry automatically.
+
+## Observability and audit
+
+Logger prefix: **`[dag-coordinator]`** (aligned with existing subsystem prefix style in `packages/engine/src/logger.ts`).
+
+Minimum events (FN-4490 contract):
+- `dag:run:start`
+- `dag:node:enqueue`
+- `dag:node:complete`
+- `dag:node:fail`
+- `dag:run:complete`
+- `dag:run:abort`
+- additive: `dag:node:blocked`
+
+Each event payload should include at least: `runId`, `dagRunId`, `nodeId`, `taskId?`, `status`, `reasonCode?`, `timestamp`.
+
+Run-audit linkage:
+- database-domain audit entries for DAG row transitions.
+- git-domain unchanged (normal task flow only).
+- filesystem-domain only for normal task artifacts, not DAG-specific side channels.
+
+## Multi-project scope
+
+Decision: **per-project coordinator instance**.
+
+Rationale (from `docs/multi-project.md`):
+- task/state persistence is project-local in `.fusion/fusion.db`.
+- central DB manages registry/global coordination, not per-project task execution state.
+- one coordinator per project runtime keeps ownership boundaries consistent with existing runtime model.
+
+## Test strategy for Milestone B implementor
+
+1. **Unit tests** (`packages/engine/src/__tests__/`) for readiness evaluation, enqueue behavior, and event emission.
+2. **Integration tests** with real SQLite project DB (same init/migration path), mock heartbeat scheduler trigger points, and fixture DAG specs.
+3. Use existing executor helper conventions in `packages/engine/src/__tests__/executor-test-helpers.ts` when synthesizing worktree/task execution preconditions.
+4. Add/update interaction tests in `packages/engine/src/__tests__/reliability-interactions/` for scheduler/executor/self-healing/restart-recovery adjacency.
+
+## Out of scope
+
+- Dashboard DAG UX/product surfaces (deferred to FN-4492 / Milestone C).
+- Cross-project or cross-node DAG execution.
+- Time-travel/replay engine.
+- Autoscaling/capacity orchestration redesign.
+
+## Open questions
+
+1. Should coordinator reconcile missed events solely via periodic `tick()` or rely on exhaustive event subscriptions plus startup scan?
+2. Where should DAG run initiation be invoked from (API route, workflow step hook, mission loop) for minimal coupling?
+3. What is the minimal policy reason-code taxonomy for `dag:node:blocked` to support diagnostics without new reliability behavior?
+4. Should node completion depend strictly on task column transitions (`done`) or include additional status fields for retry-exhausted terminal mapping?
--- a/docs/dag/milestone-b-implementation-checklist.md
+++ b/docs/dag/milestone-b-implementation-checklist.md
@@ -0,0 +1,72 @@
+# Milestone B Implementation Checklist (Executor-Ready)
+
+Related tasks: **FN-4491**, **FN-4490**, **FN-4487**, **FN-4471**, governance gate **FN-4359**.
+
+See also: [Schema migration plan](./milestone-b-schema-migration-plan.md) · [DagCoordinator design](./milestone-b-dag-coordinator-design.md) · [ADR v1](./adr-0001-dag-orchestration.md)
+
+## Gate (must be first)
+
+- [ ] **FN-4359 freeze lifted or explicit carve-out granted for DAG prototype implementation. Until then, do not implement.**
+  - Policy quote (`AGENTS.md`): “Reliability mechanism changes are currently under freeze pending FN-4359 governance hardening; treat new reliability-layer behavior changes as blocked unless explicitly approved in task scope.”
+
+## Ordered milestones
+
+1. [ ] **Land additive migration(s) from plan**
+   - Outcome: `dag_run`, `dag_node`, `dag_edge` created via next schema version block.
+   - File scope hints: `packages/core/src/db.ts`, `packages/core/src/__tests__/db*.test.ts`.
+   - Verification: `pnpm test`, `pnpm build`.
+
+2. [ ] **Add `DagCoordinator` skeleton behind feature flag, no subscribers yet**
+   - Outcome: inert class and wiring seam exists but disabled by default.
+   - File scope hints: `packages/engine/src/dag-coordinator.ts`, `packages/engine/src/project-engine.ts` (or equivalent bootstrap).
+   - Verification: `pnpm test`, `pnpm build`.
+
+3. [ ] **Wire event subscriptions behind same flag**
+   - Outcome: listens to existing `task:created|task:moved|task:updated` and performs enqueue-only transitions.
+   - File scope hints: `packages/engine/src/dag-coordinator.ts`, runtime registration surfaces.
+   - Verification: `pnpm test`, `pnpm build`.
+
+4. [ ] **Add focused unit tests**
+   - Outcome: readiness evaluation, enqueue path, block/fail/complete event coverage.
+   - File scope hints: `packages/engine/src/__tests__/dag-coordinator.test.ts` (new), helper fixtures.
+   - Verification: `pnpm test`.
+
+5. [ ] **Add reliability interaction regression tests**
+   - Outcome: explicit interaction coverage in reliability backstop suite for scheduler/executor/self-healing/restart recovery adjacency.
+   - File scope hints: `packages/engine/src/__tests__/reliability-interactions/`.
+   - Verification: `pnpm test`.
+
+6. [ ] **Add opt-in integration test for 2-node DAG**
+   - Outcome: validates end-to-end enqueue-only orchestration with real SQLite state and mocked/controlled wakeups.
+   - File scope hints: `packages/engine/src/__tests__/dag-coordinator.integration.test.ts`.
+   - Verification: `pnpm test`, `pnpm build`.
+
+7. [ ] **Finalize docs + changeset when implementation lands**
+   - Outcome: update DAG docs and settings docs for shipped flag/behavior.
+   - File scope hints: `docs/dag/*.md`, `docs/settings-reference.md`, `.changeset/*.md`.
+   - Verification: `pnpm test`, `pnpm build`.
+
+## Config flag proposal
+
+- Proposed setting: `experimentalDagCoordinator: boolean` (default `false`).
+- Planned definition locations when implementation lands:
+  - `packages/core/src/types.ts` (project settings type)
+  - settings read/write plumbing
+  - `docs/settings-reference.md` (documented only when implemented)
+
+## Changeset requirement for implementation task
+
+When Milestone B implementation (code) lands, include a changeset for **`@runfusion/fusion`** because it introduces new user-facing functionality behind a flag.
+
+- Acceptable bump: `patch` (if strictly bugfix-like/internal behavior) or `minor` (if introducing opt-in new capability surface).
+- This current FN-4491 design-only task does **not** add a changeset.
+
+## Non-goals (self-contained)
+
+- No scheduler replacement.
+- No direct `AgentSemaphore` manipulation.
+- No checkout-lease bypass.
+- No merger/audit/file-scope invariant changes.
+- No dashboard DAG UX/productization (Milestone C / FN-4492).
+- No cross-project/cross-node DAG orchestration.
+- No replay/time-travel/autoscaling orchestration engine.
--- a/docs/dag/milestone-b-schema-migration-plan.md
+++ b/docs/dag/milestone-b-schema-migration-plan.md
@@ -0,0 +1,128 @@
+# Milestone B Schema Migration Plan (Prototype Scaffold)
+
+Related tasks: **FN-4491**, **FN-4490**, **FN-4487**, **FN-4471**, governance gate **FN-4359**.
+
+See also: [DagCoordinator design](./milestone-b-dag-coordinator-design.md) · [Implementation checklist](./milestone-b-implementation-checklist.md) · [ADR v1](./adr-0001-dag-orchestration.md)
+
+## Goals and non-goals
+
+### Goals
+- Additive-only SQLite schema plan for per-project `.fusion/fusion.db` DAG persistence.
+- Preserve WAL mode and current migration runner contract in `packages/core/src/db.ts`.
+- Keep startup safe: no destructive backfills, no blocking startup jobs.
+- Ensure restart recovery can reconstruct DAG state from SQLite alone, aligned with `RestartRecoveryCoordinator` expectations from FN-4490 deliverables.
+
+### Non-goals
+- No schema file or runtime implementation in this task.
+- No column renames/drops/destructive migrations.
+- No changes to `~/.fusion/fusion-central.db` central registry schema.
+
+## Current state
+
+- Project DB migration runner lives in `packages/core/src/db.ts`:
+  - `Database.init()` calls `migrate()`.
+  - Incremental migrations are version-gated `if (version < N)` blocks.
+  - Current top migration in-tree is `version < 77` (`SCHEMA_VERSION`-driven sequence).
+  - Version bump is written through `applyMigration(targetVersion, fn)` updating `__meta.schemaVersion`.
+- Existing DB settings preserve WAL + busy timeout in constructor (`PRAGMA journal_mode = WAL`, `PRAGMA busy_timeout = ...`) in `packages/core/src/db.ts`.
+- Central registry DB is distinct (`~/.fusion/fusion-central.db`) per `docs/multi-project.md`; DAG prototype schema changes are project DB only.
+
+## Proposed additive tables
+
+Naming/type conventions follow existing project DB patterns (`snake_case`, `TEXT` IDs, `INTEGER` flags/counts, ISO timestamps as `TEXT`).
+
+### 1) `dag_run`
+
+Suggested columns:
+- `id TEXT PRIMARY KEY`
+- `project_id TEXT NOT NULL` (project scope identifier; per-project DB still records scope for auditability)
+- `status TEXT NOT NULL` (`pending|running|completed|aborted|cancelled|failed`)
+- `started_at TEXT`
+- `completed_at TEXT`
+- `created_at TEXT NOT NULL`
+- `updated_at TEXT NOT NULL`
+- `metadata TEXT` (JSON payload for run-level config/trace context)
+
+Indexes:
+- `CREATE INDEX ... ON dag_run(status)`
+- `CREATE INDEX ... ON dag_run(project_id, status)`
+- `CREATE INDEX ... ON dag_run(created_at)`
+
+Rationale:
+- `started_at`/`completed_at` nullable to support queued runs.
+- `project_id` kept explicit for consistency with audit/event payloads and future cross-node read tooling, while still local to one project DB instance.
+
+### 2) `dag_node`
+
+Suggested columns:
+- `id TEXT PRIMARY KEY`
+- `dag_run_id TEXT NOT NULL`
+- `task_id TEXT` (nullable until mapped/enqueued task exists)
+- `status TEXT NOT NULL` (`pending|ready|enqueued|running|completed|failed|blocked|skipped|cancelled`)
+- `attempt_count INTEGER NOT NULL DEFAULT 0`
+- `last_error TEXT`
+- `created_at TEXT NOT NULL`
+- `updated_at TEXT NOT NULL`
+
+Constraints/indexes:
+- `FOREIGN KEY (dag_run_id) REFERENCES dag_run(id) ON DELETE CASCADE`
+- `CREATE INDEX ... ON dag_node(dag_run_id, status)`
+- `CREATE INDEX ... ON dag_node(task_id)`
+
+Rationale:
+- `task_id` nullable to represent not-yet-materialized nodes under enqueue-only adapter.
+- `attempt_count` persisted for retry accounting aligned to FN-4490/FN-4398 retry taxonomy.
+
+### 3) `dag_edge`
+
+Suggested columns:
+- `id TEXT PRIMARY KEY`
+- `dag_run_id TEXT NOT NULL`
+- `from_node_id TEXT NOT NULL`
+- `to_node_id TEXT NOT NULL`
+- `edge_kind TEXT NOT NULL DEFAULT 'depends_on'`
+- `created_at TEXT NOT NULL`
+
+Constraints/indexes:
+- `FOREIGN KEY (dag_run_id) REFERENCES dag_run(id) ON DELETE CASCADE`
+- `FOREIGN KEY (from_node_id) REFERENCES dag_node(id) ON DELETE CASCADE`
+- `FOREIGN KEY (to_node_id) REFERENCES dag_node(id) ON DELETE CASCADE`
+- `UNIQUE(dag_run_id, from_node_id, to_node_id, edge_kind)`
+- `CREATE INDEX ... ON dag_edge(dag_run_id, to_node_id)`
+- `CREATE INDEX ... ON dag_edge(dag_run_id, from_node_id)`
+
+Rationale:
+- explicit run-scoped edge rows support deterministic readiness checks and restart rehydration.
+
+## Migration mechanics
+
+- Add one new migration block at next version slot **78** in `packages/core/src/db.ts` (`if (version < 78) { applyMigration(78, ...) }`).
+- Migration should create new tables/indexes via `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` to stay idempotent.
+- Downgrade policy: forward-only migrations (consistent with current runner). Explicitly no down migration.
+- Multi-project interaction:
+  - `~/.fusion/fusion-central.db`: unchanged.
+  - Every project’s local `.fusion/fusion.db`: independently receives the additive tables when opened.
+
+## Restart recovery contract
+
+For engine boot recovery:
+- Source of truth is SQLite rows in `dag_run` + `dag_node` + `dag_edge`.
+- Recovery should identify non-terminal runs (`pending|running`) and resume readiness evaluation from persisted statuses.
+- Required status invariants:
+  - `dag_run.status` and `dag_node.status` transitions are monotonic toward terminal states.
+  - crashes between transitions are safe because old state remains valid input for retry/re-evaluation.
+  - `task_id` linkage, once set, remains stable for node lifetime.
+
+## Rollback story
+
+Because this milestone is scaffold-only and adapter is enqueue-only:
+- if issues arise, runtime can ignore new DAG tables.
+- scheduler/executor/merger behavior remains unchanged when DAG feature flag is off.
+- no existing table semantics are modified.
+
+## Open questions
+
+1. Should `dag_run.project_id` store canonical project ID or normalized path-derived identity used by central registry APIs?
+2. Do we need a dedicated `terminal_reason` column on `dag_node` vs. deriving from `last_error` + status?
+3. Should `dag_node.task_id` be unique within a run (`UNIQUE(dag_run_id, task_id)` with NULL-safe behavior) to prevent accidental dual binding?
+4. How should cancellation provenance (operator/system/governance) be normalized for restart-safe replay?