docs(plan): step parsing becomes a parse-steps graph node (artifact + registry parser, no-steps/parse-error outcomes)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
gsxdsm
2026-06-04 11:16:02 -07:00
parent 0f15938414
commit 1dd831003d

View File

@@ -45,7 +45,7 @@ The FN-4359 reliability-freeze waiver carried by plans 002/003 continues to appl
- Runtime template expansion with deterministic instance identity, persisted instance run-state (schema v108), crash/resume reconstruction, and a stale-instance recovery sweep.
- `Task.steps[]` kept as the **physical projection sink**: instance transitions call the existing `store.updateStep`, so the merge-blocker, dashboard step display, CLI/TUI, reconcile-from-git, and lost-work reset all keep working unchanged.
- **Parallel step execution** (KTD-11): `TaskStep.dependsOn` metadata parsed from PROMPT.md `### Step N (depends: 1,2):` annotations; foreach `concurrency` config; per-instance worktrees/branches off a common base via the existing worktree pool; ordered integration (rebase/cherry-pick in step order) with `outcome:integration-conflict` routed as rework on the updated base; branch-scoped RETHINK reset in parallel mode.
- **Workflow-defined task artifacts & step source** (KTD-12): workflows declare named task documents (riding the existing task-documents machinery) and which artifact+parser produces the step list; `parseStepsFromPrompt` becomes the built-in `step-headings` parser in a parser registry; the default workflow declares `PROMPT.md` + `step-headings` for byte-identical parity.
- **Workflow-defined task artifacts & a `parse-steps` node** (KTD-12): workflows declare named task documents (riding the existing task-documents machinery), and step parsing is a first-class graph node — `parse-steps(artifact, parser)` reads the artifact and writes the step list, with `no-steps`/`parse-error` outcome edges; `parseStepsFromPrompt` becomes the built-in `step-headings` parser in a parser registry; the default workflow keeps its legacy in-execute step init for byte-identical parity.
- **Workflow-defined custom task fields** (KTD-13): typed field definitions (string/text/number/boolean/enum/multi-enum/date/url) with enum options and rendering instructions in the IR; values stored per task and validated against the schema through a single store authority; agent-tool parity.
- **Dynamic task UI** (KTD-14): TaskDetailModal renders the field schema as a form section; TaskCard renders card-front-placed fields as badges/chips; workflow editor gains a field-definitions panel.
- Built-in **stepwise coding workflow** (new, opt-in) modeling today's per-step review policy explicitly; parity assertions against legacy in-session behavior.
@@ -103,7 +103,7 @@ New node kind `foreach`, config:
- `parallel` + `shared`: **rejected by the validator** — concurrent write sessions in one worktree are unguardable races.
- **Parallel scheduling**: an instance becomes runnable when all of its step's `dependsOn` steps are integrated (KTD-11); up to `concurrency` runnable instances execute concurrently. `TaskStep.dependsOn?: number[]` is parsed from the PROMPT.md annotation `### Step N (depends: 1,2): Title`; a step with no annotation implicitly depends on the previous step, so an unannotated plan is fully sequential regardless of mode — parallelism is opt-in per step by the planner, not asserted globally by the workflow author. Dependency cycles are rejected at expansion with an audited failure.
- Instance identity is deterministic: `<foreachNodeId>#<stepIndex>:<templateNodeId>` — resume can reconstruct the full instance set from `(foreachNodeId, pinned step count)` without persisting the expansion itself.
- **Expansion-placement validation**: the validator (U1) requires that a steps-populating node (the planning seam, or the step-init point) dominates — precedes on all paths — any `foreach` with `source: "task-steps"`. This prevents a silent wrong outcome where a mis-authored graph reaches the foreach before planning, sees zero steps, and merges a task with no step work done.
- **Expansion-placement validation**: the validator (U1) requires that a `parse-steps` node (KTD-12) dominates — precedes on all paths — any `foreach` with `source: "task-steps"`. This prevents a silent wrong outcome where a mis-authored graph reaches the foreach before parsing, sees zero steps, and merges a task with no step work done.
- **Zero steps parsed** (after a dominating planning node) → the foreach immediately traverses its `success` edge (matches today: zero steps = no merge blocker, `task-merge.ts:202`).
- Step count is **pinned at expansion and persisted** (`pinnedStepCount` on the instance rows' run scope, KTD-6). PROMPT.md edits after expansion do not re-expand; `store.updateStep`'s auto-reinit path is bypassed for graph-owned tasks (the projection writes explicit indices). The agent gets a tool-message notice when it edits steps after expansion (implementation detail, U6).
- **Pin vs. git-reconcile on resume**: if resume-time `Task.steps[]` length differs from the persisted pin (re-parse or reconcile changed it), the run does **not** guess — it fails the foreach with an audited `pin-mismatch` outcome, instance rows are cleared, and the task follows the normal graph-failure recovery path (legacy requeue with git-reconciled `steps[]` as truth). U4 tests cover both grow and shrink.
@@ -153,7 +153,7 @@ The foreach/step-review machinery is interpreter functionality, gated by the exi
### KTD-9 — Built-in stepwise coding workflow is the demonstration + parity subject
A second built-in (`builtin-stepwise-coding-workflow-ir.ts`): plan seam → foreach(task-steps){ step-execute → step-review(code) with approve→exit, revise→rework, rethink→rework+reset, rework-exhausted→hold(manual) } → review seam → merge seam. Its observable step-state trajectory for equivalent inputs must match the **legacy step-session path's** trajectory (same `updateStep` sequence, same merge-blocker windows; the step-session path is the deterministic oracle — the agent-paced monolithic path is not deterministically comparable, see U7) — asserted by a characterization-style trajectory comparison, reusing the `workflow-parity.ts` observation machinery rather than inventing new drift tracking.
A second built-in (`builtin-stepwise-coding-workflow-ir.ts`): plan seam → parse-steps(PROMPT.md, step-headings) → foreach(task-steps){ step-execute → step-review(code) with approve→exit, revise→rework, rethink→rework+reset, rework-exhausted→hold(manual) } → review seam → merge seam. Its observable step-state trajectory for equivalent inputs must match the **legacy step-session path's** trajectory (same `updateStep` sequence, same merge-blocker windows; the step-session path is the deterministic oracle — the agent-paced monolithic path is not deterministically comparable, see U7) — asserted by a characterization-style trajectory comparison, reusing the `workflow-parity.ts` observation machinery rather than inventing new drift tracking.
### KTD-10 — Authoring surface: node editor additions, board untouched
@@ -171,14 +171,16 @@ A second built-in (`builtin-stepwise-coding-workflow-ir.ts`): plan seam → fore
- **Projection ordering guard**: `store.updateStep`'s out-of-order-done guard (`store.ts:7592-7610`) assumes index order; graph-source writes (U6's `source: "graph"`) relax it to **dependency order** — a done write is legal when all `dependsOn` steps are done. `currentStep` auto-advance (first non-done scan) is order-agnostic already.
- **Persistence**: instance rows gain `branchName` and `integratedAt`; status adds `awaiting-integration`. Crash under worktree isolation resumes by reconciling rows against branch existence: integrated → done; branch exists, not integrated → re-enter integration queue; branch missing → instance re-runs.
### KTD-12 — Workflow-defined task artifacts & step source
### KTD-12 — Workflow-defined task artifacts & a `parse-steps` node
Today `PROMPT.md` is hardcoded engine law: created by the planning phase, parsed by the fixed regex in `parseStepsFromPrompt` (`store.ts:8534`), and assumed by reconcile, step init, and resume prompts. Inversion:
Today `PROMPT.md` is hardcoded engine law: created by the planning phase, parsed by the fixed regex in `parseStepsFromPrompt` (`store.ts:8534`), and assumed by reconcile, step init, and resume prompts. Inversion — **step parsing is itself a graph node**, visible and reorderable workflow structure:
- **IR gains an `artifacts` declaration**: `artifacts: [{ key, title?, producedBy?: "planning" | "manual", role?: "step-source" | "context" }]`. Artifacts ride the **existing task-documents machinery** (`TaskDocument`, `fn_task_document_write/read`) — no new storage; `PROMPT.md` becomes the default workflow's declared `step-source` artifact backed by its current file location (the document layer already fronts it).
- **Step source is workflow-configured**: foreach `source` widens from the literal `"task-steps"` to `{ artifact: <key>, parser: "step-headings" | "json-steps" }` (with `"task-steps"` kept as an alias for `{artifact: "PROMPT.md", parser: "step-headings"}`). A **built-in parser registry** (same registry posture as traits — built-in-only this round) holds: `step-headings` (the extracted `parseStepsFromPrompt` logic, including the `(depends: …)` annotation from U1 — extraction, not rewrite) and `json-steps` (a structured `[{name, depends?}]` document for workflows that plan in JSON).
- **The planning seam contract**: the workflow's `producedBy: "planning"` artifacts are what the planning seam is told to produce (surfaced in the planning prompt); the engine no longer assumes PROMPT.md by name outside the default workflow's declaration. Reconcile (`reconcileStepsFromGitHistory`) and step init read through the workflow-resolved step source.
- **Parity**: the default workflow's declaration (`PROMPT.md` + `step-headings`) routes through the same extracted parser code path — byte-identical, proven by the existing parse tests running against both the direct call and the registry resolution.
- **New node kind `parse-steps`**, config `{ artifact: <key>, parser: "step-headings" | "json-steps" }`. Its handler reads the artifact at walk time, runs the named parser, and writes the canonical step list through the projection sink (`Task.steps[]` via the store, graph source — same single authority as everything else). Outcomes: `success`, `outcome:no-steps` (parsed cleanly, zero steps — routable, defaults to success), `outcome:parse-error` (malformed artifact — routable, defaults to failure). The node is the *only* graph-side writer of the step list; running a `parse-steps` after a foreach has already expanded trips the KTD-3 pin protection (audited failure), so re-plan loops cannot silently desynchronize an expanded foreach.
- **Parser registry**: a **built-in parser registry** (same registry posture as traits — built-in-only this round) holds `step-headings` (the extracted `parseStepsFromPrompt` logic, including the `(depends: …)` annotation from U1 — extraction, not rewrite) and `json-steps` (a structured `[{name, depends?}]` document for workflows that plan in JSON). The node invokes parsers from the registry; the registry is the extension point when parsers go pluggable later.
- **foreach consumes, never parses**: foreach `source: "task-steps"` just reads `Task.steps[]`; the KTD-3 dominance validation is now concrete — a `parse-steps` node must dominate any `foreach(source:"task-steps")` on all paths (the monolithic default workflow has no foreach and keeps its legacy in-`execute()` step init untouched).
- **The planning seam contract**: the workflow's `producedBy: "planning"` artifacts are what the planning seam is told to produce (surfaced in the planning prompt); the engine no longer assumes PROMPT.md by name outside the default workflow's declaration. Reconcile (`reconcileStepsFromGitHistory`) and resume read through the workflow's parse-steps declaration to know which artifact/parser governs the task.
- **Parity**: the legacy path (`parseStepsFromPrompt` callers in store/executor) delegates to the same extracted `step-headings` parser function — byte-identical, proven by the existing parse tests running against both the direct call and the registry resolution. The stepwise builtin's chain becomes: plan seam → `parse-steps(PROMPT.md, step-headings)` → foreach.
### KTD-13 — Workflow-defined custom task fields
@@ -217,7 +219,7 @@ The task model is recast as: **core fields** (title, description) + **standard m
- R13: Agent tools and plugin SDK expose the new IR types (type-only in SDK); `fn_workflow_create/update` accept the new constructs with the same validation.
- R14: Five lifecycle invariants + lost-work guard trio remain non-configurable and covered by tests on the stepwise path.
- R15: Parallel step execution per KTD-11 — foreach exposes explicit `mode` (sequential|parallel) and `isolation` (shared|worktree) axes (parallel+shared validator-rejected); `dependsOn` parsed from PROMPT.md (unannotated steps depend on the previous step, preserving sequential behavior by default); per-instance worktrees off the integration base; ordered integration flips the projection (done iff integrated); rebase conflicts route `outcome:integration-conflict` to rework on the updated base within the rework budget; concurrency clamped by semaphore availability without deadlock; dependency cycles rejected at expansion.
- R16: Workflows declare task artifacts and the step source (artifact + parser) per KTD-12; the default workflow's PROMPT.md + step-headings declaration is byte-identical to today; reconcile and step init resolve through the workflow's step source.
- R16: Step parsing is a graph node per KTD-12 — `parse-steps(artifact, parser)` is the only graph-side writer of the step list, with routable `no-steps`/`parse-error` outcomes; workflows declare task artifacts; the parser registry's `step-headings` is the extracted `parseStepsFromPrompt`, byte-identical for legacy callers; a parse-steps node must dominate any foreach; parse-steps after foreach expansion trips the pin protection.
- R17: Workflows define custom task fields (typed, enum options, render instructions) per KTD-13; values validated through a single store authority with typed rejections; field removal orphans (never destroys) values; agent tools have full read/write parity.
- R18: Task UI renders the field schema dynamically per KTD-14 — detail form widgets by type, card badges by placement, workflow-editor Fields panel; zero custom fields renders exactly today's UI.
@@ -229,9 +231,9 @@ The task model is recast as: **core fields** (title, description) + **standard m
- **Goal**: Additive WorkflowIr v2 extensions with full validation (R1, R3 validator half, R5 shape, R8 shape, R15 shape).
- **Files**: Modify `packages/core/src/workflow-ir-types.ts`, `packages/core/src/workflow-ir.ts`, `packages/core/src/types.ts` (`TaskStep.dependsOn?: number[]`), `packages/core/src/store.ts` (`parseStepsFromPrompt` regex extension for `### Step N (depends: 1,2): Title` — the current regex `^###\s+Step\s+\d+[^:]*:` breaks on the colon inside the annotation, so the updated regex must parse the annotation explicitly AND remain byte-identical for unannotated headings); tests `packages/core/src/__tests__/workflow-ir.test.ts` (extend), new `packages/core/src/__tests__/workflow-ir-foreach.test.ts`, store parse tests extended.
- **Approach**: Add `foreach` and `step-review` to `WorkflowIrNodeKind`; `WorkflowIrEdge.kind?: "rework"`; foreach `config.template` as inline `{nodes, edges}` validated recursively (single entry/exit, no nested foreach this round, `step-execute` seam legal only here, `split` branches inside templates may contain only read-only nodes — extend `SEAM_FORBIDDEN_IN_BRANCH`). Verdict-routing validation per KTD-4 including the single-writer rule (step-review inside a split is advisory-only); **dominance validation**: a steps-populating node must precede any `foreach(source:"task-steps")` on all paths (KTD-3); rework edges legal only intra-template; `mode`/`isolation`/`concurrency` validation (parallel+shared rejected; concurrency ≥1, cap 8, parallel-mode-only); `V1_NODE_KINDS` untouched so `downgradeIrToV1IfPure` refuses these (KTD-8).
- **Approach**: Add `foreach`, `step-review`, and `parse-steps` to `WorkflowIrNodeKind` (parse-steps config validation: artifact key references a declared artifact, parser in the registry whitelist); `WorkflowIrEdge.kind?: "rework"`; foreach `config.template` as inline `{nodes, edges}` validated recursively (single entry/exit, no nested foreach this round, `step-execute` seam legal only here, `split` branches inside templates may contain only read-only nodes — extend `SEAM_FORBIDDEN_IN_BRANCH`). Verdict-routing validation per KTD-4 including the single-writer rule (step-review inside a split is advisory-only); **dominance validation**: a steps-populating node must precede any `foreach(source:"task-steps")` on all paths (KTD-3); rework edges legal only intra-template; `mode`/`isolation`/`concurrency` validation (parallel+shared rejected; concurrency ≥1, cap 8, parallel-mode-only); `V1_NODE_KINDS` untouched so `downgradeIrToV1IfPure` refuses these (KTD-8).
- **Patterns to follow**: `validateParallelism` / `walkBranchToJoin` (`workflow-ir.ts:92-180`) for region validation; hold-node config validation (`workflow-ir.ts:214-221`).
- **Test scenarios**: parse/validate happy path; template with 0/2 entries rejected; step-execute outside foreach rejected; step-execute inside a split branch rejected; rework edge crossing template boundary rejected; unrouted approve/revise rejected; unrouted rethink defaults to revise target; unrouted rework-exhausted defaults to failure; foreach not dominated by a planning node rejected; verdict-authoring step-review inside a split rejected (advisory-only allowed); maxReworkCycles clamp (0→1? reject; 99→10); parallel+shared isolation rejected; concurrency clamp + concurrency-on-sequential rejected; `(depends: 1,2)` parsed into dependsOn; unannotated headings parse byte-identically to today; malformed depends annotation falls back to plain-name parse; v1 round-trip refusal (`downgradeIrToV1IfPure` returns v2 unchanged); JSON round-trip stability.
- **Test scenarios**: parse/validate happy path; template with 0/2 entries rejected; step-execute outside foreach rejected; step-execute inside a split branch rejected; rework edge crossing template boundary rejected; unrouted approve/revise rejected; unrouted rethink defaults to revise target; unrouted rework-exhausted defaults to failure; foreach not dominated by a parse-steps node rejected; parse-steps referencing an undeclared artifact or unknown parser rejected; verdict-authoring step-review inside a split rejected (advisory-only allowed); maxReworkCycles clamp (0→1? reject; 99→10); parallel+shared isolation rejected; concurrency clamp + concurrency-on-sequential rejected; `(depends: 1,2)` parsed into dependsOn; unannotated headings parse byte-identically to today; malformed depends annotation falls back to plain-name parse; v1 round-trip refusal (`downgradeIrToV1IfPure` returns v2 unchanged); JSON round-trip stability.
- **Verification**: new + extended IR tests green; `pnpm --filter @fusion/core build`.
### U2 — Substrate seams: `runTaskStep` / `resetStepToBaseline`
@@ -291,7 +293,7 @@ The task model is recast as: **core fields** (title, description) + **standard m
- **Files**: Modify `packages/dashboard/app/components/WorkflowNodeEditor.tsx`, `packages/dashboard/app/components/workflow-flow-mapping.ts`, `packages/dashboard/app/components/WorkflowNodeEditor.css` (or sibling component CSS per the extraction convention); tests extend `packages/dashboard/app/components/__tests__/WorkflowNodeEditor.test.tsx`, `workflow-flow-mapping.test.ts`; register any new component test in `packages/dashboard/vitest.config.ts` `qualityAppComponentTests`.
- **Approach** (design decisions committed here, not deferred):
- **Template authoring is inline**: template nodes are always-visible React Flow children of the foreach group node (`parentId` set to the group), no drill-in canvas mode. `flowToIr` partitions nodes by `parentId` — children of a foreach group reassemble into that node's `config.template`; everything else stays top-level. Empty foreach groups render an empty-state hint ("drag a step-execute node here").
- **Palette**: add `foreach` (preset `source:"task-steps"`, auto-populating one `step-execute` child so the group is never confusingly empty) and `step-review` entries to the PALETTE array (`WorkflowNodeEditor.tsx:87`).
- **Palette**: add `foreach` (preset `source:"task-steps"`, auto-populating one `step-execute` child so the group is never confusingly empty), `step-review`, and `parse-steps` (preset artifact `PROMPT.md`, parser `step-headings`; inspector renders an artifact select over the workflow's declared artifacts and a parser select) entries to the PALETTE array (`WorkflowNodeEditor.tsx:87`).
- **Inspector fields**: `foreach` branch renders a `mode` select (sequential|parallel), an `isolation` select (shared|worktree — shared disabled when mode is parallel, matching the validator), a numeric `concurrency` input shown only in parallel mode (min 1, max 8, placeholder 2), and a numeric `maxReworkCycles` input (min 1, max 10, placeholder 3 — mirroring the `maxRetries` input pattern at `WorkflowNodeEditor.tsx:711`); `step-review` branch renders a `type` select (plan|code) and the existing `CustomModelDropdown` (optional, like the gate node's model field).
- **Edge authoring**: selecting an edge whose source is a `step-review` node shows an edge inspector with a condition dropdown (approve/revise/rethink/unavailable — stored as `outcome:<verdict>` conditions, displayed as short labels) and a "rework" toggle (sets `kind:"rework"`). Rework edges render dashed in the accent color with a loop indicator.
- Validation errors from `parseWorkflowIr` surface inline at save (existing pattern); all strings `t("key","Default")`; i18n keys added across the 6 locales via the deep-merge convention (do not let the sync prune dynamic keys — prior incident).
@@ -317,13 +319,13 @@ The task model is recast as: **core fields** (title, description) + **standard m
- **Test scenarios**: each type validates/rejects correctly; enum membership enforced; multi-enum subsets; unknown field id rejected; required default applied at task create under the workflow; field removed → value orphaned not deleted; type change without coerce rejected, with coerce honored; workflow switch keeps same-id values and orphans the rest; zero-fields workflow → writes to custom_fields rejected cleanly; JSON round-trip.
- **Verification**: core suite green; v108 migration test covers the column.
### U12 — Workflow-defined artifacts & step-source parser registry
### U12 — `parse-steps` node, workflow artifacts & parser registry
- **Goal**: KTD-12 (R16). **Execution note: characterization-first** — pin `parseStepsFromPrompt` behavior before extraction.
- **Files**: Modify `packages/core/src/workflow-ir-types.ts` / `workflow-ir.ts` (`artifacts` declaration + step-source config validation), new `packages/core/src/step-parsers.ts` (registry + `step-headings` extraction + `json-steps`), `packages/core/src/store.ts` (step init resolves through workflow step source; `parseStepsFromPrompt` delegates to the registry), `packages/engine/src/executor.ts` (reconcile + planning-prompt artifact contract reads workflow declaration); tests new `packages/core/src/__tests__/step-parsers.test.ts`, executor reconcile tests extended.
- **Approach**: per KTD-12 — registry is built-in-only; `"task-steps"` alias preserved so U1–U10 IR remains valid; default workflow declaration routes through the same extracted code path (parity by construction); planning seam surfaces `producedBy: "planning"` artifact keys in its prompt.
- **Patterns to follow**: trait registry (`trait-registry.ts`) for the registry shape; task-documents machinery for artifact backing.
- **Test scenarios**: step-headings extraction byte-identical on existing fixtures (incl. depends annotations); json-steps parses `[{name, depends}]`; malformed json-steps → audited failure, not crash; foreach resolves a custom artifact source; missing artifact at expansion → dominance-style audited failure; default workflow parity (direct call vs registry resolution identical); reconcile reads through the resolved source.
- **Goal**: KTD-12 (R16) — the `parse-steps` node handler, artifact declarations, and the parser registry. **Execution note: characterization-first** — pin `parseStepsFromPrompt` behavior before extraction.
- **Files**: Modify `packages/core/src/workflow-ir-types.ts` / `workflow-ir.ts` (`artifacts` declaration — parse-steps node-kind validation itself lands in U1), new `packages/core/src/step-parsers.ts` (registry + `step-headings` extraction + `json-steps`), `packages/core/src/store.ts` (`parseStepsFromPrompt` delegates to the registry), `packages/engine/src/workflow-node-handlers.ts` (parse-steps handler: read artifact → run parser → write step list via the graph-source projection path → outcome mapping), `packages/engine/src/executor.ts` (reconcile + planning-prompt artifact contract reads the workflow declaration; pin-protection check for parse-steps after expansion); tests new `packages/core/src/__tests__/step-parsers.test.ts`, new `packages/engine/src/__tests__/workflow-parse-steps.test.ts`, executor reconcile tests extended.
- **Approach**: per KTD-12 — the node is the only graph-side step-list writer; registry is built-in-only; foreach `source: "task-steps"` reads the projection; the default workflow keeps its legacy in-execute step init (untouched code path) while legacy `parseStepsFromPrompt` callers delegate to the same extracted parser (parity by construction); planning seam surfaces `producedBy: "planning"` artifact keys in its prompt.
- **Patterns to follow**: trait registry (`trait-registry.ts`) for the registry shape; `createGateHandler` for handler wiring; task-documents machinery for artifact backing.
- **Test scenarios**: step-headings extraction byte-identical on existing fixtures (incl. depends annotations); json-steps parses `[{name, depends}]`; parse-steps writes steps through the graph-source projection path; `outcome:no-steps` on clean-empty parse routes (defaults success); `outcome:parse-error` on malformed artifact routes (defaults failure), not crash; missing artifact → parse-error; parse-steps after foreach expansion → pin-protection audited failure; default workflow parity (direct call vs registry resolution identical); reconcile reads through the workflow's parse-steps declaration.
- **Verification**: characterization tests green pre/post extraction; core + engine suites green.
### U13 — Dynamic task UI: field rendering + Fields panel