Tighten the executor agent guidance and the fn_run_verification tool guidance so
verification stays scoped to changed files instead of running the full workspace
test suite, which (for a foundational-package edit) reverse-expands across the
whole workspace and stalls the task.
- agent-prompts.ts: remove the "during final integration" blanket permission to
run workspace-wide suites; name the forbidden full-run commands explicitly.
- run-verification-tool.ts: strengthen BOUNDED_VERIFICATION_GUIDANCE to forbid
`pnpm test:full` / `pnpm verify:workspace` / whole-package runs as verification.
Engine + core typecheck pass; run-verification-command and executor-review-verdicts
prompt-assertion suites stay green (93 tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Read hasResumableProgress from latestTask (the store snapshot fetched
at ~9226), not the frozen dispatch-time task param, so a freshly
dispatched task that commits step progress mid-session is preserved on
a hard-cancel teardown — not just the re-dispatch case (greptile P1)
- Add companion regression test for the fresh-task-with-in-session-progress
case, which fails against the stale-task snapshot (greptile P1)
- Reformat the requirement comment block to the canonical FNXC heading
(FNXC:WorkflowLifecycle yyyy-MM-dd-hh:mm:) convention (CodeRabbit)
Fusion-Task-Id: FN-6722
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the engine aborted in-flight work mid-execution and bounced the task
back to `todo`, the single-session teardown cleared the task `branch` and
re-queued without `preserveResumeState` — resetting every step to `pending`
(store.moveTaskInternal resetAllStepsToPending) and dropping the pointer to
the commits already on the task branch. The next dispatch then re-planned
from Step 0 and the committed work was stranded, observed as FN-6722 losing
all its progress and getting stuck in todo.
The teardown now keeps the branch and moves with `preserveResumeState`
whenever the task has resumable step progress, matching the sibling
step-session (executor ~8065) and pause-park (executor ~1826) paths, so
execute() resumes onto the existing branch from the first incomplete step.
The worktree is still removed to free its concurrency slot (FN-6782) — only
the durable pointers (branch + step state) are kept.
Adds a regression test driving the exact single-session catch teardown.
Fusion-Task-Id: FN-6722
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move externalEngines.delete(projectId) to immediately after acquiring the
singleton lock instead of after engine.start() succeeds. If a project was
marked external, the holder exits, acquire succeeds, but start() then throws,
the success-path delete never ran and hasRunningEngine() reported a phantom
engine forever. Added a regression test for the failed-takeover path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Stop reconciliation/startAll/onProjectAccessed from warning every tick for
externally-owned engines: swallow EngineAlreadyRunningError in the outer
catches (it's expected and already logged once in createAndStart)
- Add FNXC:DashboardHealth requirement-trace comments on the externalEngines
field and the dashboard hasRunningEngine health check
- Add regression test: reconciliation stays quiet across ticks for an
externally-owned engine (inner refusal logged once, outer failure suppressed)
- Add regression test: hasDashboardEngine legacy fallback to getAllEngines when
hasRunningEngine is unavailable on the manager
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dashboard's engine-availability health check only counted engines
this process started. A second launch (e.g. `pnpm dev dashboard`
alongside an already-running `fusion`) is correctly refused the
per-machine engine singleton lock, so its engine map stays empty and
the dashboard showed a false "engine not running" banner even though an
engine was live on the machine.
ProjectEngineManager now records projects whose singleton lock is held
by another process (via EngineAlreadyRunningError) and exposes
hasRunningEngine(), which the health endpoint consults so the banner
reflects machine-level truth. Reconciliation still retries so this
process takes over if the other exits, and the "refusing to start" log
fires once per project instead of on every 30s reconciliation tick.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add JSDoc to resolveCustomProviderApiType (docstring coverage + AGENTS.md
jsdoc convention) and convert the inline rationale to FNXC format.
- FNXC-format the test rationale comment.
- Strengthen the regression test per FN-5893 (fix the invariant, not the repro):
- negative assertion that no provider is ever registered with the bare
"anthropic" api key;
- assert every api key passed to registerProvider is one pi-ai's registry
actually registers, so a typo in any arm (not just anthropic) fails here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolveCustomProviderApiType mapped the `anthropic-compatible` provider type
to the api key "anthropic", but pi-ai (@earendil-works/pi-ai) registers the
Anthropic Messages API under "anthropic-messages". Any custom provider
configured as anthropic-compatible selected a model whose `api` did not match
a registered provider, throwing "No API provider registered for api: anthropic"
at stream time (the model registered fine, but failed when a task tried to
stream).
The openai-responses and default (openai-completions) arms already map to real
registry keys and work; only the anthropic arm pointed at an unregistered key.
Extend the existing custom-provider registration test (which covered
openai-compatible and openai-responses but not anthropic-compatible) with a
regression assertion that anthropic-compatible maps to "anthropic-messages".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add fire-time safety guard tests: assert auto-continue aborts when the task
becomes paused, user-paused, moved out of todo, or deleted during the backoff
window (greptile P1 / coderabbit) — previously the guard was untested.
- Split the user-pause/global-pause negative test into it.each so a failure
names the offending case (greptile P2).
- Document that the exhausted-retry fallback leaves the shared graphResumeRetryCount
budget at MAX and that it cross-drains with the transient-resume path (greptile P2).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve conflicts from main's analytics schema additions (plugin
activations, per-model token buckets) against the PR's contract-assertion
type column:
- db.ts: renumber behavioral-verification migration 124 -> 126, bump
SCHEMA_VERSION to 126 so it follows main's migrations 124/125
- core/roadmap tests: adopt main's SCHEMA_VERSION-constant assertions
instead of stale literal 124
- test-quarantine.json: keep all four quarantine entries from both sides
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- step() test helper now carries skillName, so the WS-6 round-trip fixture
actually exercises the INVERSION CONTRACT for skillName (was silently dropped).
- executeWorkflowStep now strips an inherited FUSION_HEADLESS on board runs
(unattended=false), preserving the U3 default-safe invariant — a board step
nested under a headless-env parent could otherwise skip user questions.
Added a regression test for the inherited-env strip.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The stepwise-coding IR had no workflow-step seam node, so a per-task
enabledWorkflowSteps entry (e.g. browser verification) would never execute.
Add the seam node on the success path (steps -> workflow-step -> review,
once post-foreach) and declare browser-verification as an optional step,
matching the coding workflow. Covers the dead-toggle gap with resolver and
engine execution-divergence tests.
When the engine hard-cancels in-flight work during a pause/resume cycle and the
workflow graph run ends with the task re-queued to `todo`, the executor used to
leave it for a fresh scheduler dispatch and fire a spurious failure
notification. It now continues the agent session in place via a bounded internal
retry (reusing the graph-resume retry budget + backoff), falling back to the
benign todo re-queue only after retries are exhausted.
- Scoped strictly to engine-internal aborts via a typed `isEngineInternalAbort`
discriminant; genuine user/global/task pauses are never auto-resumed.
- Re-checks the task at retry fire time and aborts the auto-continue if it was
paused, moved, or deleted during the backoff window.
- Clears any stale `failed` status and emits an `Auto-recovered:` log so the
deferred failure notification is suppressed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address confirmed code-review findings on the CE workflow-step change:
- Graph-path spawn lifecycle (adversarial A-1/A-2): the graph path returns from
execute() before its outer finally that calls terminateAllChildren, so U8's new
coding-mode fn_spawn_agent children orphaned their sessions/worktrees and their
ids accumulated in the per-parent spawn budget, starving later steps' fan-out.
Call terminateAllChildren in maybeExecuteWorkflowGraph's finally (mirrors the
non-graph cleanup).
- INVERSION CONTRACT parity (api-contract AC-2 + testing TF-001): add skillName to
the workflow-steps-to-ir round-trip projections + a skill-step fixture, so the
contract the comment claims is actually asserted.
- Silent skill-load degradation (adversarial A-3 / Risk-4): warn when a step names
a skill but FUSION_CE_SKILLS_DIR is unset, instead of failing silent.
- Dead branch (maintainability M-01): drop the always-false unattendedRun guard;
keep the delete + extension-point comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two engine tests for the new compound-engineering workflow-step wiring:
- conventions: assert the exported preamble carries the await-input sentinel,
FUSION_HEADLESS degrade, and path-confined persona/systemPromptOverride fan-out.
- executor: drive runGraphCustomNode + executeWorkflowStep and assert skillName is
carried onto the synthesized step, requestedSkillNames merges bare+namespaced with
additionalSkillPaths=[FUSION_CE_SKILLS_DIR], fn_spawn_agent present only in coding,
FUSION_HEADLESS only when unattended, and the verdict-JSON contract is required
only for gate/skill-less steps (relaxed for non-gate skill steps).
Session layer is mocked (asserts engine-owned wiring, not a model run); a full
model-driven e2e remains a documented residual.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The builtin compound-engineering workflow runs via runGraphCustomNode, which
never loaded the named skill or threaded the plugin-injected runtime env, and
fn_spawn_agent was registered only in the main session. This wires the real seam:
- U8: thread injected FUSION_CE_* env into skill/model graph steps (shared
buildInjectedRuntimeEnv helper); register createSpawnAgentTool for coding-mode
skill steps (readonly still strips spawn).
- U1: merge the step's skillName (namespaced + bare) into requestedSkillNames and
pass FUSION_CE_SKILLS_DIR as additionalSkillPaths so the bundled SKILL.md is
discovered and selected.
- U2: prepend the Fusion workflow-step conventions preamble (await-input sentinel,
FUSION_HEADLESS degrade, persona fan-out via systemPromptOverride).
- U3: explicit unattended opt-in sets FUSION_HEADLESS=1 (default-safe board run).
- U9: path-confined persona read documented in the preamble; accepted
write-capability posture documented at the coding-mode tool registration.
- KTD-6: verdict-JSON contract required only for gate / skill-less steps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Reliability (P3): emit an `Auto-recovered:`-prefixed log on the
benign-todo stale-failure clear path so NotificationService
proactively cancels the pending failure timer (recoveredStatus path)
instead of relying only on the fire-time re-check, which is
race-contingent when failureNotificationDelayMs is near 0. Scoped to
the actual-clear path so the common no-failure re-queue isn't
mislabeled as a recovery.
- Project-standards (P3): add the required yyyy-MM-dd-hh:mm stamp to the
new FNXC comments (AGENTS.md FNXC_LOG convention).
- Maintainability (P3): extract the scheduler "queued" waiting marker to
a named SCHEDULER_WAITING_STATUS constant.
- Testing: pin the guard's skip on a clean todo row, assert the
Auto-recovered log fires on the stale-failure path, and add a
paused+unassigned in-progress idle case (paused early-return wins over
the ephemeral active-session path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A pause/resume abort parked status:"failed" on an earlier non-todo
observation stays dispatchable (scheduler filters on column+paused, not
status, scheduler.ts:1288) and re-enters the FN-6782 benign-todo branch.
That branch logged "benign" but left status:"failed"/error on the row,
so the board kept showing it failed and the deferred failure
notification fired (notification-service fire-time check sees
status==="failed") — contradicting the benign log. recoverPausedAbortFailures
that would clear it is suppressed during global/engine pause
(self-healing.ts:8125), so the failure survived the pause window.
Reconcile the row with the benign reclassification: clear status/error
when the live row carries them, so the board agrees it's benign and the
pending notification is suppressed at dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses findings from a multi-agent review of the two prior fixes.
P0 (executor.ts): the stale-conflict recovery force-removed worktreePath with
no bounds check; that path can come from a git admin entry resolving outside
.worktrees/. Now refuses unless the path is inside the worktrees dir, not a
symlink (realpathSync), not a registered worktree, and not actively owned, and
re-verifies liveness in the catch instead of trusting the error string. Also
excludes spawn failures (spawn git ENOENT) from the stale-path classification.
worktree-pool.ts: resolveGitdirPointer -> dotGitPointerIsDangling. Reaps only
when a .git link's gitdir target is confirmed missing; a real .git dir,
unparseable pointer, or any read/stat failure is treated as NOT dangling
(conservative) so a transient read error on a live worktree can't trigger rm.
Drops the string|"directory"|null sentinel union.
core store.ts: bypass the reconcile recency window when the live task table is
empty (corruption/restore: surviving task.json keep old mtimes) and when
fusion.db was auto-recovered on startup, so .recover row loss isn't stranded.
Adds an ignoreRecencyWindow option.
Tests: executor recovery + out-of-bounds refusal, unparseable .git skip,
recency boundary, empty-DB/forced bypass. engine 135 + core 12 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Directories under .worktrees/ that survive with a dangling .git pointer
(present on disk, but their .git/worktrees/<name> admin entry is gone) are
invisible to `git worktree list`/`prune` yet collide with freshly generated
worktree names. The executor's conflict cleanup then fails with
"is not a working tree", failing the workflow graph at node 'execute' after
3 attempts.
- executor.ts: extend FN-4813 stale-conflict recovery to also treat
"is not a working tree" and ENOENT (not just "validation failed, cannot
remove working tree") as "no live worktree here" — prune the admin entry,
force-remove the leftover dir, and proceed with fresh creation.
- worktree-pool.ts: reapOrphanWorktrees skipped any dir on mere .git-file
presence, contradicting its own documented invariant. Resolve the .git
pointer and only skip when the gitdir target exists; reap dangling
pointers like any other orphan so they stop accumulating across runs.
- Tests for both the dangling (reaped) and valid (skipped) .git cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents stop heartbeating during long legitimate work (e.g. a verification
step blocked on a multi-minute test command). The 5-minute floor could
misread a busy agent as dead and reclaim its in-progress task mid-run.
Raise MIN_HEARTBEAT_STALENESS_MS to 10 minutes and strengthen the floor
test (7-minute-silent fast-interval agent stays healthy — would have read
stale under the old 5-minute floor). Engine typecheck clean; heartbeat
suite 148/148 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gate test asserted the OLD behavior — a paused graph exit in the `todo`
column parked `status:"failed"` with "operator action required". FN-6782
made the todo case benign (no failed park; benign log + cleared marker), so
split the parameterized test: `todo` now asserts the benign path (never
parked failed), `done` keeps the operator-action surfacing (log only, no
park). Full engine-core gate suite passes (644/644).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
coderabbit Major: reapLeakedConcurrencySlots captured executingIds once
before the loop, but each holder awaits getTask — a task could start
executing mid-sweep and have its worktree slot pulled. Refresh the
executing set immediately before clearPhantomExecutorBinding and skip if
the holder is now executing (same race the A1 recovery fix closed).
clearPhantomExecutorBinding's live-session refusal remains the last line
of defense; this avoids racing it. Added a mid-sweep race test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Substantive (A1 recoverPausedAbortFailures):
- Self-guard on globalPause/enginePaused at method entry (greptile P1) — the
public method must not requeue tasks an operator intentionally froze.
- Re-validate the FULL predicate with a FRESH executing set on the re-read
before the backward move (coderabbit Major + greptile): add fresh.userPaused
and column re-check so a task that became ineligible across awaits is skipped.
- Isolate audit emission in its own try/catch (coderabbit) so an audit throw
after a successful mutation can't log a false "recovery failed".
- Decouple the recovery predicate from the literal error text via shared
PAUSE_ABORT_PARK_ERROR_MARKER/OPERATOR_MARKER constants (greptile) — the
executor builds the parked message from the same constants.
- Use the wired clearPhantomExecutorBinding (live-session-guarded) instead of
the declared-but-never-wired releaseExecutorWorktreeOwnership, which no-op'd.
Nits:
- FNXC-prefix new comments in executor.ts, run-audit.ts, and the benign test
per repo comment policy.
- Fix a test-only type error on the clearPhantomExecutorBinding mock.
Added a test asserting the globalPause self-guard. Engine typecheck clean;
pause-abort/reaper/benign + regression suites pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
reapLeakedConcurrencySlots() reclaims in-memory worktree slots whose
holder is no longer in-progress (the FN-6756 "in todo yet still a
maxWorktrees holder" leak) without an engine restart — defense-in-depth
behind the source fix.
- executor: new listWorktreeHolders() read-only introspection over
activeWorktrees; wired through in-process-runtime to SelfHealingManager.
- reaper releases ONLY when every guard agrees: not executing, task
missing or in todo/triage, past a 60s grace, and clearPhantomExecutor
Binding itself refuses (returns false) if a live session surface is
registered — so it can never pull a worktree from a running agent.
- registered in maintenance batch 2 (respects globalPause/enginePaused
skip + FN-4962 ordering).
- widened the clearPhantomExecutorBinding option type to surface its
boolean refusal signal.
Engine typecheck clean; 19 tests pass (new reaper 7 cases + regression).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A global pause/resume cycle parked tasks that had re-queued to todo as
status:"failed" ("operator action required") and leaked their in-memory
worktree slot. The scheduler kept re-dispatching the todo task, the
genuine-pause-abort branch re-fired on the still-set pausedAborted marker,
and it re-parked instantly with no backoff — a retry storm (75x/hr) that
pinned maxWorktrees=3/3 and concurrency-starved the whole queue.
- R1+R2 (executor.ts handleGraphFailure): treat a pause-abort that left a
task in `todo` as benign (FN-6782) — don't park failed, clear the
pausedAborted marker so the next dispatch is clean, and release the
leaked activeWorktrees slot. Operator-action failure preserved for
genuinely stranded non-todo columns (FN-6478).
- A1 (self-healing.ts recoverPausedAbortFailures): new maintenance sweep
that auto-recovers any pause-abort park still on the board and requeues
it (status:null = schedulable) so the board self-heals.
- run-audit.ts: new mutation types for the recovery telemetry.
Corrected the spec's null-vs-queued assumption: the scheduler dispatch set
is column==="todo" && !paused (scheduler.ts:1288); status:"queued" is the
*blocked* marker, status:null is runnable — so recovered tasks are left null.
Deferred (documented): A2 leaked-slot reaper needs a new executor
listWorktreeHolders introspection API to reap in-memory worktree slots
safely; R1 closes the observed leak at its source.
Tests: self-healing-paused-abort-recovery.test.ts (3),
executor-paused-abort-todo-benign.test.ts (2). Engine typecheck clean;
106 existing pause/graph-failure/limbo tests still pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reorder backend/timeoutMsOverride params so timeoutMsOverride remains at
position 10 (matching callers in executor.ts, merger.ts from main).
Update mission-verification.ts call sites to pass undefined for
timeoutMsOverride before the isolating backend argument.
Recover wedged in-progress tasks by clearing stale executor bindings only after liveness proves the owner is gone.
- Add a guarded executor escape hatch that clears only stale in-memory task bookkeeping while refusing live session surfaces.
- Teach self-healing to identify phantom executor-active bindings using age, checkout, heartbeat, run-audit, and worktree liveness signals before requeueing preserved work.
- Record reclaim events in run audit and cover preserved-worktree recovery with reliability interaction tests.
- Document the recovery path and add a patch changeset for the published CLI package.
Files changed:
.changeset/fn-6736-phantom-executor-binding.md | 5 +
AGENTS.md | 1 +
docs/architecture.md | 1 +
.../reclaim-phantom-executor-binding.test.ts | 244 +++++++++++++++++++++
packages/engine/src/executor.ts | 35 +++
packages/engine/src/run-audit.ts | 2 +
packages/engine/src/runtimes/in-process-runtime.ts | 3 +-
packages/engine/src/self-healing.ts | 112 ++++++++++
8 files changed, 402 insertions(+), 1 deletion(-)
Fusion-Task-Id: FN-6736
Fusion-Task-Lineage: c76191ba-f4c3-4832-a790-67676e258ba2
