docs: capture engine singleton-lock learning (EngineAlreadyRunningError != no engine)

Document the false "engine not running" banner root cause and fix as a
docs/solutions learning, and add an "Engine Singleton Lock" entry to
CONCEPTS.md: a failed per-machine lock acquisition is proof an engine is
running elsewhere, not "no engine."

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CONCEPTS.md

@@ -82,6 +82,9 @@ The core board entity: a unit of work that moves through columns (triage, todo,
 ### Workflow Runtime
 The authoritative task lifecycle runtime. It resolves a Task to workflow IR, walks the graph, routes node outcomes, and invokes runtime primitives for side effects. The engine substrate still owns scheduling, routing claims, persistence, concurrency, process supervision, storage, and audit plumbing; lifecycle policy lives in workflow nodes and built-in workflow IR.
 
+### Engine Singleton Lock
+A per-machine mutual-exclusion guard ensuring only one fusion process runs the engine for a given project, combining a lockfile in the project's `.fusion/` directory with a per-project loopback socket. Failure to acquire it (`EngineAlreadyRunningError`) is **positive proof an engine is already running** for that project elsewhere on the machine — not an error to swallow and not "no engine." A process refused the lock keeps that as a fact: it reports the engine as available (so UI surfaces don't claim it's down) while reconciliation keeps retrying, so it takes over if the current owner exits.
+
 ### ACP Ask Path
 A one-turn read-only model ask routed through the ACP runtime rather than a CLI print mode. The runner accumulates streamed prose, may recover a trailing JSON object for structured seams, and treats abnormal ACP stop reasons as incomplete answers for validator use.
 

docs/solutions/integration-issues/engine-already-running-is-not-no-engine.md

@@ -0,0 +1,119 @@
+---
+title: "EngineAlreadyRunningError means an engine IS running, not 'no engine'"
+date: "2026-06-21"
+category: integration-issues
+module: "packages/engine project-engine-manager + packages/dashboard server health"
+problem_type: integration_issue
+component: engine_runtime
+symptoms:
+  - "Dashboard shows a persistent 'engine not running' banner even though an engine is running"
+  - "Logs repeat 'Refusing to start engine for <projectId>: Another engine is already running...' every 30s reconciliation tick"
+  - "Reconciliation tries to start engines for N projects, then errors because each is already running"
+root_cause: incorrect_state_derivation
+resolution_type: code_fix
+severity: medium
+related_components:
+  - engine
+  - dashboard
+  - development_workflow
+tags:
+  - engine-singleton-lock
+  - multi-process
+  - reconciliation
+  - dashboard-health
+  - regression
+related_prs:
+  - "https://github.com/Runfusion/Fusion/pull/1704"
+---
+
+## Problem
+
+Starting a second fusion process (e.g. `pnpm dev dashboard --no-auth` while an installed `pnpm fusion` is already running) showed a false **"engine not running"** banner — even though an engine was live on the machine for every project.
+
+## Symptoms
+
+- Banner visible despite a running engine.
+- Repeating log line every reconciliation tick: `Refusing to start engine for <projectId>: Another engine is already running for project <projectId> on this machine (blocked by socket)`.
+- The collision affects every registered project (the log "tries to start engines for two projects then refuses" maps 1:1 to the live `fusion-engine-<hash>.sock` files under `$TMPDIR`).
+
+## What Didn't Work
+
+- Looking only at the dashboard frontend / banner component — the banner faithfully renders `dashboardHealth.engine.available === false`; the wrong value comes from the server.
+- Treating it as a warm-up race (engines still starting in the background). The banner is *permanent*, not transient: reconciliation retries forever and keeps refusing, because the lock is held by a **different process** that never exits.
+
+## Root Cause
+
+The engine uses a **per-machine singleton lock** (`packages/engine/src/engine-singleton-lock.ts`): a lockfile under `<workingDir>/.fusion/engine.lock` plus a loopback socket `fusion-engine-<sha1(projectId)>.sock`. Only one fusion process may own the engine for a given project. When a second process launches, `acquireEngineSingleton` correctly throws `EngineAlreadyRunningError` — **positive proof an engine is live** for that project.
+
+But that proof was discarded. In `ProjectEngineManager.createAndStart` the catch logged "Refusing to start" and rethrew **without recording that an engine exists**. The dashboard's health check `hasDashboardEngine` (`packages/dashboard/src/server.ts`) then only counted engines *this* process owns:
+
+```ts
+// before
+function hasDashboardEngine(options?: ServerOptions): boolean {
+  const engines = options?.engineManager?.getAllEngines?.();
+  return Boolean(options?.engine || (engines && engines.size > 0));
+}
+```
+
+So `getAllEngines()` was empty → `/api/health` reported `engine.available: false` → banner. Three views of "is an engine running" disagreed: the singleton lock (machine truth: *yes*), reconciliation's `has()` (this-process-owns: *no*), and the banner (this-process-owns: *no*).
+
+Surfaced by `feat: start engines by default` (#1699), which made the dashboard actively attempt engine startup instead of running UI-only.
+
+## Solution
+
+Track engines owned by **another** process and report them as available, while still retrying so this process can take over if the other dies.
+
+```ts
+// ProjectEngineManager
+private externalEngines = new Set<string>();
+
+hasRunningEngine(): boolean {
+  return this.engines.size > 0 || this.externalEngines.size > 0;
+}
+
+// in createAndStart's acquireEngineSingleton catch:
+if (err instanceof EngineAlreadyRunningError) {
+  if (!this.externalEngines.has(projectId)) {       // log once, not every tick
+    runtimeLog.warn(`Refusing to start engine for ${projectId}: ${err.message}`);
+  }
+  this.externalEngines.add(projectId);
+}
+throw err;
+
+// on successful (own) start: this.externalEngines.delete(projectId);
+// in stopAll(): this.externalEngines.clear();
+```
+
+```ts
+// dashboard server.ts — consult machine-level truth, fall back for old managers/test doubles
+function hasDashboardEngine(options?: ServerOptions): boolean {
+  if (options?.engine) return true;
+  const manager = options?.engineManager;
+  if (!manager) return false;
+  if (typeof manager.hasRunningEngine === "function") return manager.hasRunningEngine();
+  const engines = manager.getAllEngines?.();
+  return Boolean(engines && engines.size > 0);
+}
+```
+
+Reconciliation deliberately keeps retrying (it does NOT treat external engines as `has()`), so when the lock-holder exits this process acquires the lock and clears the external marker — verified live: killing the holder let the dev dashboard immediately adopt the socket.
+
+## Why This Works
+
+`EngineAlreadyRunningError` is the *only* signal that distinguishes "no engine anywhere" from "an engine runs in another process." Capturing it makes the dashboard report machine-level reality instead of process-local ownership. Automation still runs because the owning process's engine polls the shared DB; the second process is a correct UI-over-the-same-store.
+
+## Prevention
+
+- **Treat lock-acquisition failure as evidence, not an error to swallow.** A failed mutual-exclusion acquire (lock/socket/PID) usually proves the guarded resource *is* alive elsewhere — don't let downstream "is it running?" checks read process-local state instead of the lock's machine-level truth.
+- Tests added (had zero prior coverage — they assumed the manager always owns what it starts):
+  - `packages/engine/src/__tests__/project-engine-manager.test.ts`: reports a running engine when the lock is held elsewhere; logs the refusal once across repeated attempts; takes over and clears the marker once the lock frees.
+  - `packages/dashboard/src/__tests__/server.test.ts`: `/api/health` reports `available: true` when another process owns the engine.
+
+### Operational gotcha
+
+`pnpm fusion` launches the full dashboard + engine and **never exits**. Piping it (e.g. `pnpm fusion 2>&1 | grep -i auth` to inspect auth output) leaves the server running in the background, orphaned, holding the engine singleton sockets indefinitely. To find/clear a stray holder:
+
+```bash
+lsof "$TMPDIR"/fusion-engine-*.sock   # PID owning the engine locks
+kill <pid>                            # releases the locks; reconciliation in any waiting process takes over
+```