fix(engine): close executor/merger concurrency races and reviewer pause TOCTOU

FN-2910 surfaced concurrent reviewer + merger activity on the same task.
Root cause: asymmetric in-flight guards let an unpause-resume kick off a
fresh executor session while a recovery path was already running, and the
auto-merge handoff fired before the executor's finally block finished
cleanup. This sweeps the surrounding lifecycle paths for similar races and
tightens the reviewer pause gate against TOCTOU through runtime setup.

- Symmetric in-flight tracking across `executing`, `recoveringCompleted`,
  and `resumingUnpaused`; `recoverCompletedTask` bails when any are set.
- Atomic claim of the recovery slot in the completed-task watchdog before
  any awaited work.
- Workflow-rerun bounce returns "bounced" | "skipped-pending" so the
  watchdog can no longer log a false-success retry when the original
  bounce is still mid-flight.
- Self-healing's completed-task scan re-checks executing IDs inside the
  loop instead of trusting a pre-await snapshot.
- 300ms grace period before auto-merge enqueue, giving the executor's
  finally block (session disposal, child cleanup) time to drain and
  eliminating the residual log-overlap symptom from FN-2910. Test uses
  fake timers, no real sleep added.
- New AgentSemaphore.runNested for synchronously nested helper agents
  (reviewers): bumps activeCount for honest observability while bypassing
  the wait queue, preserving forward-progress fairness for the parent at
  low maxConcurrent. Both createReviewStepTool and triage's
  createReviewSpecTool now use it.
- New beforeSpawnSession hook on AgentRuntimeOptions/AgentOptions fired
  inside createFnAgent immediately before createAgentSession, past every
  awaited setup step. Reviewer wires a pause re-check that throws a
  sentinel error converted to UNAVAILABLE, closing the TOCTOU window
  where pause flipped during runtime resolution or resource loading.

All 2887 engine tests pass; engine + core + cli + dashboard + plugin-sdk
+ pi-claude-cli + desktop typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
gsxdsm
2026-04-29 11:26:39 -07:00
parent c6c89a0fdf
commit f1c87c38d4
10 changed files with 342 additions and 90 deletions

View File

@@ -1,7 +1,6 @@
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { ProjectEngine } from "../project-engine.js";
import { runtimeLog } from "../logger.js";
import { aiMergeTask } from "../merger.js";
import { TunnelProcessManager } from "../remote-access/tunnel-process-manager.js";
const mocks = vi.hoisted(() => ({
@@ -1224,11 +1223,18 @@ describe("ProjectEngine swallowed error hardening", () => {
expect(handler).toBeTypeOf("function");
if (!handler) throw new Error("task:moved handler was not registered");
// Auto-merge enqueue runs inside a setTimeout grace period (~300ms) to
// let the executor's finally block complete before the merger starts.
// Use fake timers so the test doesn't actually sleep 300ms.
vi.useFakeTimers();
await handler({
task: { id: "FN-001", column: "in-review" },
to: "in-review",
});
await vi.advanceTimersByTimeAsync(500);
expect(warnSpy).toHaveBeenCalledWith(
expect.stringContaining("Auto-merge: failed to read settings for task:moved on FN-001"),
);