FN-8693: refresh reused worktree bases before execution
Refresh reused execution worktrees against the current integration baseline. - Rebase or reset clean reused worktrees before coding sessions while preserving task commits. - Persist and audit refreshed base SHAs, and block unsafe refresh states before execution. - Cover executor, graph, and heartbeat refresh paths with regression tests. Files changed: .changeset/fn-8693-stale-worktree-base.md | 7 + docs/architecture.md | 1 + .../src/__tests__/agent-heartbeat-worktree.test.ts | 28 ++++ .../__tests__/ce-workflow-step-executor.test.ts | 44 ++++++ .../src/__tests__/worktree-base-refresh.test.ts | 90 ++++++++++++ packages/engine/src/agent-heartbeat.ts | 35 ++++- packages/engine/src/executor.ts | 67 ++++++++- packages/engine/src/merger.ts | 5 + packages/engine/src/run-audit.ts | 11 ++ packages/engine/src/workflow-graph-executor.ts | 32 ++++- packages/engine/src/worktree-acquisition.ts | 28 +++- packages/engine/src/worktree-base-refresh.ts | 158 +++++++++++++++++++++ 12 files changed, 498 insertions(+), 8 deletions(-) Fusion-Task-Id: FN-8693 Fusion-Task-Lineage: e39a441f-39b5-4723-b503-753e921018f3 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
7
.changeset/fn-8693-stale-worktree-base.md
Normal file
7
.changeset/fn-8693-stale-worktree-base.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
summary: Refresh reused execution worktrees against merged dependency changes.
|
||||
category: fix
|
||||
dev: Execution refresh persists the integration baseline while preserving rebased task commits.
|
||||
@@ -606,6 +606,7 @@ See [Memory Plugin Contract](./memory-plugin-contract.md) for the full plan.
|
||||
### Agent roles
|
||||
- **Planning**: the planning processor generates task plans (`PROMPT.md`) and selects eligible planning tasks by priority first, then FIFO (`createdAt` ascending) within each priority tier. If the stuck-task detector kills a not-yet-approved planning session after a non-empty `PROMPT.md` draft exists, the retry is requeued as `needs-replan` and seeds the next prompt in revision mode from that draft instead of cold-starting. When `PROMPT.md` is absent, a non-empty `plan` task document written through `fn_task_document_write` is the fallback seed; missing or whitespace-only drafts still cold-start.
|
||||
- **Executor**: `TaskExecutor` (`executor.ts`) implements tasks in worktrees
|
||||
- **Execution-only reused-base refresh (FN-8693):** planning creates isolated worktrees but does not refresh them; immediately before a graph `code` node, normal executor dispatch, or durable-agent heartbeat session, refresh-enabled reuse resolves the current integration target C1 and compares it with durable `task.baseCommitSha`. A clean no-own-commit checkout resets to C1; a clean own-commit checkout rebases and retains its resulting C2 `HEAD`, while storing C1—not C2—as the baseline. A durable C0/C1 mismatch is rechecked from git and durable metadata on every acquisition, so restart reconciliation needs no in-memory marker. Dirty, unresolved, unsupported worktrunk, git, conflict, persistence, and unprovable-reconciliation cases are typed non-execution outcomes that park before session start. If baseline persistence fails after git moves `HEAD`, the engine compensates to the original clean checkout and emits `worktree:base-refresh-persistence-failed-compensated`; otherwise it requires later proof-based reconciliation. Audit events are `worktree:base-refreshed`, `worktree:base-refresh-blocked`, `worktree:base-refresh-conflict`, `worktree:base-refresh-persistence-failed-compensated`, and `worktree:base-refresh-reconciled`. Plan/review/gate acquisition and merger acquisition remain excluded; merger owns its separate auto-prerebase policy.
|
||||
- **Reviewer**: `reviewStep()` (`reviewer.ts`) performs plan/code/spec reviews
|
||||
- **Merger**: `aiMergeTask()` (`merger.ts`) merges approved work
|
||||
- **Task-detail chat / steering comments**: `TaskStore.addSteeringComment()` writes chat steering text to both `task.comments` and `task.steeringComments`. The executor still uses `steeringComments` for live in-session injection, while next-prompt agent lanes read canonical user-authored `task.comments`: planning/spec generation, spec review, plan/code reviewers, standard merger prompts, and clean-room AI merge + merge-review prompts all surface recent user comments through the shared `agent-user-comments.ts` formatter.
|
||||
|
||||
@@ -75,6 +75,34 @@ describe("heartbeat worktree cwd", () => {
|
||||
expect(taskStore.updateTask).toHaveBeenCalledWith("FN-1", { recoveryRetryCount: 1 });
|
||||
});
|
||||
|
||||
it("parks typed base-refresh refusals without consuming acquisition retries", async () => {
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:33:
|
||||
A stale checkout is a deliberate no-session outcome. It must retain the concrete reason for
|
||||
the next heartbeat rather than converting into the unrelated acquisition retry cap.
|
||||
*/
|
||||
vi.spyOn(worktreeAcquisition, "acquireTaskWorktree").mockRejectedValueOnce(
|
||||
new worktreeAcquisition.WorktreeBaseRefreshError({
|
||||
kind: "base-reconciliation-required",
|
||||
executionSafe: false,
|
||||
durableBaseSha: "c0",
|
||||
baseSha: "c1",
|
||||
}),
|
||||
);
|
||||
const monitor = new HeartbeatMonitor({ store, taskStore, rootDir: "/repo" });
|
||||
|
||||
await monitor.executeHeartbeat({ agentId: "a1", source: "on_demand" });
|
||||
|
||||
expect(piModule.createFnAgent).not.toHaveBeenCalled();
|
||||
expect(taskStore.updateTask).not.toHaveBeenCalledWith("FN-1", expect.objectContaining({ recoveryRetryCount: expect.anything() }));
|
||||
expect(taskStore.logEntry).toHaveBeenCalledWith(
|
||||
"FN-1",
|
||||
"Worktree base refresh blocked heartbeat execution (base-reconciliation-required)",
|
||||
expect.any(String),
|
||||
);
|
||||
expect(taskStore.moveTask).toHaveBeenCalledWith("FN-1", "todo", { preserveProgress: true });
|
||||
});
|
||||
|
||||
// FN-7721 regression: reproduces the reported "worktree-setup loop" symptom
|
||||
// (identical `git worktree add -b <branch>` failure repeated indefinitely
|
||||
// across heartbeat cycles, ~16.2h in the reported incident) and asserts the
|
||||
|
||||
@@ -32,6 +32,7 @@ import "./executor-test-helpers.js";
|
||||
import { TaskExecutor } from "../executor.js";
|
||||
import type { PluginRunner } from "../plugin-runner.js";
|
||||
import { WorkflowGraphExecutor } from "../workflow-graph-executor.js";
|
||||
import { WorktreeBaseRefreshError } from "../worktree-acquisition.js";
|
||||
import {
|
||||
createMockStore,
|
||||
mockedCreateFnAgent,
|
||||
@@ -40,6 +41,49 @@ import {
|
||||
resetExecutorMocks,
|
||||
} from "./executor-test-helpers.js";
|
||||
|
||||
describe("typed worktree base refresh graph refusal", () => {
|
||||
it("does not immediately retry or erase a code-node refresh reason", async () => {
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:33:
|
||||
The graph must stop before its code handler/session when reuse cannot prove a current,
|
||||
durable-aligned checkout. The refresh outcome remains routable rather than generic exception.
|
||||
*/
|
||||
const handler = vi.fn();
|
||||
const prepare = vi.fn().mockRejectedValue(new WorktreeBaseRefreshError({
|
||||
kind: "base-reconciliation-required",
|
||||
executionSafe: false,
|
||||
durableBaseSha: "c0",
|
||||
baseSha: "c1",
|
||||
}));
|
||||
const graph = new WorkflowGraphExecutor({
|
||||
handlers: { code: handler },
|
||||
prepareNodeExecution: prepare,
|
||||
maxRetriesPerNode: 3,
|
||||
});
|
||||
const ir: WorkflowIr = {
|
||||
version: "v2",
|
||||
name: "typed-refresh-refusal",
|
||||
columns: [{ id: "in-progress", name: "In Progress", traits: [] }],
|
||||
nodes: [
|
||||
{ id: "start", kind: "start" },
|
||||
{ id: "execute", kind: "code", column: "in-progress", config: { source: "return {};" } },
|
||||
{ id: "end", kind: "end" },
|
||||
],
|
||||
edges: [
|
||||
{ from: "start", to: "execute" },
|
||||
{ from: "execute", to: "end", condition: "success" },
|
||||
],
|
||||
};
|
||||
|
||||
const result = await graph.run({ id: "FN-REFRESH", column: "in-progress", steps: [] } as any, {}, ir);
|
||||
|
||||
expect(prepare).toHaveBeenCalledTimes(1);
|
||||
expect(handler).not.toHaveBeenCalled();
|
||||
expect(result.outcome).toBe("failure");
|
||||
expect(result.context?.["node:execute:value"]).toBe("base-reconciliation-required");
|
||||
});
|
||||
});
|
||||
|
||||
type CapturedSession = {
|
||||
customTools?: Array<{ name?: string }>;
|
||||
systemPrompt?: string;
|
||||
|
||||
90
packages/engine/src/__tests__/worktree-base-refresh.test.ts
Normal file
90
packages/engine/src/__tests__/worktree-base-refresh.test.ts
Normal file
@@ -0,0 +1,90 @@
|
||||
import { execSync } from "node:child_process";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { refreshReusedWorktreeBase } from "../worktree-base-refresh.js";
|
||||
|
||||
const paths: string[] = [];
|
||||
const git = (cwd: string, command: string) => execSync(`git ${command}`, { cwd, encoding: "utf8" }).trim();
|
||||
function fixture() {
|
||||
const root = mkdtempSync(join(tmpdir(), "fn-8693-base-"));
|
||||
paths.push(root);
|
||||
git(root, "init -b main");
|
||||
git(root, 'config user.email "test@example.com"');
|
||||
git(root, 'config user.name "Test"');
|
||||
writeFileSync(join(root, "README.md"), "C0\n");
|
||||
git(root, "add README.md && git commit -m C0");
|
||||
const c0 = git(root, "rev-parse HEAD");
|
||||
const worktree = join(root, "task");
|
||||
git(root, `worktree add -b fusion/fn-1 ${JSON.stringify(worktree)} ${c0}`);
|
||||
writeFileSync(join(root, "scaffold.ts"), "export const scaffold = true;\n");
|
||||
git(root, "add scaffold.ts && git commit -m C1");
|
||||
return { root, worktree, c0, c1: git(root, "rev-parse HEAD") };
|
||||
}
|
||||
afterEach(() => paths.splice(0).forEach((path) => rmSync(path, { recursive: true, force: true })));
|
||||
|
||||
/*
|
||||
FNXC:WorktreeBaseRefreshTests 2026-08-01-16:04:
|
||||
These temp-git fixtures reproduce a dependency task planned at C0 while main advances to C1.
|
||||
They prove execution refresh stores integration C1, never stale C0, before a session caller may proceed.
|
||||
*/
|
||||
describe("refreshReusedWorktreeBase", () => {
|
||||
it("resets a clean planning branch to C1 and persists C1", async () => {
|
||||
const { root, worktree, c0, c1 } = fixture();
|
||||
const store = { updateTask: vi.fn().mockResolvedValue(undefined) } as any;
|
||||
const result = await refreshReusedWorktreeBase({
|
||||
task: { id: "FN-1", baseCommitSha: c0 } as any, rootDir: root, worktreePath: worktree, store, settings: {},
|
||||
});
|
||||
expect(result).toMatchObject({ kind: "reset-to-base", executionSafe: true, baseSha: c1 });
|
||||
expect(git(worktree, "rev-parse HEAD")).toBe(c1);
|
||||
expect(store.updateTask).toHaveBeenCalledWith("FN-1", { baseCommitSha: c1 });
|
||||
});
|
||||
|
||||
it("rebases own commit C2 onto C1 while storing C1, not C2", async () => {
|
||||
const { root, worktree, c0, c1 } = fixture();
|
||||
writeFileSync(join(worktree, "implementation.ts"), "export const implementation = true;\n");
|
||||
git(worktree, "add implementation.ts && git commit -m C2");
|
||||
const store = { updateTask: vi.fn().mockResolvedValue(undefined) } as any;
|
||||
const result = await refreshReusedWorktreeBase({
|
||||
task: { id: "FN-1", baseCommitSha: c0 } as any, rootDir: root, worktreePath: worktree, store, settings: {},
|
||||
});
|
||||
const c2 = git(worktree, "rev-parse HEAD");
|
||||
expect(result).toMatchObject({ kind: "rebased", executionSafe: true, baseSha: c1, observedHead: c2 });
|
||||
expect(c2).not.toBe(c1);
|
||||
expect(git(worktree, `merge-base --is-ancestor ${c1} ${c2}; echo $?`)).toBe("0");
|
||||
expect(store.updateTask).toHaveBeenCalledWith("FN-1", { baseCommitSha: c1 });
|
||||
});
|
||||
|
||||
it("returns the compensated persistence failure after restoring C0", async () => {
|
||||
const { root, worktree, c0 } = fixture();
|
||||
const store = { updateTask: vi.fn().mockRejectedValue(new Error("database unavailable")) } as any;
|
||||
const result = await refreshReusedWorktreeBase({
|
||||
task: { id: "FN-1", baseCommitSha: c0 } as any, rootDir: root, worktreePath: worktree, store, settings: {},
|
||||
});
|
||||
expect(result.kind).toBe("base-persistence-failed-compensated");
|
||||
expect(git(worktree, "rev-parse HEAD")).toBe(c0);
|
||||
});
|
||||
|
||||
it("statelessly reconciles stale durable C0 when clean HEAD is already C1", async () => {
|
||||
const { root, worktree, c0, c1 } = fixture();
|
||||
git(worktree, `reset --hard ${c1}`);
|
||||
const store = { updateTask: vi.fn().mockResolvedValue(undefined) } as any;
|
||||
const result = await refreshReusedWorktreeBase({
|
||||
task: { id: "FN-1", baseCommitSha: c0 } as any, rootDir: root, worktreePath: worktree, store, settings: {},
|
||||
});
|
||||
expect(result).toMatchObject({ kind: "up-to-date", executionSafe: true, baseSha: c1 });
|
||||
expect(store.updateTask).toHaveBeenCalledWith("FN-1", { baseCommitSha: c1 });
|
||||
});
|
||||
|
||||
it("blocks a dirty checkout without changing the durable baseline", async () => {
|
||||
const { root, worktree, c0 } = fixture();
|
||||
writeFileSync(join(worktree, "dirty.txt"), "keep me\n");
|
||||
const store = { updateTask: vi.fn().mockResolvedValue(undefined) } as any;
|
||||
const result = await refreshReusedWorktreeBase({
|
||||
task: { id: "FN-1", baseCommitSha: c0 } as any, rootDir: root, worktreePath: worktree, store, settings: {},
|
||||
});
|
||||
expect(result.kind).toBe("dirty-worktree");
|
||||
expect(store.updateTask).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -87,7 +87,7 @@ FN-7672 requires durable agent error recovery to stay classification-gated: only
|
||||
FNXC:HeartbeatRecovery 2026-07-15-08:50:
|
||||
heartbeat-model-unavailable parks from assignment/on-demand runs were terminal until a human Retry, even when the next attempt succeeds with unchanged credentials (false "model unavailable" / registry / credential-probe blips). Admit those parks to the same bounded heartbeatErrorRecovery budget as error-state recovery so the engine auto-retries like operator Retry, while genuine missing credentials re-park after the budget exhausts.
|
||||
*/
|
||||
import { acquireTaskWorktree } from "./worktree-acquisition.js";
|
||||
import { acquireTaskWorktree, WorktreeBaseRefreshError } from "./worktree-acquisition.js";
|
||||
import { createRunAuditor, generateSyntheticRunId, type DatabaseMutationType, type EngineRunContext } from "./run-audit.js";
|
||||
import { promptWithFallback } from "./pi.js";
|
||||
import { withRateLimitRetry } from "./rate-limit-retry.js";
|
||||
@@ -2952,12 +2952,45 @@ export class HeartbeatMonitor {
|
||||
runContext,
|
||||
runInitCommand: false,
|
||||
secretsStore: this.secretsStore,
|
||||
refreshStaleBase: true,
|
||||
});
|
||||
sessionCwd = acquisition.worktreePath;
|
||||
} catch (worktreeErr) {
|
||||
const detail = worktreeErr instanceof Error ? worktreeErr.message : String(worktreeErr);
|
||||
const refreshKind = worktreeErr instanceof WorktreeBaseRefreshError
|
||||
? worktreeErr.refresh.kind
|
||||
: undefined;
|
||||
heartbeatLog.warn(`Heartbeat worktree acquisition failed for ${agentId}: ${detail}`);
|
||||
|
||||
/*
|
||||
* FNXC:WorktreeBaseRefresh 2026-08-01-16:33:
|
||||
* Refresh refusals are deliberately recoverable pre-session parks, distinct from a
|
||||
* broken acquisition. Their typed outcome remains in task/run records and is retried
|
||||
* only on a later heartbeat after git state can change; never consume the generic
|
||||
* three-strike acquisition budget or replace the reason with terminal failure.
|
||||
*/
|
||||
if (refreshKind) {
|
||||
if (!(await isTaskInTerminalLane(taskStore, taskDetail))) {
|
||||
await taskStore.logEntry(
|
||||
taskDetail.id,
|
||||
`Worktree base refresh blocked heartbeat execution (${refreshKind})`,
|
||||
detail,
|
||||
);
|
||||
await taskStore.moveTask(
|
||||
taskDetail.id,
|
||||
await resolveHeartbeatReboundColumn(taskStore, taskDetail.id),
|
||||
{ preserveProgress: true },
|
||||
);
|
||||
}
|
||||
await this.completeRun(agentId, run.id, {
|
||||
status: "completed",
|
||||
resultJson: { reason: "worktree_base_refresh_blocked", refreshKind, detail },
|
||||
stderrExcerpt: detail,
|
||||
skipStateTransition: true,
|
||||
});
|
||||
return (await this.store.getRunDetail(agentId, run.id))!;
|
||||
}
|
||||
|
||||
/*
|
||||
* FNXC:WorktreeAcquisition 2026-07-09-00:00:
|
||||
* Bound consecutive cross-heartbeat acquisition failures for this task
|
||||
|
||||
@@ -9216,6 +9216,7 @@ export class TaskExecutor {
|
||||
task: TaskDetail,
|
||||
settings: Settings,
|
||||
nodeId: string,
|
||||
refreshStaleBase = false,
|
||||
): Promise<TaskDetail> {
|
||||
/*
|
||||
FNXC:WorkflowExecution 2026-06-29-08:21:
|
||||
@@ -9271,6 +9272,7 @@ export class TaskExecutor {
|
||||
}),
|
||||
taskEnv: process.env,
|
||||
secretsStore: this.options.secretsStore,
|
||||
refreshStaleBase,
|
||||
});
|
||||
this.addActiveWorktree(task.id, acquisition.worktreePath);
|
||||
if (!acquisition.isResume) {
|
||||
@@ -9392,8 +9394,14 @@ export class TaskExecutor {
|
||||
): Promise<void> {
|
||||
if (!requirement.requiresWorktree) return;
|
||||
const live = await this.store.getTask(nodeTask.id);
|
||||
if (live.worktree && existsSync(live.worktree)) return;
|
||||
const taskForAcquisition = live.worktree
|
||||
const executionCodeNode = node.kind === "code";
|
||||
if (live.worktree && existsSync(live.worktree) && !executionCodeNode) return;
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:32:
|
||||
An existing code-node checkout must remain attached to the acquisition input so it takes the
|
||||
guarded reuse/refresh path. Only a missing recorded path is cleared to permit fresh creation.
|
||||
*/
|
||||
const taskForAcquisition = live.worktree && !existsSync(live.worktree)
|
||||
? ({ ...live, worktree: undefined, sessionFile: undefined } as TaskDetail)
|
||||
: live;
|
||||
if (live.worktree) {
|
||||
@@ -9412,7 +9420,13 @@ export class TaskExecutor {
|
||||
FNXC:WorkflowExecution 2026-06-29-09:50:
|
||||
The workflow graph decides which nodes require pre-execution lifecycle resources. This adapter only fulfills a graph-declared worktree requirement with executor-owned git mechanics; custom-node handlers remain ordinary node execution and no longer decide when to bootstrap task isolation.
|
||||
*/
|
||||
await this.ensureGraphCustomNodeWorktree(taskForAcquisition, settings, node.id);
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:04:
|
||||
Code nodes are the sole graph implementation boundary. They reacquire an existing planning
|
||||
worktree with refresh enabled before a model session can start; planning and review nodes keep
|
||||
their C0 checkout so lane isolation does not become an implicit rebase policy.
|
||||
*/
|
||||
await this.ensureGraphCustomNodeWorktree(taskForAcquisition, settings, node.id, executionCodeNode);
|
||||
}
|
||||
|
||||
private async finalizeMergeConfirmedWorkflowGraphTask(taskId: string, reason: string): Promise<boolean> {
|
||||
@@ -10124,6 +10138,19 @@ export class TaskExecutor {
|
||||
return this.graphFailureErrorTexts(result).some((text) => isSessionContentionError(text));
|
||||
}
|
||||
|
||||
/** True only for the pre-session refresh refusal values emitted by graph preparation. */
|
||||
private isWorktreeBaseRefreshGraphFailure(result: WorkflowGraphTaskRunResult): boolean {
|
||||
return new Set([
|
||||
"stale-base-conflict",
|
||||
"dirty-worktree",
|
||||
"base-unresolvable",
|
||||
"worktrunk-refresh-unsupported",
|
||||
"git-refresh-failed",
|
||||
"base-persistence-failed-compensated",
|
||||
"base-reconciliation-required",
|
||||
]).has(this.graphFailureValue(result) ?? "");
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:SessionContention 2026-07-25-21:30 (self-recovering wait — the task is never parked):
|
||||
Retry the graph in place on an exponential backoff while the holder finishes. The counter is
|
||||
@@ -11298,6 +11325,39 @@ export class TaskExecutor {
|
||||
return;
|
||||
}
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:33:
|
||||
Code-node acquisition publishes every stale/unknown checkout refusal as a typed graph value.
|
||||
Keep it in the same bounded delayed-resume lane as other recoverable pre-session failures so
|
||||
no handler runs, no failure edge mislabels it as a plan defect, and its exact reason survives
|
||||
in the task log. Exhaustion deliberately leaves the task held for a later clean acquisition.
|
||||
*/
|
||||
if (this.isWorktreeBaseRefreshGraphFailure(result)) {
|
||||
const refreshKind = this.graphFailureValue(result)!;
|
||||
const priorRetries = live.graphResumeRetryCount ?? 0;
|
||||
if (priorRetries < MAX_TRANSIENT_GRAPH_RESUME_RETRIES) {
|
||||
const nextRetries = priorRetries + 1;
|
||||
const message = `Worktree base refresh blocked execution (${refreshKind}) — retrying in place (${nextRetries}/${MAX_TRANSIENT_GRAPH_RESUME_RETRIES})`;
|
||||
await this.store.logEntry(task.id, message, undefined, this.getRunContextFor(task.id));
|
||||
await this.store.updateTask(task.id, { graphResumeRetryCount: nextRetries }, this.getRunContextFor(task.id));
|
||||
const scheduleRetry = () => {
|
||||
this.execute(live).catch((err) =>
|
||||
executorLog.error(`Failed worktree base refresh retry for ${task.id}:`, err),
|
||||
);
|
||||
};
|
||||
const handle = setTimeout(scheduleRetry, TRANSIENT_GRAPH_RESUME_RETRY_BACKOFF_MS);
|
||||
handle.unref?.();
|
||||
} else {
|
||||
await this.store.logEntry(
|
||||
task.id,
|
||||
`Worktree base refresh remains blocked (${refreshKind}) — retry budget exhausted; task remains held`,
|
||||
undefined,
|
||||
this.getRunContextFor(task.id),
|
||||
);
|
||||
}
|
||||
await this.persistTokenUsage(task.id);
|
||||
return;
|
||||
}
|
||||
/*
|
||||
FNXC:MissingWorktreeRecovery 2026-07-16-18:25:
|
||||
An unusable-worktree session-start refusal inside a graph node must route to the bounded
|
||||
worktree-session recovery BEFORE any other classifier: FN-7977's provider-failure hold
|
||||
@@ -12991,6 +13051,7 @@ export class TaskExecutor {
|
||||
}),
|
||||
taskEnv,
|
||||
secretsStore: this.options.secretsStore,
|
||||
refreshStaleBase: true,
|
||||
});
|
||||
} finally {
|
||||
this.unregisterConfiguredCommandController(task.id, taskCommandAbortController);
|
||||
|
||||
@@ -7099,6 +7099,11 @@ export async function aiMergeTask(
|
||||
projectRootDir,
|
||||
);
|
||||
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:04:
|
||||
Merge deliberately leaves refreshStaleBase off. The merge lane owns its rebase policy through
|
||||
decideAutoPrerebase/runAutoPrerebase; refreshing here would double-rebase a branch after review.
|
||||
*/
|
||||
const acquisition = await acquireTaskWorktree({
|
||||
task,
|
||||
rootDir: projectRootDir,
|
||||
|
||||
@@ -96,6 +96,17 @@ export type GitMutationType =
|
||||
| "worktree:remove-classification-probe-failed"
|
||||
| "worktree:remove-leaked-registered-worktree"
|
||||
| "worktree:reuse"
|
||||
/*
|
||||
* FNXC:WorktreeBaseRefresh 2026-08-01-16:04:
|
||||
* Execution-only reused-worktree refresh records outcome-only audit events so operators can
|
||||
* distinguish a mechanical C1 advance, a typed block/conflict, compensated persistence failure,
|
||||
* and stateless C0-to-C1 reconciliation without recording command output or branch prose.
|
||||
*/
|
||||
| "worktree:base-refreshed"
|
||||
| "worktree:base-refresh-blocked"
|
||||
| "worktree:base-refresh-conflict"
|
||||
| "worktree:base-refresh-persistence-failed-compensated"
|
||||
| "worktree:base-refresh-reconciled"
|
||||
/*
|
||||
* FNXC:TaskPinnedWorktrees 2026-07-16-00:00:
|
||||
* Emitted when task-pinned acquisition (`worktreeNaming: "task-id"`) corrects a `task.worktree` cache that
|
||||
|
||||
@@ -46,6 +46,7 @@ import { runLoop, runOptionalGroup } from "./workflow-graph-loop.js";
|
||||
import type { WorkflowNodeRunnerRegistry } from "./workflow-node-runner.js";
|
||||
import { workflowNodeRequiresWorktree } from "./workflow-node-execution-needs.js";
|
||||
import type { WorkflowColumnBoundary } from "./workflow-column-boundary.js";
|
||||
import { WorktreeBaseRefreshError } from "./worktree-acquisition.js";
|
||||
|
||||
export type WorkflowNodeOutcome = "success" | "failure";
|
||||
|
||||
@@ -1596,6 +1597,27 @@ export class WorkflowGraphExecutor {
|
||||
return projected;
|
||||
} catch (error) {
|
||||
if (signal?.aborted) return this.withEnginePauseAbortContext(node, { outcome: "failure", value: "aborted" });
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:33:
|
||||
A code-node refresh refusal is a pre-session, typed non-execution result, not a handler
|
||||
exception. Do not spend immediate node retries or erase its actionable reason: returning
|
||||
the refresh kind lets the graph route/park it while preserving the checkout for a later,
|
||||
independently verified acquisition.
|
||||
*/
|
||||
if (error instanceof WorktreeBaseRefreshError) {
|
||||
const failureResult: WorkflowNodeResult = {
|
||||
outcome: "failure",
|
||||
value: error.refresh.kind,
|
||||
contextPatch: {
|
||||
[`node:${node.id}:error`]: error.message,
|
||||
[`node:${node.id}:baseRefresh`]: error.refresh.kind,
|
||||
},
|
||||
};
|
||||
if (recordProgress && this.shouldRecordNodeProgress(node)) {
|
||||
await this.recordNodeProgressFinish(task.id, node, null, failureResult);
|
||||
}
|
||||
return failureResult;
|
||||
}
|
||||
lastError = error;
|
||||
/*
|
||||
FNXC:SessionContention 2026-07-25-21:30:
|
||||
@@ -1742,13 +1764,19 @@ export class WorkflowGraphExecutor {
|
||||
* disables inline fixes, preserving the default-enabled review worktree contract
|
||||
* that prevents issue #2075's pre-review no-worktree failure.
|
||||
*/
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:04:
|
||||
A graph `code` node is the implementation boundary even when its sandbox runner does not
|
||||
otherwise advertise a worktree need. Force preparation so an existing planning checkout is
|
||||
refreshed before code executes; review and planning retain their normal classifier behavior.
|
||||
*/
|
||||
const requiresWorktree = workflowNodeRequiresWorktree(node, {
|
||||
optionalGroupId,
|
||||
reviewerInlineFixes: settings?.reviewerInlineFixes,
|
||||
});
|
||||
}) || node.kind === "code";
|
||||
return {
|
||||
requiresWorktree,
|
||||
reason: requiresWorktree ? "write-capable-node" : undefined,
|
||||
reason: node.kind === "code" ? "implementation-code-node" : requiresWorktree ? "write-capable-node" : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -43,6 +43,7 @@ import { copyConfiguredWorktreeFiles, type WorktreeCopyFileResult } from "./work
|
||||
import { resolveCapturedBaseCommitSha } from "./base-commit-capture.js";
|
||||
import { resolveIntegrationBranch } from "./integration-branch.js";
|
||||
import { activeSessionRegistry, type ActiveSessionRegistry } from "./active-session-registry.js";
|
||||
import { refreshReusedWorktreeBase, type WorktreeBaseRefreshResult } from "./worktree-base-refresh.js";
|
||||
|
||||
const execAsync = promisify(exec);
|
||||
|
||||
@@ -81,6 +82,8 @@ export interface AcquireTaskWorktreeOptions {
|
||||
}>;
|
||||
taskEnv?: NodeJS.ProcessEnv;
|
||||
backend?: WorktreeBackend;
|
||||
/** Execution callers opt in; planning, review, and merge reuse remain unchanged. */
|
||||
refreshStaleBase?: boolean;
|
||||
}
|
||||
|
||||
export interface AcquireTaskWorktreeResult {
|
||||
@@ -93,6 +96,15 @@ export interface AcquireTaskWorktreeResult {
|
||||
existingTipSha?: string;
|
||||
strandedCommitCount?: number;
|
||||
};
|
||||
baseRefresh?: WorktreeBaseRefreshResult;
|
||||
}
|
||||
|
||||
/** A typed refresh refusal: callers must park before creating a coding session. */
|
||||
export class WorktreeBaseRefreshError extends Error {
|
||||
constructor(public readonly refresh: WorktreeBaseRefreshResult) {
|
||||
super(`Worktree base refresh blocked execution: ${refresh.kind}`);
|
||||
this.name = "WorktreeBaseRefreshError";
|
||||
}
|
||||
}
|
||||
|
||||
type InitCommandResult = Awaited<ReturnType<NonNullable<AcquireTaskWorktreeOptions["runConfiguredCommand"]>>>;
|
||||
@@ -201,6 +213,16 @@ async function pinnedWorktreeBranchMatches(rootDir: string, worktreePath: string
|
||||
|
||||
export async function acquireTaskWorktree(opts: AcquireTaskWorktreeOptions): Promise<AcquireTaskWorktreeResult> {
|
||||
const { task, rootDir, store, settings, pool, logger, audit, runContext, createWorktree, runConfiguredCommand, runInitCommand, taskEnv, secretsStore } = opts;
|
||||
const refreshExistingWorktree = async (path: string): Promise<WorktreeBaseRefreshResult | undefined> => {
|
||||
if (!opts.refreshStaleBase) return undefined;
|
||||
const refresh = await refreshReusedWorktreeBase({ task, rootDir, worktreePath: path, store, settings, audit, logger });
|
||||
if (!refresh.executionSafe) {
|
||||
await audit?.git({ type: refresh.kind === "stale-base-conflict" ? "worktree:base-refresh-conflict" : "worktree:base-refresh-blocked", target: path, metadata: { taskId: task.id, outcome: refresh.kind } });
|
||||
await store.logEntry(task.id, `Worktree base refresh blocked execution (${refresh.kind})`, refresh.detail, runContext);
|
||||
throw new WorktreeBaseRefreshError(refresh);
|
||||
}
|
||||
return refresh;
|
||||
};
|
||||
const notifyFallback = async (op: WorktrunkOpName, stderr?: string) => {
|
||||
await store.logEntry(task.id, `Worktrunk ${op} failed; continuing with native worktree backend (${stderr ?? "no stderr"})`, undefined, runContext);
|
||||
};
|
||||
@@ -550,7 +572,8 @@ export async function acquireTaskWorktree(opts: AcquireTaskWorktreeOptions): Pro
|
||||
logger,
|
||||
runContext,
|
||||
});
|
||||
return guardAcquisitionReturn({ worktreePath: path, branch: resumedBranch, source, hydrated, isResume: true });
|
||||
const baseRefresh = await refreshExistingWorktree(path);
|
||||
return guardAcquisitionReturn({ worktreePath: path, branch: resumedBranch, source, hydrated, isResume: true, baseRefresh });
|
||||
};
|
||||
|
||||
/*
|
||||
@@ -674,7 +697,8 @@ export async function acquireTaskWorktree(opts: AcquireTaskWorktreeOptions): Pro
|
||||
runContext,
|
||||
});
|
||||
// FN-4912: resume path reuses the prior on-disk .env (and its fingerprint sidecar). Rewrite is owned by the next fresh acquisition.
|
||||
return guardAcquisitionReturn({ worktreePath, branch: resumedBranch, source: "existing", hydrated, isResume: true });
|
||||
const baseRefresh = await refreshExistingWorktree(worktreePath);
|
||||
return guardAcquisitionReturn({ worktreePath, branch: resumedBranch, source: "existing", hydrated, isResume: true, baseRefresh });
|
||||
}
|
||||
|
||||
if (!isResume && pool && settings.recycleWorktrees) {
|
||||
|
||||
158
packages/engine/src/worktree-base-refresh.ts
Normal file
158
packages/engine/src/worktree-base-refresh.ts
Normal file
@@ -0,0 +1,158 @@
|
||||
import { exec } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import type { Settings, Task, TaskStore } from "@fusion/core";
|
||||
import { resolveIntegrationBranch } from "./integration-branch.js";
|
||||
import type { GitAuditInput } from "./run-audit.js";
|
||||
|
||||
const execAsync = promisify(exec);
|
||||
const GIT_TIMEOUT_MS = 120_000;
|
||||
|
||||
export type WorktreeBaseRefreshKind =
|
||||
| "up-to-date"
|
||||
| "reset-to-base"
|
||||
| "rebased"
|
||||
| "stale-base-conflict"
|
||||
| "dirty-worktree"
|
||||
| "base-unresolvable"
|
||||
| "worktrunk-refresh-unsupported"
|
||||
| "git-refresh-failed"
|
||||
| "base-persistence-failed-compensated"
|
||||
| "base-reconciliation-required";
|
||||
|
||||
export interface WorktreeBaseRefreshResult {
|
||||
kind: WorktreeBaseRefreshKind;
|
||||
executionSafe: boolean;
|
||||
integrationBranch?: string;
|
||||
baseSha?: string;
|
||||
originalHead?: string;
|
||||
observedHead?: string;
|
||||
durableBaseSha?: string | null;
|
||||
detail?: string;
|
||||
}
|
||||
|
||||
export interface RefreshReusedWorktreeBaseInput {
|
||||
task: Task;
|
||||
rootDir: string;
|
||||
worktreePath: string;
|
||||
store: TaskStore;
|
||||
settings: Partial<Settings>;
|
||||
/** Audit is intentionally structural so acquisition can use its run-scoped auditor. */
|
||||
audit?: { git?: (event: GitAuditInput) => Promise<void> };
|
||||
logger?: { warn: (message: string) => void };
|
||||
}
|
||||
|
||||
function quote(value: string): string {
|
||||
return `'${value.replace(/'/g, "'\\''")}'`;
|
||||
}
|
||||
|
||||
async function git(cwd: string, command: string): Promise<string> {
|
||||
const { stdout } = await execAsync(`git ${command}`, { cwd, encoding: "utf8", timeout: GIT_TIMEOUT_MS, maxBuffer: 10 * 1024 * 1024 });
|
||||
return stdout.trim();
|
||||
}
|
||||
|
||||
async function isAncestor(cwd: string, ancestor: string, descendant: string): Promise<boolean> {
|
||||
try {
|
||||
await git(cwd, `merge-base --is-ancestor ${quote(ancestor)} ${quote(descendant)}`);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function compensate(cwd: string, originalHead: string): Promise<boolean> {
|
||||
try {
|
||||
await git(cwd, `rebase --abort`);
|
||||
} catch {
|
||||
// No rebase may be in progress; reset below is the proof-bearing restoration.
|
||||
}
|
||||
try {
|
||||
await git(cwd, `reset --hard ${quote(originalHead)}`);
|
||||
const [head, dirty] = await Promise.all([git(cwd, "rev-parse HEAD"), git(cwd, "status --porcelain")]);
|
||||
return head === originalHead && !dirty;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:WorktreeBaseRefresh 2026-08-01-16:04:
|
||||
Planning deliberately creates an isolated task worktree, but execution may begin after dependencies land.
|
||||
Only refresh-enabled execution reuse reaches this primitive: planning, review, gates, and merge retain their
|
||||
existing worktrees. The primitive refuses dirty, unresolved, worktrunk, conflict, git, persistence, and
|
||||
reconciliation-unknown states with typed non-execution outcomes. It resolves integration C1 independently
|
||||
on every reuse and aligns durable baseCommitSha to C1; a rebased own-commit HEAD C2 is preserved and is never
|
||||
stored as the baseline. Git mutations are compensated back to C0 when baseline persistence fails; a later
|
||||
process can statelessly reconcile C0 versus clean C1/C2 from durable metadata and git proof alone.
|
||||
*/
|
||||
export async function refreshReusedWorktreeBase(input: RefreshReusedWorktreeBaseInput): Promise<WorktreeBaseRefreshResult> {
|
||||
const { task, rootDir, worktreePath, store, settings, audit, logger } = input;
|
||||
if (settings.worktrunk?.enabled) {
|
||||
return { kind: "worktrunk-refresh-unsupported", executionSafe: false, durableBaseSha: task.baseCommitSha ?? null };
|
||||
}
|
||||
|
||||
let integrationBranch: string;
|
||||
let baseSha: string;
|
||||
let originalHead: string;
|
||||
let dirty: string;
|
||||
try {
|
||||
integrationBranch = await resolveIntegrationBranch(rootDir, settings, { logger: logger ?? console });
|
||||
[baseSha, originalHead, dirty] = await Promise.all([
|
||||
git(rootDir, `rev-parse --verify ${quote(`${integrationBranch}^{commit}`)}`),
|
||||
git(worktreePath, "rev-parse HEAD"),
|
||||
git(worktreePath, "status --porcelain"),
|
||||
]);
|
||||
} catch (error) {
|
||||
return { kind: "base-unresolvable", executionSafe: false, durableBaseSha: task.baseCommitSha ?? null, detail: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
const common = { integrationBranch, baseSha, originalHead, durableBaseSha: task.baseCommitSha ?? null };
|
||||
if (dirty) return { kind: "dirty-worktree", executionSafe: false, observedHead: originalHead, ...common };
|
||||
|
||||
const baselineMatches = task.baseCommitSha === baseSha;
|
||||
const headCurrent = await isAncestor(worktreePath, baseSha, originalHead);
|
||||
if (headCurrent) {
|
||||
if (baselineMatches) return { kind: "up-to-date", executionSafe: true, observedHead: originalHead, ...common };
|
||||
try {
|
||||
await store.updateTask(task.id, { baseCommitSha: baseSha });
|
||||
await audit?.git?.({ type: "worktree:base-refresh-reconciled", target: worktreePath, metadata: { taskId: task.id, outcome: "reconciled" } });
|
||||
return { kind: "up-to-date", executionSafe: true, observedHead: originalHead, ...common };
|
||||
} catch (error) {
|
||||
return { kind: "base-reconciliation-required", executionSafe: false, observedHead: originalHead, ...common, detail: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
}
|
||||
|
||||
let mergeBase: string;
|
||||
try {
|
||||
mergeBase = await git(worktreePath, `merge-base HEAD ${quote(baseSha)}`);
|
||||
} catch (error) {
|
||||
return { kind: "git-refresh-failed", executionSafe: false, observedHead: originalHead, ...common, detail: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
let kind: "reset-to-base" | "rebased";
|
||||
try {
|
||||
const ownCommits = await git(worktreePath, `rev-list --count ${quote(`${mergeBase}..HEAD`)}`);
|
||||
if (Number(ownCommits) === 0) {
|
||||
await git(worktreePath, `reset --hard ${quote(baseSha)}`);
|
||||
kind = "reset-to-base";
|
||||
} else {
|
||||
try {
|
||||
await git(worktreePath, `rebase ${quote(baseSha)}`);
|
||||
kind = "rebased";
|
||||
} catch (error) {
|
||||
const restored = await compensate(worktreePath, originalHead);
|
||||
return { kind: restored ? "stale-base-conflict" : "base-reconciliation-required", executionSafe: false, observedHead: originalHead, ...common, detail: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
return { kind: "git-refresh-failed", executionSafe: false, observedHead: originalHead, ...common, detail: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
|
||||
const observedHead = await git(worktreePath, "rev-parse HEAD").catch(() => undefined);
|
||||
try {
|
||||
await store.updateTask(task.id, { baseCommitSha: baseSha });
|
||||
await audit?.git?.({ type: "worktree:base-refreshed", target: worktreePath, metadata: { taskId: task.id, outcome: kind } });
|
||||
return { kind, executionSafe: true, observedHead, ...common };
|
||||
} catch (error) {
|
||||
const restored = await compensate(worktreePath, originalHead);
|
||||
await audit?.git?.({ type: restored ? "worktree:base-refresh-persistence-failed-compensated" : "worktree:base-refresh-blocked", target: worktreePath, metadata: { taskId: task.id, outcome: restored ? "compensated" : "reconciliation-required" } }).catch(() => undefined);
|
||||
return { kind: restored ? "base-persistence-failed-compensated" : "base-reconciliation-required", executionSafe: false, observedHead, ...common, detail: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user