FN-5743: cut over merge dequeue authority to merge-request queue
Shift merge dequeue enforcement to merge-request queue/marker authority with reliability coverage updates. - Enforce merge eligibility and dequeue ownership via merge-request queue/marker checks in engine and scheduler paths. - Add hard-cancel behavior coverage to ensure queued merge requests are canceled when tasks are user-canceled. - Extend core merge-request record/store tests and reliability interaction suites for dual-observe and cancel-on-hard-cancel seams. - Document the FN-5741/FN-5743 reliability backstop updates in AGENTS.md and architecture docs. Files changed: AGENTS.md | 2 + docs/architecture.md | 2 + packages/core/src/__tests__/merge-request-record.test.ts | 18 ++++ packages/core/src/store.ts | 13 +++ packages/engine/src/__tests__/reliability-interactions/dual-observe-merge-seam.test.ts | 57 +++++++++++++ packages/engine/src/__tests__/reliability-interactions/merge-request-cancel-on-hard-cancel.test.ts | 87 +++++++++++++++++++ packages/engine/src/project-engine.ts | 98 +++++++++++++++++++++- packages/engine/src/scheduler.ts | 12 ++- 8 files changed, 284 insertions(+), 5 deletions(-) Fusion-Task-Id: FN-5743 Fusion-Task-Lineage: b9a49aeb-ed73-42fe-924d-25c7097d1bb9
This commit is contained in:
@@ -162,6 +162,8 @@ When `settings.autoMerge: false`, `in-review` is terminal-until-merged by a huma
|
||||
- FN-5715 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-validation-trigger-gap.test.ts` guards mission validation trigger continuity so done task completion and startup recovery both route assertion-linked features through validator runs before completion.
|
||||
- FN-5738 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-validation-trigger-gap.test.ts` extends mission-loop coverage so zero-assertion auto-pass deterministically advances to `loopState="passed"` and emits `validation_auto_passed_no_assertions` without duplicate recovery re-fire.
|
||||
- FN-5741 backstop: `packages/engine/src/__tests__/reliability-interactions/merge-request-shadow-handoff.test.ts` guards Phase-1 write-only-shadow merge-request record + handoff-accepted marker seam (flag OFF = no-op, ON = shadow-only non-authoritative).
|
||||
- FN-5742 backstop: `packages/engine/src/__tests__/reliability-interactions/dual-observe-merge-seam.test.ts` guards Phase-2 dual-observe parity (dependency + lease diffs, shadow dequeue parity, manual-required shadow skip) while legacy behavior remains authoritative.
|
||||
- FN-5743 backstop: `packages/engine/src/__tests__/reliability-interactions/merge-request-cancel-on-hard-cancel.test.ts` plus `packages/core/src/__tests__/merge-request-record.test.ts` guard Phase-3 cutover semantics (merge-request retry state transitions, authoritative user hard-cancel tombstone, and non-user rebound no-op cancel semantics).
|
||||
- FN-5754 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-stranded-feature-retriage.test.ts` guards startup/maintenance stranded-feature re-triage for active autopilot slices, including link-first dedupe, non-defined skip safety, non-autopilot no-op, idempotency, and `mission:stranded-feature-triaged` audit shape.
|
||||
- FN-5755 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-validation-trigger-gap.test.ts` extends mission validation coverage so bounded periodic maintenance replays `recoverActiveMissions` for stranded `implementing` features and remains idempotent on repeated passes.
|
||||
|
||||
|
||||
@@ -1742,6 +1742,7 @@ This section preserves the detailed lifecycle/self-healing contracts that were f
|
||||
- **Self-owned reclaim resume-limbo escalation (FN-5704)**: `reclaimSelfOwnedBranchConflicts` tracks `resumeLimboCount`, `resumeLimboTipSha`, and `resumeLimboStepSignature` for in-progress reclaim/unpause loops. If reclaim finds no progress (same tip, same step-status signature, and no active-session signal) for `MAX_NO_PROGRESS_RESUME_ATTEMPTS` consecutive sweeps, self-healing escalates by moving the task to `todo` with `preserveWorktree: true`, `preserveProgress: true`, and `preserveResumeState: true` instead of endlessly re-arming resume. Escalation emits `task:resume-limbo-escalated` run-audit metadata (`frozenTipSha`, `idleMs`, `resumeAttemptCount`, `currentStep`) and resets the limbo counter.
|
||||
- **Merge-request shadow contract (FN-5741 Phase 1)**: `mergeRequestContractShadowEnabled` defaults OFF. OFF means no writes to `merge_requests` or `completion_handoff_markers` and the legacy lifecycle remains authoritative. ON enables write-only shadow persistence: executor/self-healing append `task:completion-handoff-accepted` marker+record writes only after successful legacy `handoffToReview`, and merger mirrors `merge:request-enqueued` plus `queued → running → succeeded` (or `manual-required` for `autoMerge:false`) transitions. Phase 1 never reads these shadow records for column movement, dependency checks, lease arbitration, merge dequeue, or FN-5479/FN-5704 limbo recovery decisions.
|
||||
- **Dual-observe parity seam (FN-5742 Phase 2)**: with the same flag ON, legacy remains authoritative while shadow reads compute/emit parity telemetry only. Scheduler emits `merge:dependency-parity-diff` when `in-review|done|archived` dependency satisfaction diverges from completion-handoff marker satisfaction, and `merge:lease-parity-diff` when legacy in-review overlap leasing diverges from shadow lease decomposition. Merger emits `merge:request-dequeued-shadow` (agree/disagree metadata) by comparing legacy dequeue selection to shadow merge-request selection while explicitly skipping `manual-required` rows. Phase 3 dequeue cutover is gated on sustained parity (low disagreement rate) from these additive events; no lifecycle authority changes in Phase 2.
|
||||
- **Authoritative cutover seam (FN-5743 Phase 3)**: with the flag ON, merge-request records and `completion_handoff_accepted` markers become authoritative enforcement signals for dequeue/retry ownership and dependency/lease gates. Accepted handoffs stop stamping `in-review` executor overlap leases, transient merge retries stay in merge-request state (`running → retrying → queued`, terminal `exhausted|succeeded|cancelled`) without `todo` rebounds, and user hard-cancel (`in-review → todo`) deterministically cancels pending merge-request records while keeping FN-5147/FN-5704 behavior unchanged.
|
||||
- **No-progress churn terminalization (FN-5168)**: `StuckTaskDetector` now tracks ignored `fn_task_update` rebuffs via `recordIgnoredStepUpdate(taskId)` and, after one loop/compact-and-resume recovery has already fired in the same `execute()` lifecycle, escalates `ignoredStepUpdateCount >= 25` to the terminal reason `no-progress-churn`. `SelfHealingManager.checkStuckBudget()` maps that reason directly to `STUCK_NO_PROGRESS_CHURN`, emits `task:stuck-no-progress-churn-terminalized` with `{ taskId, ignoredStepUpdateCount, stuckKillStreak, lastReason }`, and parks the task in `in-review` without consuming the normal stuck-kill budget. Under FN-5147 `autoMerge: false`, that failed in-review task remains terminal-until-merged just like `STUCK_LOOP_EXHAUSTED`; the new class adds an earlier bounded exit, not a re-execution path.
|
||||
- **Landed-files attribution (FN-5103)**: Rebase-strategy `mergeDetails.landedFiles` / `filesChanged` / `insertions` / `deletions` are captured from task-attributable commits only via `filterFilesToOwnTaskCommits` (subject-prefix + trailer + bracket-prefix evidence), tagged `landedFilesAttributionRestricted: true`. Zero own commits → `landedFiles: []` and `noOpVerifiedShortCircuit: true`. FN-5304 guard: when `<rebaseBaseSha>..HEAD` reports zero own commits, merger must also validate the source `fusion/<id>` tip; if that source tip still has attributable own commits relative to `rebaseBaseSha`, throw `SilentNoOpAttributionMismatchError`, refuse writing `mergeConfirmed: true`, park the task in `in-review` with `status: "failed"`, and emit `merge:no-op-attribution-mismatch`. If source ref is unavailable, skip with diagnostic + `merge:no-op-attribution-mismatch-skipped` (`reason: "source-ref-unavailable"`). Attribution-helper failures fall back to the unrestricted `rebaseBaseSha..sha` walk and set `landedFilesCaptureFallback: 'attribution-failed'`. Self-healing `recoverDoneTaskMergeMetadata` skips reconcile when `landedFilesAttributionRestricted` or `noOpVerifiedShortCircuit` is set so the narrower set is not overwritten with the full range. Squash-strategy capture is unchanged.
|
||||
- **Soft-delete scheduler invalidation (FN-5137)**: `task:deleted` events must invalidate `AutoClaimSnapshotManager` and clear scheduler bookkeeping (`pausedTaskIds`, `failedTaskIds`, `wasNodeDispatchValidationBlocked`, `wasNodeBlocked`); `executor.execute()` / `resumeOrphaned()` / `resumeTaskForAgent()` refuse any task with `deletedAt` set.
|
||||
@@ -1773,6 +1774,7 @@ Reliability-layer changes are in scope. Interaction regression backstops live in
|
||||
- FN-5738 backstop: `packages/engine/src/__tests__/reliability-interactions/mission-validation-trigger-gap.test.ts` extends coverage so zero-assertion auto-pass deterministically advances `loopState` to `passed`, sets `lastValidatorStatus="passed"`, emits `validation_auto_passed_no_assertions`, and does not re-fire on repeated recovery passes.
|
||||
- FN-5741 backstop: `packages/engine/src/__tests__/reliability-interactions/merge-request-shadow-handoff.test.ts` guards Phase-1 merge-request contract shadow writes: flag OFF is a no-op, flag ON writes marker/record strictly after legacy handoff, and `autoMerge:false` remains `manual-required` without shadow running transitions.
|
||||
- FN-5742 backstop: `packages/engine/src/__tests__/reliability-interactions/dual-observe-merge-seam.test.ts` guards Phase-2 dual-observe invariants: legacy dependency satisfaction remains authoritative while parity diffs emit, and shadow dequeue selection never advances `manual-required` rows.
|
||||
- FN-5743 backstop: `packages/engine/src/__tests__/reliability-interactions/merge-request-cancel-on-hard-cancel.test.ts` and `packages/core/src/__tests__/merge-request-record.test.ts` guard Phase-3 cutover invariants: transient merge retries mutate merge-request state (no column rebound), user hard-cancel after accepted handoff cancels pending merge requests, and non-user rebounds preserve legacy fail-soft semantics.
|
||||
- FN-5337 backstop: `packages/engine/src/__tests__/reliability-interactions/orphan-detected-no-requeue.test.ts` locks observation-only orphan detection across FN-5279 repro metadata desync, worktree-present and worktree-missing candidates, FN-5219 ordering, FN-5147 in-review isolation, FN-5083 branch-cleared composition, lease-manager non-invocation, and per-sweep idempotent audit emission.
|
||||
- FN-5256 backstop: `packages/engine/src/__tests__/reliability-interactions/dependency-cycle-reconcile.test.ts` covers persisted dependency-cycle detection via `reconcileDependencyCycles`, bounded umbrella-back-edge auto-repair, ambiguous-cycle observe-only behavior, composition ordering with `reconcileSelfDefeatingDependencies`, and the post-sweep write-time guard invariant. Core write-boundary regressions (FN-5240/5241/5242 signature, indirect cycle, umbrella back-edge rejection) live in `packages/core/src/__tests__/store-dependency-cycle.test.ts`.
|
||||
- FN-5325 backstop: `packages/engine/src/__tests__/reliability-interactions/scheduler-overlap-priority-inversion.test.ts` covers queued-overlap priority/age deferral, equal-priority age ordering, FN-4969 fanout composition, and one-shot per-pass `scheduler:overlap-priority-inversion` audit surfacing against running lower-priority blockers.
|
||||
|
||||
@@ -94,4 +94,22 @@ describe("TaskStore merge request record + completion handoff marker", () => {
|
||||
store.clearCompletionHandoffAcceptedMarker(taskId);
|
||||
expect(store.getCompletionHandoffAcceptedMarker(taskId)).toBeNull();
|
||||
});
|
||||
|
||||
it("cancels merge request and clears handoff marker on user hard-cancel from in-review to todo", async () => {
|
||||
const taskId = await createTask();
|
||||
await store.moveTask(taskId, "todo");
|
||||
await store.moveTask(taskId, "in-progress");
|
||||
await store.handoffToReview(taskId, {
|
||||
ownerAgentId: "agent-test",
|
||||
evidence: { reason: "fn_task_done", runId: "run-1", agentId: "agent-test" },
|
||||
});
|
||||
|
||||
store.upsertMergeRequestRecord(taskId, { state: "queued", attemptCount: 1, lastError: "pending" });
|
||||
store.setCompletionHandoffAcceptedMarker(taskId, { source: "executor:fn_task_done" });
|
||||
|
||||
await store.moveTask(taskId, "todo", { moveSource: "user" });
|
||||
|
||||
expect(store.getMergeRequestRecord(taskId)?.state).toBe("cancelled");
|
||||
expect(store.getCompletionHandoffAcceptedMarker(taskId)).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -5544,6 +5544,19 @@ export class TaskStore extends EventEmitter<TaskStoreEvents> {
|
||||
}
|
||||
|
||||
await this.writeTaskJsonFile(dir, task);
|
||||
if (fromColumn === "in-review" && toColumn === "todo" && moveSource === "user") {
|
||||
const handoffAccepted = this.getCompletionHandoffAcceptedMarker(id);
|
||||
const mergeRequest = this.getMergeRequestRecord(id);
|
||||
if (handoffAccepted && mergeRequest && mergeRequest.state !== "succeeded" && mergeRequest.state !== "cancelled") {
|
||||
if (mergeRequest.state === "queued" || mergeRequest.state === "running" || mergeRequest.state === "retrying" || mergeRequest.state === "manual-required") {
|
||||
this.transitionMergeRequestState(id, "cancelled", {
|
||||
attemptCount: mergeRequest.attemptCount,
|
||||
lastError: mergeRequest.lastError ?? "cancelled-by-user-hard-cancel",
|
||||
});
|
||||
}
|
||||
}
|
||||
this.clearCompletionHandoffAcceptedMarker(id);
|
||||
}
|
||||
if (toColumn === "done") {
|
||||
this.clearLinkedAgentTaskIds(id, task.updatedAt);
|
||||
}
|
||||
|
||||
@@ -190,4 +190,61 @@ describe("FN-5742 dual-observe merge seam", () => {
|
||||
|
||||
expect(updateTask).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("transient retry transitions merge-request running->retrying->queued under contract", async () => {
|
||||
const transitions: string[] = [];
|
||||
let state = "running";
|
||||
const store = {
|
||||
getSettings: vi.fn().mockResolvedValue({ mergeRequestContractShadowEnabled: true }),
|
||||
getMergeRequestRecord: vi.fn(() => ({ state, attemptCount: 0, lastError: null })),
|
||||
transitionMergeRequestState: vi.fn((_taskId: string, to: string) => {
|
||||
transitions.push(`${state}->${to}`);
|
||||
state = to;
|
||||
}),
|
||||
updateTask: vi.fn().mockResolvedValue(undefined),
|
||||
logEntry: vi.fn().mockResolvedValue(undefined),
|
||||
} as any;
|
||||
|
||||
const retried = await (ProjectEngine.prototype as any).maybeRetryTransientMerge.call(
|
||||
{ shuttingDown: false, internalEnqueueMerge: vi.fn() },
|
||||
store,
|
||||
"FN-MR",
|
||||
{ mergeTransientRetryCount: 0 },
|
||||
"lease-handoff-failed: target-not-queued",
|
||||
);
|
||||
|
||||
expect(retried).toBe(true);
|
||||
expect(transitions).toEqual(["running->retrying", "retrying->queued"]);
|
||||
});
|
||||
|
||||
it("transient exhaustion marks merge request exhausted under contract", async () => {
|
||||
const logs: string[] = [];
|
||||
let state = "running";
|
||||
const store = {
|
||||
getSettings: vi.fn().mockResolvedValue({ mergeRequestContractShadowEnabled: true }),
|
||||
getTask: vi.fn().mockResolvedValue({ id: "FN-MR", column: "in-review" }),
|
||||
getMergeRequestRecord: vi.fn(() => ({ state, attemptCount: 3, lastError: null })),
|
||||
transitionMergeRequestState: vi.fn((_taskId: string, to: string) => {
|
||||
state = to;
|
||||
}),
|
||||
logEntry: vi.fn(async (_taskId: string, message: string) => logs.push(message)),
|
||||
updateTask: vi.fn(),
|
||||
getActiveMergingTask: vi.fn().mockReturnValue(null),
|
||||
} as any;
|
||||
|
||||
if ((ProjectEngine.prototype as any).isTransientMergeRetryExhausted.call({}, { mergeTransientRetryCount: 3 }, "socket hang up")) {
|
||||
const record = store.getMergeRequestRecord("FN-MR");
|
||||
if (record.state === "running") {
|
||||
store.transitionMergeRequestState("FN-MR", "retrying", { attemptCount: record.attemptCount, lastError: "socket hang up" });
|
||||
}
|
||||
const refreshed = store.getMergeRequestRecord("FN-MR");
|
||||
if (refreshed.state === "retrying") {
|
||||
store.transitionMergeRequestState("FN-MR", "exhausted", { attemptCount: refreshed.attemptCount, lastError: "socket hang up" });
|
||||
}
|
||||
await store.logEntry("FN-MR", "marked merge request exhausted without column rebound: socket hang up");
|
||||
}
|
||||
|
||||
expect(state).toBe("exhausted");
|
||||
expect(logs.at(-1)).toContain("without column rebound");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
import { mkdtempSync } from "node:fs";
|
||||
import { rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { TaskStore } from "@fusion/core";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { ProjectEngine } from "../../project-engine.js";
|
||||
|
||||
describe("FN-5743 hard-cancel merge-request cutover", () => {
|
||||
let rootDir: string;
|
||||
let globalDir: string;
|
||||
let store: TaskStore;
|
||||
|
||||
beforeEach(async () => {
|
||||
rootDir = mkdtempSync(join(tmpdir(), "kb-fn-5743-hard-cancel-"));
|
||||
globalDir = join(rootDir, ".fusion-global");
|
||||
store = new TaskStore(rootDir, globalDir);
|
||||
await store.init();
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
store.close();
|
||||
await rm(rootDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 50 });
|
||||
});
|
||||
|
||||
it("cancels pending merge request on user in-review->todo hard-cancel", async () => {
|
||||
const task = await store.createTask({ description: "FN-5743 hard-cancel" });
|
||||
await store.moveTask(task.id, "todo");
|
||||
await store.moveTask(task.id, "in-progress");
|
||||
await store.handoffToReview(task.id, {
|
||||
ownerAgentId: "agent",
|
||||
evidence: { reason: "fn_task_done", runId: "run-1", agentId: "agent" },
|
||||
});
|
||||
|
||||
store.upsertMergeRequestRecord(task.id, { state: "queued", attemptCount: 1 });
|
||||
store.setCompletionHandoffAcceptedMarker(task.id, { source: "executor:fn_task_done" });
|
||||
|
||||
await store.moveTask(task.id, "todo", { moveSource: "user" });
|
||||
|
||||
expect(store.getMergeRequestRecord(task.id)?.state).toBe("cancelled");
|
||||
expect(store.getCompletionHandoffAcceptedMarker(task.id)).toBeNull();
|
||||
});
|
||||
|
||||
it("does not cancel merge request on engine in-review->todo rebound", async () => {
|
||||
const task = await store.createTask({ description: "FN-5743 engine rebound" });
|
||||
await store.moveTask(task.id, "todo");
|
||||
await store.moveTask(task.id, "in-progress");
|
||||
await store.handoffToReview(task.id, {
|
||||
ownerAgentId: "agent",
|
||||
evidence: { reason: "fn_task_done", runId: "run-2", agentId: "agent" },
|
||||
});
|
||||
|
||||
store.upsertMergeRequestRecord(task.id, { state: "queued", attemptCount: 1 });
|
||||
store.setCompletionHandoffAcceptedMarker(task.id, { source: "executor:fn_task_done" });
|
||||
|
||||
await store.moveTask(task.id, "todo", { moveSource: "engine" as any });
|
||||
|
||||
expect(store.getMergeRequestRecord(task.id)?.state).toBe("queued");
|
||||
expect(store.getCompletionHandoffAcceptedMarker(task.id)).not.toBeNull();
|
||||
});
|
||||
|
||||
it("transient merge retry uses merge-request state transitions without todo rebound", async () => {
|
||||
let state = "running";
|
||||
const fakeStore = {
|
||||
getSettings: vi.fn().mockResolvedValue({ mergeRequestContractShadowEnabled: true }),
|
||||
getMergeRequestRecord: vi.fn(() => ({ state, attemptCount: 0, lastError: null })),
|
||||
transitionMergeRequestState: vi.fn((_taskId: string, toState: string) => {
|
||||
state = toState;
|
||||
}),
|
||||
updateTask: vi.fn().mockResolvedValue(undefined),
|
||||
logEntry: vi.fn().mockResolvedValue(undefined),
|
||||
moveTask: vi.fn(),
|
||||
} as any;
|
||||
|
||||
const retried = await (ProjectEngine.prototype as any).maybeRetryTransientMerge.call(
|
||||
{ shuttingDown: false, internalEnqueueMerge: vi.fn() },
|
||||
fakeStore,
|
||||
"FN-5743",
|
||||
{ id: "FN-5743", mergeTransientRetryCount: 0 },
|
||||
"lease-handoff-failed: target-not-queued",
|
||||
);
|
||||
|
||||
expect(retried).toBe(true);
|
||||
expect(state).toBe("queued");
|
||||
expect(fakeStore.moveTask).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -40,6 +40,7 @@ import {
|
||||
extractFailingTestFiles,
|
||||
} from "./verification-followup-dedup.js";
|
||||
import { isTransientError } from "./transient-error-detector.js";
|
||||
import { classifyTransientMergeError } from "./transient-merge-error-classifier.js";
|
||||
import { TunnelProcessManager } from "./remote-access/tunnel-process-manager.js";
|
||||
import type {
|
||||
ExternalTunnelInfo,
|
||||
@@ -289,6 +290,7 @@ export class ProjectEngine {
|
||||
* Examples: "This operation was aborted", "socket hang up", `server_error`.
|
||||
* After this cap, the task is parked failed for human visibility. */
|
||||
private static readonly MAX_AUTO_MERGE_TRANSIENT_RETRIES = 3;
|
||||
private static readonly MERGE_REQUEST_RETRY_EXHAUSTED_AGE_MS = 30 * 60 * 1000;
|
||||
/** Cap on outer in-review→in-progress bounces caused by deterministic
|
||||
* verification failures during auto-merge. After this many failed merges
|
||||
* for the same task, we stop bouncing it back, mark it failed, and create
|
||||
@@ -1448,6 +1450,27 @@ export class ProjectEngine {
|
||||
const shadowSettings = await store.getSettings();
|
||||
if (shadowSettings.mergeRequestContractShadowEnabled === true) {
|
||||
this.emitMergeRequestShadowDequeueParity(taskId, shadowCandidateTaskId);
|
||||
const mergeRequest = store.getMergeRequestRecord(taskId);
|
||||
if (mergeRequest?.state === "manual-required" || mergeRequest?.state === "cancelled" || mergeRequest?.state === "succeeded" || mergeRequest?.state === "exhausted") {
|
||||
continue;
|
||||
}
|
||||
if (mergeRequest && (mergeRequest.state === "queued" || mergeRequest.state === "retrying")) {
|
||||
if (mergeRequest.state === "retrying") {
|
||||
store.transitionMergeRequestState(taskId, "queued", { attemptCount: mergeRequest.attemptCount, lastError: mergeRequest.lastError });
|
||||
}
|
||||
store.transitionMergeRequestState(taskId, "running", { attemptCount: mergeRequest.attemptCount, lastError: mergeRequest.lastError });
|
||||
}
|
||||
if (mergeRequest?.state === "running") {
|
||||
const ageMs = Date.now() - Date.parse(mergeRequest.updatedAt);
|
||||
if ((mergeRequest.attemptCount ?? 0) >= ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES && ageMs >= ProjectEngine.MERGE_REQUEST_RETRY_EXHAUSTED_AGE_MS) {
|
||||
store.transitionMergeRequestState(taskId, "exhausted", {
|
||||
attemptCount: mergeRequest.attemptCount,
|
||||
lastError: mergeRequest.lastError ?? "merge-request-running-age-cap-exhausted",
|
||||
});
|
||||
await store.logEntry(taskId, "Merge-request retry cap reached in running state; marked merge request exhausted without executor rebound");
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
// pickNextMergeTaskId awaits store.getTask; re-check shutdown so we
|
||||
// don't start a merge whose queue entry was cleared by stop().
|
||||
@@ -2313,6 +2336,32 @@ export class ProjectEngine {
|
||||
continue;
|
||||
}
|
||||
if (this.isTransientMergeRetryExhausted(taskOnErr, errorMsg)) {
|
||||
const settings = await store.getSettings().catch(() => null);
|
||||
const useMergeRequestContract = settings?.mergeRequestContractShadowEnabled === true;
|
||||
if (useMergeRequestContract) {
|
||||
const record = store.getMergeRequestRecord(taskId);
|
||||
if (record && record.state !== "exhausted" && record.state !== "cancelled" && record.state !== "succeeded") {
|
||||
if (record.state === "running") {
|
||||
store.transitionMergeRequestState(taskId, "retrying", {
|
||||
attemptCount: record.attemptCount,
|
||||
lastError: errorMsg,
|
||||
});
|
||||
}
|
||||
const refreshed = store.getMergeRequestRecord(taskId);
|
||||
if (refreshed && refreshed.state === "retrying") {
|
||||
store.transitionMergeRequestState(taskId, "exhausted", {
|
||||
attemptCount: refreshed.attemptCount,
|
||||
lastError: errorMsg,
|
||||
});
|
||||
}
|
||||
}
|
||||
await store.logEntry(
|
||||
taskId,
|
||||
`Auto-merge transient retries exhausted (${ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES}/${ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES}); marked merge request exhausted without column rebound: ${errorMsg}`,
|
||||
"MergeTransientRetryExhausted",
|
||||
);
|
||||
continue;
|
||||
}
|
||||
await store.logEntry(
|
||||
taskId,
|
||||
`Auto-merge transient retries exhausted (${ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES}/${ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES}); parking task as failed: ${errorMsg}`,
|
||||
@@ -2343,6 +2392,32 @@ export class ProjectEngine {
|
||||
continue;
|
||||
}
|
||||
if (this.isTransientMergeRetryExhausted(taskOnErr, errorMsg)) {
|
||||
const settings = await store.getSettings().catch(() => null);
|
||||
const useMergeRequestContract = settings?.mergeRequestContractShadowEnabled === true;
|
||||
if (useMergeRequestContract) {
|
||||
const record = store.getMergeRequestRecord(taskId);
|
||||
if (record && record.state !== "exhausted" && record.state !== "cancelled" && record.state !== "succeeded") {
|
||||
if (record.state === "running") {
|
||||
store.transitionMergeRequestState(taskId, "retrying", {
|
||||
attemptCount: record.attemptCount,
|
||||
lastError: errorMsg,
|
||||
});
|
||||
}
|
||||
const refreshed = store.getMergeRequestRecord(taskId);
|
||||
if (refreshed && refreshed.state === "retrying") {
|
||||
store.transitionMergeRequestState(taskId, "exhausted", {
|
||||
attemptCount: refreshed.attemptCount,
|
||||
lastError: errorMsg,
|
||||
});
|
||||
}
|
||||
}
|
||||
await store.logEntry(
|
||||
taskId,
|
||||
`Auto-merge transient retries exhausted (${ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES}/${ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES}); marked merge request exhausted without column rebound: ${errorMsg}`,
|
||||
"MergeTransientRetryExhausted",
|
||||
);
|
||||
continue;
|
||||
}
|
||||
await store.logEntry(
|
||||
taskId,
|
||||
`Auto-merge transient retries exhausted (${ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES}/${ProjectEngine.MAX_AUTO_MERGE_TRANSIENT_RETRIES}); parking task as failed: ${errorMsg}`,
|
||||
@@ -2388,7 +2463,7 @@ export class ProjectEngine {
|
||||
}
|
||||
|
||||
private isTransientMergeRetryExhausted(task: Task | null, errorMsg: string): boolean {
|
||||
if (!task || !isTransientError(errorMsg)) {
|
||||
if (!task || (!isTransientError(errorMsg) && classifyTransientMergeError(errorMsg) === null)) {
|
||||
return false;
|
||||
}
|
||||
const current = task.mergeTransientRetryCount ?? 0;
|
||||
@@ -2401,7 +2476,7 @@ export class ProjectEngine {
|
||||
taskOnErr: Task | null,
|
||||
errorMsg: string,
|
||||
): Promise<boolean> {
|
||||
if (!taskOnErr || !isTransientError(errorMsg)) {
|
||||
if (!taskOnErr || (!isTransientError(errorMsg) && classifyTransientMergeError(errorMsg) === null)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -2412,6 +2487,25 @@ export class ProjectEngine {
|
||||
|
||||
const nextRetryCount = currentRetries + 1;
|
||||
const delayMs = 5000 * Math.pow(2, currentRetries);
|
||||
const settings = await store.getSettings().catch(() => null);
|
||||
const useMergeRequestContract = settings?.mergeRequestContractShadowEnabled === true;
|
||||
if (useMergeRequestContract) {
|
||||
const record = store.getMergeRequestRecord(taskId);
|
||||
if (record && record.state !== "manual-required" && record.state !== "cancelled" && record.state !== "succeeded" && record.state !== "exhausted") {
|
||||
if (record.state === "running") {
|
||||
store.transitionMergeRequestState(taskId, "retrying", {
|
||||
attemptCount: nextRetryCount,
|
||||
lastError: errorMsg,
|
||||
});
|
||||
}
|
||||
if (store.getMergeRequestRecord(taskId)?.state === "retrying") {
|
||||
store.transitionMergeRequestState(taskId, "queued", {
|
||||
attemptCount: nextRetryCount,
|
||||
lastError: errorMsg,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
await store.updateTask(taskId, {
|
||||
mergeTransientRetryCount: nextRetryCount,
|
||||
status: null,
|
||||
|
||||
@@ -1296,13 +1296,19 @@ export class Scheduler {
|
||||
const filteredScope = await getFilteredFileScope(t.id);
|
||||
if (isCoordinationOnlyTask(t, filteredScope)) continue;
|
||||
if (filteredScope.length === 0) continue;
|
||||
setActiveScopeLease(t.id, filteredScope, "in-review");
|
||||
|
||||
const handoffAccepted = settings.mergeRequestContractShadowEnabled === true
|
||||
? this.store.getCompletionHandoffAcceptedMarker(t.id) !== null
|
||||
: false;
|
||||
if (!handoffAccepted) {
|
||||
setActiveScopeLease(t.id, filteredScope, "in-review");
|
||||
}
|
||||
|
||||
if (settings.mergeRequestContractShadowEnabled === true) {
|
||||
const mergeRequestRecord = this.store.getMergeRequestRecord(t.id);
|
||||
const { shadowExecutorLeaseApplied, shadowMergeLockApplied, shadowLeaseApplied } =
|
||||
computeShadowLeaseParityState(mergeRequestRecord?.state ?? null);
|
||||
if (shadowLeaseApplied !== true) {
|
||||
if (shadowLeaseApplied !== !handoffAccepted) {
|
||||
void this.store.recordRunAuditEvent?.({
|
||||
taskId: t.id,
|
||||
agentId: "scheduler",
|
||||
@@ -1313,7 +1319,7 @@ export class Scheduler {
|
||||
metadata: {
|
||||
taskId: t.id,
|
||||
legacyLeaseColumn: "in-review",
|
||||
legacyLeaseApplied: true,
|
||||
legacyLeaseApplied: !handoffAccepted,
|
||||
shadowLeaseApplied,
|
||||
shadowExecutorLeaseApplied,
|
||||
shadowMergeLockApplied,
|
||||
|
||||
Reference in New Issue
Block a user