FN-9168: Record terminal merge-boundary parks in run audit
Add failure-isolated audit visibility for merge-boundary proof failures without changing terminal park behavior. - Emit a redacted event from retry-boundary and graph-terminal park paths. - Bound audit sink latency and isolate absent, throwing, rejecting, or hung sinks. - Add reason-code coverage, catalogue documentation, and a patch changeset. Files changed: .changeset/fn-9168-merge-boundary-audit.md | 7 ++ AGENTS.md | 1 + docs/run-audit.md | 1 + .../src/__tests__/executor-graph-boundary.test.ts | 33 ++++++--- .../__tests__/merge-boundary-unproven-park.test.ts | 78 +++++++++++++++++++- .../executor/emit-merge-boundary-unproven-audit.ts | 82 ++++++++++++++++++++++ .../engine/src/executor/handle-graph-failure.ts | 23 +++++- .../executor/route-graph-merge-failure-to-retry.ts | 36 +++++++++- .../engine/src/executor/workflow-merge-boundary.ts | 38 ++++++++-- .../engine/src/run-audit/run-audit-catalogue.ts | 5 +- packages/engine/src/util/run-audit.ts | 10 +++ 11 files changed, 293 insertions(+), 21 deletions(-) Fusion-Task-Id: FN-9168 Fusion-Task-Lineage: cdeb5c1f-4b22-4531-878e-b18955e6ea5a Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
7
.changeset/fn-9168-merge-boundary-audit.md
Normal file
7
.changeset/fn-9168-merge-boundary-audit.md
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
---
|
||||||
|
"@runfusion/fusion": patch
|
||||||
|
---
|
||||||
|
|
||||||
|
summary: Record merge-boundary proof parks in the run-audit history.
|
||||||
|
category: feature
|
||||||
|
dev: Adds `task:merge-boundary-unproven-parked` at both terminal park sites with closed reason codes and a bounded, failure-isolated emit seam.
|
||||||
@@ -307,6 +307,7 @@ Scoped exception (FN-5819/FN-8823): while project auto-merge is On, shared-branc
|
|||||||
- Workspace (Phase D U1): self-healing emits `task:reconcile-workspace-partial-land` when it re-enqueues a partial/zero-landed workspace task's per-repo land (or parks it `failed` for proven branch absence or exhausted `evidence-unavailable` branch reads), and `task:reconcile-workspace-partial-land-no-action` when `autoMerge:false`, user-pause, a live sub-repo worktree (workspace-aware liveness), or `evidence-unavailable` blocks that backward move. The bounded evidence-exhaustion reason is `evidence-unavailable-exhausted`; audit metadata remains ids/counts/outcomes-only.
|
- Workspace (Phase D U1): self-healing emits `task:reconcile-workspace-partial-land` when it re-enqueues a partial/zero-landed workspace task's per-repo land (or parks it `failed` for proven branch absence or exhausted `evidence-unavailable` branch reads), and `task:reconcile-workspace-partial-land-no-action` when `autoMerge:false`, user-pause, a live sub-repo worktree (workspace-aware liveness), or `evidence-unavailable` blocks that backward move. The bounded evidence-exhaustion reason is `evidence-unavailable-exhausted`; audit metadata remains ids/counts/outcomes-only.
|
||||||
- Workspace (Phase D U1): self-healing emits `task:reclaim-phantom-workspace-land-lease` when it clears a leaked `workspace-repo-land` lease whose owning task is terminal/dead and older than the FN-6736 staleness floor. Archived-role and soft-deleted owners are terminal; live merging, executing, or merge-pending owners are untouched.
|
- Workspace (Phase D U1): self-healing emits `task:reclaim-phantom-workspace-land-lease` when it clears a leaked `workspace-repo-land` lease whose owning task is terminal/dead and older than the FN-6736 staleness floor. Archived-role and soft-deleted owners are terminal; live merging, executing, or merge-pending owners are untouched.
|
||||||
- FN-9164: `worktree:workspace-repo-base-branch` records per-repo base resolution with exactly `taskId`, `repoRelPath`, `stage`, `source`, `outcome`, and optional `fallbackReason`; branch/ref names are deliberately excluded from metadata and `target`, living only in the durable entry and task log.
|
- FN-9164: `worktree:workspace-repo-base-branch` records per-repo base resolution with exactly `taskId`, `repoRelPath`, `stage`, `source`, `outcome`, and optional `fallbackReason`; branch/ref names are deliberately excluded from metadata and `target`, living only in the durable entry and task log.
|
||||||
|
- FN-9168: `task:merge-boundary-unproven-parked` is emitted once per terminal merge-boundary-unproven park at the bounded-retry router and reachable graph terminal-merge park. Metadata is ids/counts/fixed outcomes only (`taskId`, `nodeId`, `failureValue`, `source`, optional `reasonCode`/`missingInstanceCount`, `priorColumn`, `priorStatus`, `outcome`) and never boundary reason prose, foreach instance IDs, or error text. `emitMergeBoundaryUnprovenParked` swallows absent/throwing/rejecting sinks and time-bounds a hung one with `MERGE_BOUNDARY_UNPROVEN_AUDIT_EMIT_TIMEOUT_MS`; late settlement is swallowed and the unref'd timer is cleared, so best-effort telemetry never alters, delays, aborts, or wedges the terminal park.
|
||||||
- FN-9058: `worktree:workspace-main-checkout-edit` records workspace completion guard evidence with ids/counts/fixed outcomes only: task/repo IDs, file/commit counts, evidence or warning reason enum, `taskDoneRetryCount`, and `blocked`/`warned`/`skipped`; never paths, file content, or commit prose.
|
- FN-9058: `worktree:workspace-main-checkout-edit` records workspace completion guard evidence with ids/counts/fixed outcomes only: task/repo IDs, file/commit counts, evidence or warning reason enum, `taskDoneRetryCount`, and `blocked`/`warned`/`skipped`; never paths, file content, or commit prose.
|
||||||
- FN-9059: workspace coordination emits `workspace-lease:*` events for lease acquisition, renewal, release, `fence-published`, `fence-superseded`, `reclaimed`, and `reclaim-refused`, plus `workspace-land-intent:*` events for write-ahead intent lifecycle and `resolve-refused`. Metadata is ids, SHAs, counts, and fixed outcomes only; it never includes a credential-bearing remote URL.
|
- FN-9059: workspace coordination emits `workspace-lease:*` events for lease acquisition, renewal, release, `fence-published`, `fence-superseded`, `reclaimed`, and `reclaim-refused`, plus `workspace-land-intent:*` events for write-ahead intent lifecycle and `resolve-refused`. Metadata is ids, SHAs, counts, and fixed outcomes only; it never includes a credential-bearing remote URL.
|
||||||
- FN-9056: self-healing emits `task:reconcile-orphaned-workspace-worktree` when it reclaims a complete-lane or conservatively-idle failed/soft-deleted workspace entry. It vetoes raw/canonical active paths, task-session/executor/merge liveness, pauses and scheduled recovery; archived rows remain archive-lifecycle-owned. It runs `git worktree prune` even for already-gone paths and deletes only safely-discardable canonical `fusion/<id>` branches. Duplicate, foreign, unowned, or outside-root claims are skipped without git work; one entry-scoped `MAX_STARVATION_DROPS` budget plus settlement bounds retries. Metadata is ids/counts/fixed outcomes: task/repo/path, success/reason/lane, worktree/prune/branch outcomes, and attempt.
|
- FN-9056: self-healing emits `task:reconcile-orphaned-workspace-worktree` when it reclaims a complete-lane or conservatively-idle failed/soft-deleted workspace entry. It vetoes raw/canonical active paths, task-session/executor/merge liveness, pauses and scheduled recovery; archived rows remain archive-lifecycle-owned. It runs `git worktree prune` even for already-gone paths and deletes only safely-discardable canonical `fusion/<id>` branches. Duplicate, foreign, unowned, or outside-root claims are skipped without git work; one entry-scoped `MAX_STARVATION_DROPS` budget plus settlement bounds retries. Metadata is ids/counts/fixed outcomes: task/repo/path, success/reason/lane, worktree/prune/branch outcomes, and attempt.
|
||||||
|
|||||||
@@ -35,6 +35,7 @@ Events that close a task's delivery: blocked/advanced completion parks, already-
|
|||||||
| `task:no-commits-finalize-blocked-incomplete-steps` | Finalize is blocked for a zero-commit task with incomplete workflow steps (FN-6461 lane). |
|
| `task:no-commits-finalize-blocked-incomplete-steps` | Finalize is blocked for a zero-commit task with incomplete workflow steps (FN-6461 lane). |
|
||||||
| `task:empty-merge-finalize-blocked-no-landed-proof` | The AI empty-merge lane vetoes a zero-diff no-op finalize with no landed proof (FN-8141). |
|
| `task:empty-merge-finalize-blocked-no-landed-proof` | The AI empty-merge lane vetoes a zero-diff no-op finalize with no landed proof (FN-8141). |
|
||||||
| `task:finalize-unproven-blocked` | Finalize is blocked because finalization has not been proven against the landing truth. |
|
| `task:finalize-unproven-blocked` | Finalize is blocked because finalization has not been proven against the landing truth. |
|
||||||
|
| `task:merge-boundary-unproven-parked` | A workflow merge boundary could not be proven and its terminal park is recorded with best-effort, time-bounded telemetry that never blocks or stalls the park. |
|
||||||
| `task:finalize-lost-work-blocked` | Finalize is blocked because it would discard work (lost-work guard). |
|
| `task:finalize-lost-work-blocked` | Finalize is blocked because it would discard work (lost-work guard). |
|
||||||
| `task:auto-recover-stale-merger-status` | Self-healing clears a stale merger status left on a finalize path. |
|
| `task:auto-recover-stale-merger-status` | Self-healing clears a stale merger status left on a finalize path. |
|
||||||
|
|
||||||
|
|||||||
@@ -222,19 +222,36 @@ describe("U5a — IR-driven merge boundary (scenario 1)", () => {
|
|||||||
expect(store.moveTask).toHaveBeenCalledWith("FN-B1", "in-review", expect.anything());
|
expect(store.moveTask).toHaveBeenCalledWith("FN-B1", "in-review", expect.anything());
|
||||||
});
|
});
|
||||||
|
|
||||||
it("keeps incomplete foreach coverage and zero expected steps blocked", async () => {
|
it("maps each incomplete merge-boundary proof to a redacted audit code", async () => {
|
||||||
for (const steps of [
|
const cases = [
|
||||||
[{ id: "0", title: "Implement", status: "pending" as const }],
|
{
|
||||||
[],
|
steps: [{ id: "0", title: "Implement", status: "pending" as const }],
|
||||||
]) {
|
workflowStepResults: [],
|
||||||
|
code: "no-node-result",
|
||||||
|
missingInstanceCount: 1,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
steps: [{ id: "0", title: "Implement", status: "pending" as const }],
|
||||||
|
workflowStepResults: [{ workflowStepId: "review", workflowStepName: "Review", source: "node" as const, phase: "pre-merge" as const, status: "pending" as const, completedAt: "2026-01-01" }],
|
||||||
|
code: "non-terminal-node-result",
|
||||||
|
missingInstanceCount: 1,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
steps: [{ id: "0", title: "Implement", status: "pending" as const }],
|
||||||
|
workflowStepResults: [{ workflowStepId: "review", workflowStepName: "Review", source: "node" as const, phase: "pre-merge" as const, status: "passed" as const, completedAt: "2026-01-01" }],
|
||||||
|
code: "missing-foreach-instances",
|
||||||
|
missingInstanceCount: 1,
|
||||||
|
},
|
||||||
|
];
|
||||||
|
for (const { steps, workflowStepResults, code, missingInstanceCount } of cases) {
|
||||||
const { executor, liveTask } = makeExecutor({
|
const { executor, liveTask } = makeExecutor({
|
||||||
selection: { workflowId: "custom:foreach", stepIds: [] }, ir: foreachIr(), steps, workflowStepResults: [],
|
selection: { workflowId: "custom:foreach", stepIds: [] }, ir: foreachIr(), steps, workflowStepResults,
|
||||||
});
|
});
|
||||||
const result = await executor.ensureWorkflowMergeBoundaryTask(
|
const result = await executor.ensureWorkflowMergeBoundaryTask(
|
||||||
liveTask,
|
liveTask,
|
||||||
{ reason: "workflow-merge-boundary", nodeId: "merge", workflowId: "custom:foreach", runId: "r1" },
|
{ reason: "workflow-merge-boundary", nodeId: "merge", workflowId: "custom:foreach", runId: "r1" },
|
||||||
) as { blocked?: { reason: string } };
|
) as { blocked?: { code: string; missingInstanceCount: number } };
|
||||||
expect(result.blocked?.reason).toBe("no pre-merge node result recorded");
|
expect(result.blocked).toMatchObject({ code, missingInstanceCount });
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ import { MERGE_BOUNDARY_UNPROVEN_VALUE, classifyMergePrimitiveResult, runWorkflo
|
|||||||
import { graphFailureValue, isMergeGraphFailure } from "../executor/graph-failure-pure.js";
|
import { graphFailureValue, isMergeGraphFailure } from "../executor/graph-failure-pure.js";
|
||||||
import { isTerminalMergeGraphFailureValue } from "../executor/task-predicates.js";
|
import { isTerminalMergeGraphFailureValue } from "../executor/task-predicates.js";
|
||||||
import { routeGraphMergeFailureToRetry } from "../executor/route-graph-merge-failure-to-retry.js";
|
import { routeGraphMergeFailureToRetry } from "../executor/route-graph-merge-failure-to-retry.js";
|
||||||
|
import { MERGE_BOUNDARY_UNPROVEN_AUDIT_EMIT_TIMEOUT_MS } from "../executor/emit-merge-boundary-unproven-audit.js";
|
||||||
import { shouldHoldActiveFileScopeLease } from "../scheduler.js";
|
import { shouldHoldActiveFileScopeLease } from "../scheduler.js";
|
||||||
|
|
||||||
const task = { id: "FN-9157", column: "in-review", steps: [], dependencies: [], log: [], createdAt: "2026-08-20T00:00:00.000Z", updatedAt: "2026-08-20T00:00:00.000Z", title: "t", description: "", prompt: "# t" } as any;
|
const task = { id: "FN-9157", column: "in-review", steps: [], dependencies: [], log: [], createdAt: "2026-08-20T00:00:00.000Z", updatedAt: "2026-08-20T00:00:00.000Z", title: "t", description: "", prompt: "# t" } as any;
|
||||||
@@ -36,7 +37,7 @@ describe("FN-9157 merge-boundary-unproven terminal routing", () => {
|
|||||||
store: { updateTask, logEntry } as any,
|
store: { updateTask, logEntry } as any,
|
||||||
getRunContextFor: () => undefined,
|
getRunContextFor: () => undefined,
|
||||||
mergeRequester,
|
mergeRequester,
|
||||||
ensureWorkflowMergeBoundaryTask: vi.fn().mockResolvedValue({ task: live, blocked: { reason: "no pre-merge node result recorded" } }),
|
ensureWorkflowMergeBoundaryTask: vi.fn().mockResolvedValue({ task: live, blocked: { reason: "no pre-merge node result recorded", code: "no-node-result", missingInstanceCount: 0 } }),
|
||||||
persistTokenUsage: vi.fn(),
|
persistTokenUsage: vi.fn(),
|
||||||
}, live, graphResult(), undefined);
|
}, live, graphResult(), undefined);
|
||||||
expect(handled).toBe(true);
|
expect(handled).toBe(true);
|
||||||
@@ -45,4 +46,79 @@ describe("FN-9157 merge-boundary-unproven terminal routing", () => {
|
|||||||
expect(logEntry).toHaveBeenCalledWith("FN-9157", expect.stringContaining("retry parked task"), undefined, undefined);
|
expect(logEntry).toHaveBeenCalledWith("FN-9157", expect.stringContaining("retry parked task"), undefined, undefined);
|
||||||
expect(shouldHoldActiveFileScopeLease({ ...live, status: "failed" }, [])).toBe(false);
|
expect(shouldHoldActiveFileScopeLease({ ...live, status: "failed" }, [])).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("emits redacted audit metadata for parked and already-terminal retry boundaries", async () => {
|
||||||
|
const live = { ...task, status: undefined };
|
||||||
|
const updateTask = vi.fn(async (_id, patch) => ({ ...live, ...patch }));
|
||||||
|
const recordRunAuditEvent = vi.fn().mockResolvedValue(undefined);
|
||||||
|
const base = {
|
||||||
|
store: { updateTask, logEntry: vi.fn(), recordRunAuditEvent } as any,
|
||||||
|
getRunContextFor: () => undefined,
|
||||||
|
mergeRequester: vi.fn(),
|
||||||
|
persistTokenUsage: vi.fn(),
|
||||||
|
};
|
||||||
|
await expect(routeGraphMergeFailureToRetry({
|
||||||
|
...base,
|
||||||
|
ensureWorkflowMergeBoundaryTask: vi.fn().mockResolvedValue({ task: live, blocked: { reason: "foreach step instances incomplete at merge boundary: missing secret-a, secret-b", code: "missing-foreach-instances", missingInstanceCount: 2 } }),
|
||||||
|
}, live, graphResult(), undefined)).resolves.toBe(true);
|
||||||
|
expect(recordRunAuditEvent).toHaveBeenCalledTimes(1);
|
||||||
|
expect(recordRunAuditEvent).toHaveBeenLastCalledWith(expect.objectContaining({
|
||||||
|
mutationType: "task:merge-boundary-unproven-parked", target: "FN-9157",
|
||||||
|
metadata: expect.objectContaining({ taskId: "FN-9157", source: "retry-boundary", reasonCode: "missing-foreach-instances", missingInstanceCount: 2, outcome: "parked" }),
|
||||||
|
}));
|
||||||
|
expect(JSON.stringify(recordRunAuditEvent.mock.calls[0][0].metadata)).not.toContain("secret-a");
|
||||||
|
|
||||||
|
const terminal = { ...live, status: "failed", error: "existing" };
|
||||||
|
await expect(routeGraphMergeFailureToRetry({
|
||||||
|
...base,
|
||||||
|
ensureWorkflowMergeBoundaryTask: vi.fn().mockResolvedValue({ task: terminal, blocked: { reason: "no pre-merge node result recorded", code: "no-node-result", missingInstanceCount: 0 } }),
|
||||||
|
}, terminal, graphResult(), undefined)).resolves.toBe(true);
|
||||||
|
expect(recordRunAuditEvent.mock.calls[1][0].metadata.outcome).toBe("already-terminal");
|
||||||
|
expect(updateTask).toHaveBeenCalledTimes(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
["absent", undefined],
|
||||||
|
["rejects", vi.fn().mockRejectedValue(new Error("audit sink down"))],
|
||||||
|
["throws", vi.fn(() => { throw new Error("audit sink boom"); })],
|
||||||
|
])("keeps the terminal park intact when the audit sink %s", async (_name, recordRunAuditEvent) => {
|
||||||
|
const live = { ...task, status: undefined };
|
||||||
|
const updateTask = vi.fn(async (_id, patch) => ({ ...live, ...patch }));
|
||||||
|
const persistTokenUsage = vi.fn();
|
||||||
|
await expect(routeGraphMergeFailureToRetry({
|
||||||
|
store: { updateTask, logEntry: vi.fn(), recordRunAuditEvent } as any,
|
||||||
|
getRunContextFor: () => undefined,
|
||||||
|
mergeRequester: vi.fn(),
|
||||||
|
ensureWorkflowMergeBoundaryTask: vi.fn().mockResolvedValue({ task: live, blocked: { reason: "no pre-merge node result recorded", code: "no-node-result", missingInstanceCount: 0 } }),
|
||||||
|
persistTokenUsage,
|
||||||
|
}, live, graphResult(), undefined)).resolves.toBe(true);
|
||||||
|
expect(updateTask).toHaveBeenCalledWith("FN-9157", expect.objectContaining({ status: "failed", error: expect.stringContaining("MERGE_BOUNDARY_UNPROVEN:") }), undefined);
|
||||||
|
expect(persistTokenUsage).toHaveBeenCalledWith("FN-9157");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("bounds a hung audit sink without skipping token usage", async () => {
|
||||||
|
vi.useFakeTimers();
|
||||||
|
try {
|
||||||
|
const live = { ...task, status: undefined };
|
||||||
|
const updateTask = vi.fn(async (_id, patch) => ({ ...live, ...patch }));
|
||||||
|
const persistTokenUsage = vi.fn();
|
||||||
|
const handled = routeGraphMergeFailureToRetry({
|
||||||
|
store: { updateTask, logEntry: vi.fn(), recordRunAuditEvent: vi.fn(() => new Promise<void>(() => {})) } as any,
|
||||||
|
getRunContextFor: () => undefined,
|
||||||
|
mergeRequester: vi.fn(),
|
||||||
|
ensureWorkflowMergeBoundaryTask: vi.fn().mockResolvedValue({ task: live, blocked: { reason: "no pre-merge node result recorded", code: "no-node-result", missingInstanceCount: 0 } }),
|
||||||
|
persistTokenUsage,
|
||||||
|
}, live, graphResult(), undefined);
|
||||||
|
let settled = false;
|
||||||
|
void handled.then(() => { settled = true; });
|
||||||
|
await Promise.resolve();
|
||||||
|
expect(settled).toBe(false);
|
||||||
|
await vi.advanceTimersByTimeAsync(MERGE_BOUNDARY_UNPROVEN_AUDIT_EMIT_TIMEOUT_MS);
|
||||||
|
await expect(handled).resolves.toBe(true);
|
||||||
|
expect(persistTokenUsage).toHaveBeenCalledWith("FN-9157");
|
||||||
|
expect(updateTask).toHaveBeenCalledWith("FN-9157", expect.objectContaining({ error: expect.stringContaining("MERGE_BOUNDARY_UNPROVEN:") }), undefined);
|
||||||
|
} finally {
|
||||||
|
vi.useRealTimers();
|
||||||
|
}
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
import type { TaskStore } from "@fusion/core";
|
||||||
|
import { executorLog } from "../logger.js";
|
||||||
|
import { generateSyntheticRunId } from "../util/run-audit.js";
|
||||||
|
import type { MergeBoundaryUnprovenReasonCode } from "./workflow-merge-boundary.js";
|
||||||
|
|
||||||
|
export const MERGE_BOUNDARY_UNPROVEN_AUDIT_EMIT_TIMEOUT_MS = 2_000;
|
||||||
|
|
||||||
|
type MergeBoundaryUnprovenParkedAuditPayload = {
|
||||||
|
taskId: string;
|
||||||
|
nodeId: string;
|
||||||
|
failureValue: string;
|
||||||
|
source: "retry-boundary" | "graph-terminal-park";
|
||||||
|
reasonCode?: MergeBoundaryUnprovenReasonCode;
|
||||||
|
missingInstanceCount?: number;
|
||||||
|
priorColumn: string;
|
||||||
|
priorStatus: string | null | undefined;
|
||||||
|
outcome: "parked" | "already-terminal";
|
||||||
|
runId?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
/**
|
||||||
|
* FNXC:RunAudit 2026-08-20-02:00:
|
||||||
|
* FN-9168 requires observability never to regress or stall delivery. The merge-boundary park is
|
||||||
|
* terminal and correct on its own, so absent, throwing, rejecting, or never-settling audit sinks
|
||||||
|
* are swallowed and time-bounded here. Callers await only ordering, never success: an unbounded
|
||||||
|
* await after the terminal write would wedge the executor branch, skipping token persistence and
|
||||||
|
* its return. Failure isolation is swallow-log-and-bound, with no retry, backoff, or queueing.
|
||||||
|
*/
|
||||||
|
export async function emitMergeBoundaryUnprovenParked(
|
||||||
|
store: TaskStore | null | undefined,
|
||||||
|
payload: MergeBoundaryUnprovenParkedAuditPayload,
|
||||||
|
): Promise<void> {
|
||||||
|
const sink = store?.recordRunAuditEvent;
|
||||||
|
if (typeof sink !== "function") return;
|
||||||
|
|
||||||
|
let sinkPromise: Promise<unknown>;
|
||||||
|
try {
|
||||||
|
sinkPromise = Promise.resolve(sink.call(store, {
|
||||||
|
taskId: payload.taskId,
|
||||||
|
agentId: "executor",
|
||||||
|
runId: payload.runId ?? generateSyntheticRunId("merge-boundary-unproven-park", payload.taskId),
|
||||||
|
domain: "database",
|
||||||
|
mutationType: "task:merge-boundary-unproven-parked",
|
||||||
|
target: payload.taskId,
|
||||||
|
metadata: {
|
||||||
|
taskId: payload.taskId,
|
||||||
|
nodeId: payload.nodeId,
|
||||||
|
failureValue: payload.failureValue,
|
||||||
|
source: payload.source,
|
||||||
|
...(payload.reasonCode === undefined ? {} : { reasonCode: payload.reasonCode }),
|
||||||
|
...(payload.missingInstanceCount === undefined ? {} : { missingInstanceCount: payload.missingInstanceCount }),
|
||||||
|
priorColumn: payload.priorColumn,
|
||||||
|
priorStatus: payload.priorStatus ?? null,
|
||||||
|
outcome: payload.outcome,
|
||||||
|
},
|
||||||
|
}));
|
||||||
|
} catch {
|
||||||
|
executorLog.warn("[run-audit] failed to record task:merge-boundary-unproven-parked");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Observe late rejection before the bounded wait returns so it cannot become unhandled.
|
||||||
|
void sinkPromise.catch(() => undefined);
|
||||||
|
await new Promise<void>((resolve) => {
|
||||||
|
const timer = setTimeout(() => {
|
||||||
|
executorLog.warn("[run-audit] timed out recording task:merge-boundary-unproven-parked");
|
||||||
|
resolve();
|
||||||
|
}, MERGE_BOUNDARY_UNPROVEN_AUDIT_EMIT_TIMEOUT_MS);
|
||||||
|
timer.unref?.();
|
||||||
|
void sinkPromise.then(
|
||||||
|
() => {
|
||||||
|
clearTimeout(timer);
|
||||||
|
resolve();
|
||||||
|
},
|
||||||
|
() => {
|
||||||
|
clearTimeout(timer);
|
||||||
|
executorLog.warn("[run-audit] failed to record task:merge-boundary-unproven-parked");
|
||||||
|
resolve();
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -30,6 +30,8 @@ import { getPromptPath } from "../execution/spec-staleness.js";
|
|||||||
import { moveTaskToReplanColumn, resolveReplanTargetColumn } from "../execution/replan-target.js";
|
import { moveTaskToReplanColumn, resolveReplanTargetColumn } from "../execution/replan-target.js";
|
||||||
import { executorLog } from "../logger.js";
|
import { executorLog } from "../logger.js";
|
||||||
import { generateSyntheticRunId, type EngineRunContext } from "../util/run-audit.js";
|
import { generateSyntheticRunId, type EngineRunContext } from "../util/run-audit.js";
|
||||||
|
import { MERGE_BOUNDARY_UNPROVEN_VALUE } from "../workflows/workflow-merge-nodes.js";
|
||||||
|
import { emitMergeBoundaryUnprovenParked } from "./emit-merge-boundary-unproven-audit.js";
|
||||||
import { PAUSE_ABORT_PARK_ERROR_MARKER, PAUSE_ABORT_PARK_OPERATOR_MARKER } from "../self-healing.js";
|
import { PAUSE_ABORT_PARK_ERROR_MARKER, PAUSE_ABORT_PARK_OPERATOR_MARKER } from "../self-healing.js";
|
||||||
import {
|
import {
|
||||||
graphFailureValue,
|
graphFailureValue,
|
||||||
@@ -932,9 +934,28 @@ export async function handleGraphFailure(
|
|||||||
const message = `Workflow graph terminal merge failure at node '${failedNode ?? "unknown"}' (${failureValue}) — operator action required`;
|
const message = `Workflow graph terminal merge failure at node '${failedNode ?? "unknown"}' (${failureValue}) — operator action required`;
|
||||||
executorLog.warn(`${task.id}: ${message}`);
|
executorLog.warn(`${task.id}: ${message}`);
|
||||||
await deps.store.logEntry(task.id, message, undefined, deps.getRunContextFor(task.id));
|
await deps.store.logEntry(task.id, message, undefined, deps.getRunContextFor(task.id));
|
||||||
if (live.status == null && live.error == null) {
|
const outcome = live.status == null && live.error == null ? "parked" as const : "already-terminal" as const;
|
||||||
|
if (outcome === "parked") {
|
||||||
await deps.store.updateTask(task.id, { error: message, status: "failed" }, deps.getRunContextFor(task.id));
|
await deps.store.updateTask(task.id, { error: message, status: "failed" }, deps.getRunContextFor(task.id));
|
||||||
}
|
}
|
||||||
|
if (failureValue === MERGE_BOUNDARY_UNPROVEN_VALUE) {
|
||||||
|
/*
|
||||||
|
FNXC:RunAudit 2026-08-20-02:00:
|
||||||
|
FN-9168 records this reachable graph-terminal merge-boundary-unproven park exactly once.
|
||||||
|
Other terminal merge failures remain unchanged. The bounded emitter contains audit failure
|
||||||
|
and hangs after the status write, so observability cannot alter or wedge the park.
|
||||||
|
*/
|
||||||
|
await emitMergeBoundaryUnprovenParked(deps.store, {
|
||||||
|
taskId: task.id,
|
||||||
|
nodeId: failedNode ?? "unknown",
|
||||||
|
failureValue,
|
||||||
|
source: "graph-terminal-park",
|
||||||
|
priorColumn: live.column,
|
||||||
|
priorStatus: live.status,
|
||||||
|
outcome,
|
||||||
|
runId: deps.getRunContextFor(task.id)?.runId,
|
||||||
|
});
|
||||||
|
}
|
||||||
await deps.persistTokenUsage(task.id);
|
await deps.persistTokenUsage(task.id);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,6 +13,8 @@ import { graphFailureValue } from "./graph-failure-pure.js";
|
|||||||
import type { EngineRunContext } from "../util/run-audit.js";
|
import type { EngineRunContext } from "../util/run-audit.js";
|
||||||
import { executorLog } from "../logger.js";
|
import { executorLog } from "../logger.js";
|
||||||
import { MERGE_BOUNDARY_UNPROVEN_VALUE } from "../workflows/workflow-merge-nodes.js";
|
import { MERGE_BOUNDARY_UNPROVEN_VALUE } from "../workflows/workflow-merge-nodes.js";
|
||||||
|
import { emitMergeBoundaryUnprovenParked } from "./emit-merge-boundary-unproven-audit.js";
|
||||||
|
import type { MergeBoundaryUnprovenReasonCode } from "./workflow-merge-boundary.js";
|
||||||
|
|
||||||
export type RouteGraphMergeFailureToRetryDeps = {
|
export type RouteGraphMergeFailureToRetryDeps = {
|
||||||
store: TaskStore;
|
store: TaskStore;
|
||||||
@@ -21,7 +23,14 @@ export type RouteGraphMergeFailureToRetryDeps = {
|
|||||||
ensureWorkflowMergeBoundaryTask: (
|
ensureWorkflowMergeBoundaryTask: (
|
||||||
live: TaskDetail,
|
live: TaskDetail,
|
||||||
opts: { reason: string; nodeId: string; workflowId: string; runId: string },
|
opts: { reason: string; nodeId: string; workflowId: string; runId: string },
|
||||||
) => Promise<{ task: TaskDetail; blocked?: { reason: string } }>;
|
) => Promise<{
|
||||||
|
task: TaskDetail;
|
||||||
|
blocked?: {
|
||||||
|
reason: string;
|
||||||
|
code: MergeBoundaryUnprovenReasonCode;
|
||||||
|
missingInstanceCount: number;
|
||||||
|
};
|
||||||
|
}>;
|
||||||
persistTokenUsage: (taskId: string) => Promise<void>;
|
persistTokenUsage: (taskId: string) => Promise<void>;
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -52,15 +61,36 @@ export async function routeGraphMergeFailureToRetry(
|
|||||||
work, rather than silently retaining an in-review blocker.
|
work, rather than silently retaining an in-review blocker.
|
||||||
*/
|
*/
|
||||||
if (mergeBoundary.blocked) {
|
if (mergeBoundary.blocked) {
|
||||||
const reason = mergeBoundary.blocked.reason;
|
const { reason, code, missingInstanceCount } = mergeBoundary.blocked;
|
||||||
await deps.store.logEntry(live.id, `Workflow merge boundary retry parked task: ${reason}`, undefined, deps.getRunContextFor(live.id));
|
await deps.store.logEntry(live.id, `Workflow merge boundary retry parked task: ${reason}`, undefined, deps.getRunContextFor(live.id));
|
||||||
if (mergeBoundary.task.status !== "failed" || !mergeBoundary.task.error) {
|
const outcome = mergeBoundary.task.status !== "failed" || !mergeBoundary.task.error
|
||||||
|
? "parked" as const
|
||||||
|
: "already-terminal" as const;
|
||||||
|
if (outcome === "parked") {
|
||||||
await deps.store.updateTask(
|
await deps.store.updateTask(
|
||||||
live.id,
|
live.id,
|
||||||
{ status: "failed", error: `${MERGE_BOUNDARY_UNPROVEN_VALUE.toUpperCase().replaceAll("-", "_")}: ${reason}` },
|
{ status: "failed", error: `${MERGE_BOUNDARY_UNPROVEN_VALUE.toUpperCase().replaceAll("-", "_")}: ${reason}` },
|
||||||
deps.getRunContextFor(live.id),
|
deps.getRunContextFor(live.id),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
/*
|
||||||
|
FNXC:RunAudit 2026-08-20-02:00:
|
||||||
|
FN-9168 records exactly one terminal merge-boundary-unproven park here. The boundary
|
||||||
|
helper's blocked return is not a park and remains silent; its bounded audit seam contains
|
||||||
|
failure and hangs, so telemetry cannot delay or alter this terminal write or return path.
|
||||||
|
*/
|
||||||
|
await emitMergeBoundaryUnprovenParked(deps.store, {
|
||||||
|
taskId: live.id,
|
||||||
|
nodeId: failedNode,
|
||||||
|
failureValue: MERGE_BOUNDARY_UNPROVEN_VALUE,
|
||||||
|
source: "retry-boundary",
|
||||||
|
reasonCode: code,
|
||||||
|
missingInstanceCount,
|
||||||
|
priorColumn: live.column,
|
||||||
|
priorStatus: live.status,
|
||||||
|
outcome,
|
||||||
|
runId: deps.getRunContextFor(live.id)?.runId,
|
||||||
|
});
|
||||||
await deps.persistTokenUsage(live.id);
|
await deps.persistTokenUsage(live.id);
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,9 +17,18 @@ export type WorkflowMergeBoundaryProof = {
|
|||||||
missingInstanceIds: string[];
|
missingInstanceIds: string[];
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export type MergeBoundaryUnprovenReasonCode =
|
||||||
|
| "no-node-result"
|
||||||
|
| "non-terminal-node-result"
|
||||||
|
| "missing-foreach-instances";
|
||||||
|
|
||||||
export type WorkflowMergeBoundaryResult = {
|
export type WorkflowMergeBoundaryResult = {
|
||||||
task: TaskDetail;
|
task: TaskDetail;
|
||||||
blocked?: { reason: string };
|
blocked?: {
|
||||||
|
reason: string;
|
||||||
|
code: MergeBoundaryUnprovenReasonCode;
|
||||||
|
missingInstanceCount: number;
|
||||||
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
export type WorkflowMergeBoundaryDeps = {
|
export type WorkflowMergeBoundaryDeps = {
|
||||||
@@ -81,13 +90,28 @@ export async function ensureWorkflowMergeBoundaryTask(
|
|||||||
*/
|
*/
|
||||||
const mergeProof = await deps.evaluateWorkflowMergeBoundary(live, metadata.runId);
|
const mergeProof = await deps.evaluateWorkflowMergeBoundary(live, metadata.runId);
|
||||||
if (mergeProof.hasForeachStepExecute && !mergeProof.complete) {
|
if (mergeProof.hasForeachStepExecute && !mergeProof.complete) {
|
||||||
const reason = !mergeProof.hasRelevantNodeResult
|
const blocked = !mergeProof.hasRelevantNodeResult
|
||||||
? "no pre-merge node result recorded"
|
? { reason: "no pre-merge node result recorded", code: "no-node-result" as const }
|
||||||
: !mergeProof.allResultsTerminal
|
: !mergeProof.allResultsTerminal
|
||||||
? `non-terminal pre-merge node result ${mergeProof.nonTerminalResult?.workflowStepId ?? "unknown"} (${mergeProof.nonTerminalResult?.status ?? "unknown"})`
|
? {
|
||||||
: `foreach step instances incomplete at merge boundary: missing ${mergeProof.missingInstanceIds.join(", ")}`;
|
reason: `non-terminal pre-merge node result ${mergeProof.nonTerminalResult?.workflowStepId ?? "unknown"} (${mergeProof.nonTerminalResult?.status ?? "unknown"})`,
|
||||||
await deps.store.logEntry(live.id, `Workflow merge boundary blocked: ${reason}`, undefined, deps.getRunContextFor(live.id));
|
code: "non-terminal-node-result" as const,
|
||||||
return { task: live, blocked: { reason } };
|
}
|
||||||
|
: {
|
||||||
|
reason: `foreach step instances incomplete at merge boundary: missing ${mergeProof.missingInstanceIds.join(", ")}`,
|
||||||
|
code: "missing-foreach-instances" as const,
|
||||||
|
};
|
||||||
|
/*
|
||||||
|
FNXC:RunAudit 2026-08-20-02:00:
|
||||||
|
Boundary reason prose can contain foreach instance IDs and node-result status text. Run-audit
|
||||||
|
metadata must remain ids/counts/outcomes-only, so terminal parks receive this closed code and
|
||||||
|
missing-instance count rather than this human-readable reason.
|
||||||
|
*/
|
||||||
|
await deps.store.logEntry(live.id, `Workflow merge boundary blocked: ${blocked.reason}`, undefined, deps.getRunContextFor(live.id));
|
||||||
|
return {
|
||||||
|
task: live,
|
||||||
|
blocked: { ...blocked, missingInstanceCount: mergeProof.missingInstanceIds.length },
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
if (deps.shouldCompleteChecklistAtWorkflowMerge(live, mergeProof)) {
|
if (deps.shouldCompleteChecklistAtWorkflowMerge(live, mergeProof)) {
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ export const DELIVERY_PIPELINE_RUN_AUDIT_EVENTS_LITERALS = [
|
|||||||
"task:no-commits-finalize-blocked-incomplete-steps",
|
"task:no-commits-finalize-blocked-incomplete-steps",
|
||||||
"task:empty-merge-finalize-blocked-no-landed-proof",
|
"task:empty-merge-finalize-blocked-no-landed-proof",
|
||||||
"task:finalize-unproven-blocked",
|
"task:finalize-unproven-blocked",
|
||||||
|
"task:merge-boundary-unproven-parked",
|
||||||
"task:finalize-lost-work-blocked",
|
"task:finalize-lost-work-blocked",
|
||||||
"task:auto-recover-stale-merger-status",
|
"task:auto-recover-stale-merger-status",
|
||||||
|
|
||||||
@@ -69,7 +70,7 @@ export const DELIVERY_PIPELINE_RUN_AUDIT_EVENTS_LITERALS = [
|
|||||||
] as const;
|
] as const;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The 32-event literal union of the curated catalogue. `as const` preserves the literal element
|
* The 31-event literal union of the curated catalogue. `as const` preserves the literal element
|
||||||
* tuples so the notes map can be keyed by exactly the catalogued events (rather than widening to the
|
* tuples so the notes map can be keyed by exactly the catalogued events (rather than widening to the
|
||||||
* full `DatabaseMutationType` union). The exported array below is still typed as
|
* full `DatabaseMutationType` union). The exported array below is still typed as
|
||||||
* `Readonly<DatabaseMutationType[]>`, so member-validity remains compile-time-enforced: assigning
|
* `Readonly<DatabaseMutationType[]>`, so member-validity remains compile-time-enforced: assigning
|
||||||
@@ -109,6 +110,8 @@ export const DELIVERY_PIPELINE_RUN_AUDIT_EVENT_NOTES: Readonly<Record<DeliveryPi
|
|||||||
"The AI empty-merge lane vetoes a zero-diff no-op finalize with no landed proof (FN-8141).",
|
"The AI empty-merge lane vetoes a zero-diff no-op finalize with no landed proof (FN-8141).",
|
||||||
"task:finalize-unproven-blocked":
|
"task:finalize-unproven-blocked":
|
||||||
"Finalize is blocked because finalization has not been proven against the landing truth.",
|
"Finalize is blocked because finalization has not been proven against the landing truth.",
|
||||||
|
"task:merge-boundary-unproven-parked":
|
||||||
|
"A terminal merge-boundary proof failure is parked with bounded best-effort audit telemetry.",
|
||||||
"task:finalize-lost-work-blocked":
|
"task:finalize-lost-work-blocked":
|
||||||
"Finalize is blocked because it would discard work (lost-work guard).",
|
"Finalize is blocked because it would discard work (lost-work guard).",
|
||||||
"task:auto-recover-stale-merger-status":
|
"task:auto-recover-stale-merger-status":
|
||||||
|
|||||||
@@ -846,6 +846,16 @@ export type DatabaseMutationType =
|
|||||||
| "task:in-review-stall-deadlock-disposed"
|
| "task:in-review-stall-deadlock-disposed"
|
||||||
| "task:in-review-stall-terminal-provider-error"
|
| "task:in-review-stall-terminal-provider-error"
|
||||||
| "task:finalize-unproven-blocked"
|
| "task:finalize-unproven-blocked"
|
||||||
|
/**
|
||||||
|
* FNXC:RunAudit 2026-08-20-02:02:
|
||||||
|
* Records one terminal park when a workflow merge boundary cannot be proven, at the retry
|
||||||
|
* boundary or graph-terminal park. Metadata is { taskId, nodeId, failureValue, source,
|
||||||
|
* reasonCode?, missingInstanceCount?, priorColumn, priorStatus, outcome }; it is strictly
|
||||||
|
* ids/counts/outcomes-only and never includes reason prose, instance IDs, or error text.
|
||||||
|
* This is best-effort telemetry: an absent, failed, or hung write must not alter, block, or
|
||||||
|
* stall the terminal park.
|
||||||
|
*/
|
||||||
|
| "task:merge-boundary-unproven-parked"
|
||||||
/**
|
/**
|
||||||
* FN-5490/FN-5517/FN-5526/FN-5540 lost-work guard: the merger or self-heal
|
* FN-5490/FN-5517/FN-5526/FN-5540 lost-work guard: the merger or self-heal
|
||||||
* sweep refused to finalize a task as no-op because its record claimed
|
* sweep refused to finalize a task as no-op because its record claimed
|
||||||
|
|||||||
Reference in New Issue
Block a user