FN-8976: prevent concurrent automatic validator runs
Prevent automatic validation from running alongside a fresh manual or non-memoized validator. - Serialize validator admission around a feature-scoped fresh-running check. - Report feature-versus-fingerprint blocking scope and preserve memoization semantics. - Add regression coverage, documentation, and a patch changeset. Files changed: .changeset/fn-8976-validator-admission.md | 7 ++ docs/missions.md | 4 +- .../__tests__/postgres/mission-store.pg.test.ts | 110 +++++++++++++++++++-- .../core/src/async-stores/async-mission-store.ts | 61 ++++++++++-- packages/core/src/missions/mission-types.ts | 2 + .../src/__tests__/mission-execution-loop.test.ts | 10 +- 6 files changed, 171 insertions(+), 23 deletions(-) Fusion-Task-Id: FN-8976 Fusion-Task-Lineage: 61b487e8-5542-4cce-94f7-58d5295fe177 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
7
.changeset/fn-8976-validator-admission.md
Normal file
7
.changeset/fn-8976-validator-admission.md
Normal file
@@ -0,0 +1,7 @@
|
||||
---
|
||||
"@runfusion/fusion": patch
|
||||
---
|
||||
|
||||
summary: Automatic mission validation no longer starts a second run while manual validation is in flight.
|
||||
category: fix
|
||||
dev: Uses a feature-scoped live-run check and exposes optional ValidatorRunAdmission.blockingScope.
|
||||
@@ -793,9 +793,9 @@ Autonomous no-task heartbeat agents may create or delegate implementation work o
|
||||
|
||||
## Validator memoization and failure budget (FN-8694)
|
||||
|
||||
Automatic feature validation is content-addressed by landed SHA, resolved judge provider/model, and exact built prompts. Admission is atomic per project, feature, and fingerprint: a matching running run is not duplicated; the latest terminal history is selected deterministically; static-only passes are reused; and matching failures permit at most three dispatched runs before the feature is blocked. Behavioral or mixed assertions never reuse a pass, but failures are still budgeted.
|
||||
Automatic feature validation is content-addressed by landed SHA, resolved judge provider/model, and exact built prompts. Admission is atomic per project and feature: its in-flight check observes every fresh running validator, including fingerprint-less manual and non-memo automatic runs, while honoring the reaper stale window so a dead run cannot wedge validation. After that liveness check, terminal history is selected deterministically by fingerprint: static-only passes are reused, and matching failures permit at most three dispatched runs before the feature is blocked. Behavioral or mixed assertions never reuse a pass, but failures are still budgeted.
|
||||
|
||||
Every automatic suppression appends one visible `validation memoized` activity event (`running`, `reuse-pass`, or `budget-exhausted`) with fingerprint and referenced run ID where available. Initial exhaustion additionally appends one `validation-stuck` event; later unchanged sweeps append only their memoized event. No synthetic validator run or verdict is created for reuse or exhaustion. Missing landed SHA, fallback checkout, unknown judge identity, and preparation failures fail open to ordinary validation; `error`/`blocked` outcomes are transient. Manual validation bypasses memoization and the budget. Recovery revisits only a feature bearing FN-8694's budget-block provenance: unchanged inputs remain blocked, while a changed prepared fingerprint can be admitted; unrelated blocked/remediation/operator states stay closed.
|
||||
Every automatic suppression appends one visible `validation memoized` activity event (`running`, `reuse-pass`, or `budget-exhausted`) with fingerprint and referenced run ID where available. Initial exhaustion additionally appends one `validation-stuck` event; later unchanged sweeps append only their memoized event. No synthetic validator run or verdict is created for reuse or exhaustion. Missing landed SHA, fallback checkout, unknown judge identity, and preparation failures fail open to ordinary validation; `error`/`blocked` outcomes are transient. Manual validation bypasses memoization and the budget, but its live run blocks a concurrent automatic dispatch. Recovery revisits only a feature bearing FN-8694's budget-block provenance: unchanged inputs remain blocked, while a changed prepared fingerprint can be admitted; unrelated blocked/remediation/operator states stay closed.
|
||||
|
||||
## Spec alignment
|
||||
|
||||
|
||||
@@ -959,23 +959,119 @@ pgTest("MissionStore (PostgreSQL backend mode)", () => {
|
||||
await expect(m.startManualValidatorRun(feature.id)).resolves.toMatchObject({ outcome: "started" });
|
||||
});
|
||||
|
||||
it("characterizes automatic admission's fingerprint-only boundary", async () => {
|
||||
it("refuses automatic-after-manual admission across the fingerprint-less boundary", async () => {
|
||||
/*
|
||||
FNXC:MissionValidation 2026-08-11-03:43:
|
||||
FN-8976 owns this deliberately unchanged gap: a fingerprint-less manual run is not visible to
|
||||
fingerprint-scoped automatic admission. Manual-after-automatic is guarded above; do not widen
|
||||
admitValidatorRun in the manual endpoint fix.
|
||||
FNXC:MissionValidation 2026-08-11-05:38:
|
||||
FN-8976 reconciles the former automatic-after-manual gap. A live fingerprint-less manual run
|
||||
blocks automatic admission without mutating feature state; content-addressed terminal decisions
|
||||
remain fingerprint-scoped after this feature-liveness refusal.
|
||||
*/
|
||||
const m = missions();
|
||||
const mission = await m.createMission({ title: "Fingerprint boundary" });
|
||||
const milestone = await m.addMilestone(mission.id, { title: "MS" });
|
||||
const slice = await m.addSlice(milestone.id, { title: "SL" });
|
||||
const feature = await m.addFeature(slice.id, { title: "F" });
|
||||
await expect(m.startManualValidatorRun(feature.id)).resolves.toMatchObject({ outcome: "started" });
|
||||
await expect(m.admitValidatorRun(feature.id, { inputFingerprint: "fresh-fingerprint", failureBudget: 3, reusePass: false }))
|
||||
const manual = await m.startManualValidatorRun(feature.id);
|
||||
expect(manual).toMatchObject({ outcome: "started", run: { inputFingerprint: undefined, status: "running" } });
|
||||
const before = await m.getFeature(feature.id);
|
||||
const beforeEvents = (await m.getMissionEvents(mission.id, { limit: 100 })).events.length;
|
||||
|
||||
await expect(m.admitValidatorRun(feature.id, { inputFingerprint: "c".repeat(64), failureBudget: 3, reusePass: true }))
|
||||
.resolves.toMatchObject({ outcome: "running", run: { id: manual.run.id }, blockingScope: "feature" });
|
||||
expect((await m.getValidatorRunsByFeature(feature.id)).filter((run) => run.status === "running")).toHaveLength(1);
|
||||
const after = await m.getFeature(feature.id);
|
||||
expect(after).toMatchObject({
|
||||
validatorAttemptCount: before?.validatorAttemptCount,
|
||||
lastValidatorRunId: before?.lastValidatorRunId,
|
||||
loopState: before?.loopState,
|
||||
validationBudgetFingerprint: before?.validationBudgetFingerprint,
|
||||
validationBudgetRunId: before?.validationBudgetRunId,
|
||||
validationBudgetBlockedAt: before?.validationBudgetBlockedAt,
|
||||
});
|
||||
const warnings = (await m.getMissionEvents(mission.id, { limit: 100 })).events
|
||||
.filter((event) => event.description === "validation memoized");
|
||||
expect(warnings).toHaveLength(1);
|
||||
expect((await m.getMissionEvents(mission.id, { limit: 100 })).events).toHaveLength(beforeEvents + 1);
|
||||
expect(warnings[0]?.metadata).toMatchObject({ outcome: "running", featureId: feature.id, runId: manual.run.id, blockingScope: "feature" });
|
||||
expect((await m.getMissionEvents(mission.id, { limit: 100 })).events.some((event) => event.description === "validation-stuck")).toBe(false);
|
||||
});
|
||||
|
||||
it("blocks every fresh feature-scoped running run but ignores a stale run", async () => {
|
||||
const m = missions();
|
||||
const mission = await m.createMission({ title: "Feature scoped automatic admission" });
|
||||
const milestone = await m.addMilestone(mission.id, { title: "MS" });
|
||||
const slice = await m.addSlice(milestone.id, { title: "SL" });
|
||||
const feature = await m.addFeature(slice.id, { title: "F" });
|
||||
const different = await m.startValidatorRun(feature.id, "task_completion", undefined, "different");
|
||||
await expect(m.admitValidatorRun(feature.id, { inputFingerprint: "requested", failureBudget: 3, reusePass: false }))
|
||||
.resolves.toMatchObject({ outcome: "running", run: { id: different.id }, blockingScope: "feature" });
|
||||
await expect(m.admitValidatorRun(feature.id, { inputFingerprint: "different", failureBudget: 3, reusePass: false }))
|
||||
.resolves.toMatchObject({ outcome: "running", run: { id: different.id }, blockingScope: "feature" });
|
||||
await m.completeValidatorRun(different.id, "error");
|
||||
const fallback = await m.startValidatorRun(feature.id, "task_completion");
|
||||
await expect(m.admitValidatorRun(feature.id, { inputFingerprint: "requested", failureBudget: 3, reusePass: false }))
|
||||
.resolves.toMatchObject({ outcome: "running", run: { id: fallback.id }, blockingScope: "feature" });
|
||||
await m.completeValidatorRun(fallback.id, "error");
|
||||
|
||||
const stale = await m.startValidatorRun(feature.id, "task_completion");
|
||||
await h.layer().transactionImmediate(async (tx) => {
|
||||
await tx.update(schema.project.missionValidatorRuns)
|
||||
.set({ startedAt: new Date(Date.now() - 7 * 60 * 60 * 1000).toISOString() })
|
||||
.where(eq(schema.project.missionValidatorRuns.id, stale.id));
|
||||
});
|
||||
await expect(m.admitValidatorRun(feature.id, { inputFingerprint: "requested", failureBudget: 3, reusePass: false }))
|
||||
.resolves.toMatchObject({ outcome: "start" });
|
||||
});
|
||||
|
||||
it("reports the newest fresh run when legacy data has multiple running rows", async () => {
|
||||
const m = missions();
|
||||
const mission = await m.createMission({ title: "Automatic legacy concurrent validation rows" });
|
||||
const milestone = await m.addMilestone(mission.id, { title: "MS" });
|
||||
const slice = await m.addSlice(milestone.id, { title: "SL" });
|
||||
const feature = await m.addFeature(slice.id, { title: "F" });
|
||||
const older = await m.startValidatorRun(feature.id, "task_completion");
|
||||
await h.layer().transactionImmediate(async (tx) => {
|
||||
await tx.update(schema.project.missionValidatorRuns)
|
||||
.set({ startedAt: new Date(Date.now() - 60_000).toISOString() })
|
||||
.where(eq(schema.project.missionValidatorRuns.id, older.id));
|
||||
});
|
||||
const newest = await m.startValidatorRun(feature.id, "task_completion");
|
||||
|
||||
await expect(m.admitValidatorRun(feature.id, { inputFingerprint: "requested", failureBudget: 3, reusePass: false }))
|
||||
.resolves.toMatchObject({ outcome: "running", run: { id: newest.id }, blockingScope: "feature" });
|
||||
});
|
||||
|
||||
it("keeps reuse-pass behind a live fingerprint-less run", async () => {
|
||||
const m = missions();
|
||||
const mission = await m.createMission({ title: "Live run precedes reuse" });
|
||||
const milestone = await m.addMilestone(mission.id, { title: "MS" });
|
||||
const slice = await m.addSlice(milestone.id, { title: "SL" });
|
||||
const feature = await m.addFeature(slice.id, { title: "F" });
|
||||
const passed = await m.startValidatorRun(feature.id, "task_completion", undefined, "reuse");
|
||||
await m.completeValidatorRun(passed.id, "passed");
|
||||
const manual = await m.startManualValidatorRun(feature.id);
|
||||
|
||||
await expect(m.admitValidatorRun(feature.id, { inputFingerprint: "reuse", failureBudget: 3, reusePass: true }))
|
||||
.resolves.toMatchObject({ outcome: "running", run: { id: manual.run.id }, blockingScope: "feature" });
|
||||
expect((await m.getFeature(feature.id))?.status).not.toBe("done");
|
||||
});
|
||||
|
||||
it("serializes concurrent manual and automatic admission", async () => {
|
||||
const m = missions();
|
||||
const mission = await m.createMission({ title: "Concurrent cross-surface admission" });
|
||||
const milestone = await m.addMilestone(mission.id, { title: "MS" });
|
||||
const slice = await m.addSlice(milestone.id, { title: "SL" });
|
||||
const feature = await m.addFeature(slice.id, { title: "F" });
|
||||
|
||||
const [manual, automatic] = await Promise.all([
|
||||
m.startManualValidatorRun(feature.id),
|
||||
m.admitValidatorRun(feature.id, { inputFingerprint: "concurrent", failureBudget: 3, reusePass: false }),
|
||||
]);
|
||||
expect((await m.getValidatorRunsByFeature(feature.id)).filter((run) => run.status === "running")).toHaveLength(1);
|
||||
expect([manual.outcome, automatic.outcome].filter((outcome) => outcome === "started" || outcome === "start")).toHaveLength(1);
|
||||
expect([manual.outcome, automatic.outcome].filter((outcome) => outcome === "already-running" || outcome === "running")).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("does not let reuse-pass or budget exhaustion block a manual validator run", async () => {
|
||||
const m = missions();
|
||||
const mission = await m.createMission({ title: "Terminal automatic outcomes" });
|
||||
|
||||
@@ -1838,13 +1838,37 @@ export class AsyncMissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
taskId, inputFingerprint, startedAt: now, createdAt: now, updatedAt: now };
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:MissionValidation 2026-08-11-05:38:
|
||||
Find the newest live run while callers hold the feature lock. Lock run rows in the same feature
|
||||
then runs order so manual and automatic admission remain serialized across processes.
|
||||
*/
|
||||
private async findBlockingInFlightRun(
|
||||
tx: QueryHandle,
|
||||
featureId: string,
|
||||
now = Date.now(),
|
||||
): Promise<MissionValidatorRun | undefined> {
|
||||
const rows = await tx.select().from(schema.project.missionValidatorRuns).where(and(
|
||||
eq(schema.project.missionValidatorRuns.projectId, missionProjectId()),
|
||||
eq(schema.project.missionValidatorRuns.featureId, featureId),
|
||||
eq(schema.project.missionValidatorRuns.status, "running"),
|
||||
)).orderBy(
|
||||
desc(schema.project.missionValidatorRuns.completedAt),
|
||||
desc(schema.project.missionValidatorRuns.startedAt),
|
||||
desc(schema.project.missionValidatorRuns.createdAt),
|
||||
desc(schema.project.missionValidatorRuns.id),
|
||||
).for("update");
|
||||
const cutoff = now - VALIDATION_INFLIGHT_STALE_MAX_AGE_MS;
|
||||
return rows.map((row) => rowToValidatorRun(row as never))
|
||||
.find((run) => Date.parse(run.startedAt) >= cutoff);
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:MissionValidation 2026-08-11-03:43:
|
||||
Manual validation previously had no in-flight guard: automatic admission is fingerprint-scoped
|
||||
and FN-8947 guarded only repair re-runs. This feature-scoped transaction observes engine-started
|
||||
runs, while runs beyond the reaper window do not wedge the button. A fingerprint-less manual run
|
||||
intentionally remains invisible to fingerprint-scoped automatic admission; FN-8976 owns that
|
||||
tested boundary rather than widening admitValidatorRun here.
|
||||
runs, while runs beyond the reaper window do not wedge the button. FN-8976 shares this predicate
|
||||
with automatic admission so fingerprint-less manual runs cannot create a second live validator.
|
||||
*/
|
||||
async startManualValidatorRun(
|
||||
featureId: string,
|
||||
@@ -1857,10 +1881,7 @@ export class AsyncMissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
)).for("update");
|
||||
const feature = locked[0] ? await getFeature(tx, featureId) : undefined;
|
||||
if (!feature) throw new Error(`Feature ${featureId} not found`);
|
||||
const cutoff = Date.now() - VALIDATION_INFLIGHT_STALE_MAX_AGE_MS;
|
||||
const blockingRun = (await listValidatorRunsByFeature(tx, featureId)).find(
|
||||
(run) => run.status === "running" && Date.parse(run.startedAt) >= cutoff,
|
||||
);
|
||||
const blockingRun = await this.findBlockingInFlightRun(tx, featureId);
|
||||
if (blockingRun) return { outcome: "already-running", run: blockingRun };
|
||||
const run = await this.buildValidatorRun(tx, feature, input.triggerType ?? "manual", input.taskId);
|
||||
await createValidatorRun(tx, run);
|
||||
@@ -1932,6 +1953,14 @@ export class AsyncMissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
)).for("update");
|
||||
const feature = locked[0] ? await getFeature(tx, featureId) : undefined;
|
||||
if (!feature) throw new Error(`Feature ${featureId} not found`);
|
||||
/*
|
||||
FNXC:MissionValidation 2026-08-11-05:38:
|
||||
Automatic admission must observe fingerprint-less manual and non-memo automatic runs so
|
||||
one feature cannot validate concurrently. The shared reaper window prevents a dead run
|
||||
from starving the loop; reuse-pass and failure-budget decisions below stay strictly
|
||||
fingerprint-scoped because they are content-addressed.
|
||||
*/
|
||||
const blockingRun = await this.findBlockingInFlightRun(tx, featureId);
|
||||
const rows = await tx.select().from(schema.project.missionValidatorRuns).where(and(
|
||||
eq(schema.project.missionValidatorRuns.projectId, missionProjectId()),
|
||||
eq(schema.project.missionValidatorRuns.featureId, featureId),
|
||||
@@ -1944,13 +1973,25 @@ export class AsyncMissionStore extends EventEmitter<MissionStoreEvents> {
|
||||
const slice = await getSlice(tx, feature.sliceId);
|
||||
const milestone = slice ? await getMilestone(tx, slice.milestoneId) : undefined;
|
||||
const mission = milestone ? await getMission(tx, milestone.missionId) : undefined;
|
||||
const append = async (outcome: ValidatorRunAdmission["outcome"], run?: MissionValidatorRun, stuck = false) => {
|
||||
const append = async (
|
||||
outcome: ValidatorRunAdmission["outcome"],
|
||||
run?: MissionValidatorRun,
|
||||
stuck = false,
|
||||
blockingScope?: ValidatorRunAdmission["blockingScope"],
|
||||
) => {
|
||||
if (!mission) return;
|
||||
const seq = (await getMaxEventSeq(tx)) + 1;
|
||||
await insertMissionEvent(tx, { id: this.generateId("ME"), missionId: mission.id, eventType: "warning", description: "validation memoized", metadata: { outcome, featureId, fingerprint: input.inputFingerprint, ...(run ? { runId: run.id } : {}) }, timestamp: new Date().toISOString(), seq });
|
||||
await insertMissionEvent(tx, { id: this.generateId("ME"), missionId: mission.id, eventType: "warning", description: "validation memoized", metadata: { outcome, featureId, fingerprint: input.inputFingerprint, ...(run ? { runId: run.id } : {}), ...(blockingScope ? { blockingScope } : {}) }, timestamp: new Date().toISOString(), seq });
|
||||
if (stuck) await insertMissionEvent(tx, { id: this.generateId("ME"), missionId: mission.id, eventType: "warning", description: "validation-stuck", metadata: { featureId, fingerprint: input.inputFingerprint, ...(run ? { runId: run.id } : {}) }, timestamp: new Date().toISOString(), seq: seq + 1 });
|
||||
};
|
||||
if (running) { await append("running", running); return { outcome: "running", run: running }; }
|
||||
if (blockingRun) {
|
||||
await append("running", blockingRun, false, "feature");
|
||||
return { outcome: "running", run: blockingRun, blockingScope: "feature" };
|
||||
}
|
||||
if (running) {
|
||||
await append("running", running, false, "fingerprint");
|
||||
return { outcome: "running", run: running, blockingScope: "fingerprint" };
|
||||
}
|
||||
if (terminal?.status === "passed" && input.reusePass) {
|
||||
await updateFeature(tx, { ...feature, status: "done", loopState: "passed", lastValidatorStatus: "passed", lastValidatorRunId: terminal.id, updatedAt: new Date().toISOString() });
|
||||
statusEvent = await this.recordFeatureStatusChange(tx, feature, "done", { type: "system", id: "mission-store", source: "validator-reuse-pass" });
|
||||
|
||||
@@ -698,6 +698,8 @@ export type ValidatorRunAdmissionOutcome = "start" | "running" | "reuse-pass" |
|
||||
export interface ValidatorRunAdmission {
|
||||
outcome: ValidatorRunAdmissionOutcome;
|
||||
run?: MissionValidatorRun;
|
||||
/** FNXC:MissionValidation 2026-08-11-05:38: Distinguish content and feature-liveness refusals without changing the outcome contract. */
|
||||
blockingScope?: "fingerprint" | "feature";
|
||||
}
|
||||
export interface ValidatorRunAdmissionInput {
|
||||
inputFingerprint: string;
|
||||
|
||||
@@ -1042,10 +1042,10 @@ describe("MissionExecutionLoop", () => {
|
||||
|
||||
it.each(["running", "budget-exhausted"] as const)("preserves automatic %s admission short-circuit without invoking manual admission", async (outcome) => {
|
||||
/*
|
||||
FNXC:MissionValidation 2026-08-11-04:17:
|
||||
FN-8963 changes only manual admission. The automatic loop must retain its existing
|
||||
fingerprint-scoped running and budget-exhausted short-circuits and never call the manual
|
||||
primitive; FN-8976 owns widening automatic admission to observe manual runs.
|
||||
FNXC:MissionValidation 2026-08-11-05:38:
|
||||
FN-8976 makes a feature-scoped live run report the existing running outcome. The automatic
|
||||
loop must dispose its memoized checkout and return without starting, passing, or manually
|
||||
admitting a validator; budget-exhausted retains the same disposal boundary.
|
||||
*/
|
||||
const feature = createMockFeature({ loopState: "implementing", taskId: "FN-001" });
|
||||
missionStore._setFeature(feature);
|
||||
@@ -1068,6 +1068,7 @@ describe("MissionExecutionLoop", () => {
|
||||
rootDir: "/tmp",
|
||||
checkoutMaterializer: { materialize: vi.fn().mockResolvedValue({ dir: "/inspection/landed", dispose }), assertSourceClean: vi.fn() },
|
||||
});
|
||||
const handleValidationPass = vi.spyOn(loop as any, "handleValidationPass");
|
||||
loop.start();
|
||||
|
||||
await loop.processTaskOutcome("FN-001");
|
||||
@@ -1075,6 +1076,7 @@ describe("MissionExecutionLoop", () => {
|
||||
expect(admitValidatorRun).toHaveBeenCalledOnce();
|
||||
expect(startManualValidatorRun).not.toHaveBeenCalled();
|
||||
expect(missionStore.startValidatorRun).not.toHaveBeenCalled();
|
||||
expect(handleValidationPass).not.toHaveBeenCalled();
|
||||
expect(createResolvedAgentSession).not.toHaveBeenCalled();
|
||||
expect(dispose).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user