FN-9140: make Vitest timeout ownership surveys deterministic
Replace ambiguous setup-boundary observations with calibrated, repeatable timeout ownership evidence. - record fixture lifecycle events in an append-only JSONL ledger with process-safe ordering - classify four timeout-budget arms across repeated isolate-mode cells and fail closed on incomplete evidence - expand connectionless unit coverage and document the terminal insufficient-data survey result Files changed: .../test-failures/postgres-ddl-admission-bound.md | 2 + .../vitest-setup-boundary-timeout-ownership.md | 37 +++ docs/testing.md | 2 +- scripts/__tests__/pg-setup-boundary-probe.test.mjs | 195 ++++++++---- scripts/pg-setup-boundary-probe.mjs | 341 ++++++++++++--------- 5 files changed, 371 insertions(+), 206 deletions(-) Fusion-Task-Id: FN-9140 Fusion-Task-Lineage: 9c6970b8-af80-400a-b2b2-49718d4fe87f Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
@@ -48,6 +48,8 @@ The unavailable Step-0 fields are deliberately not backfilled or inferred: no sa
|
||||
|
||||
The retained `pg-setup-participation.ts` signal is connectionless and explicit (`FUSION_PG_TEST_SETUP_PARTICIPANT=1`, overridden by `FUSION_PG_TEST_SKIP=1`), preserving non-PG setup inertness. `scripts/pg-setup-boundary-probe.mjs` and `scripts/pg-preadmission-campaign.mjs` remain as repeatable measurement tooling. Any successor must first prove ordering and off-budget ownership, keep the probe default-off and non-consumable by the harness, use interleaved one-shell measurements, and treat incomplete data as rejection.
|
||||
|
||||
FN-9140 subsequently replaced the rejected stdout survey with calibrated JSONL evidence, but its corrected full run did not complete within the hosted execution boundary and therefore records `insufficient-data`; see [Vitest setup-boundary timeout ownership](vitest-setup-boundary-timeout-ownership.md). The PostgreSQL pre-admission probe and campaign execution remain unauthorized. This successor pointer does not revise FN-9139's original honest insufficient-data result.
|
||||
|
||||
<!-- FNXC:PgDdlLaneMetric 2026-08-17-00:59: FN-9134 must establish a green, drift-resistant control band before another structural DDL candidate is allowed to claim improvement. An invalid control is terminal insufficient data, not a reason to tune timeouts or rerun unfavorable samples. -->
|
||||
|
||||
<!-- FNXC:PgDdlLaneMetric 2026-08-17-02:55: The required seven-pair interleaved campaign must remain the terminal evidence. Zeroed inline watchdogs only reflect off-hook execution, while green lanes and zero survivors decide whether a candidate can ship. -->
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
---
|
||||
category: test-failures
|
||||
module: testing
|
||||
date: 2026-08-17
|
||||
problem_type: lifecycle_timeout_ownership
|
||||
component: Vitest setup boundaries
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "Considering work before PostgreSQL test execution"
|
||||
tags:
|
||||
- vitest
|
||||
- timeouts
|
||||
- postgres
|
||||
---
|
||||
|
||||
# Vitest setup-boundary timeout ownership survey
|
||||
|
||||
FN-9139's stdout-marker survey could not distinguish an unrelated failure from a timeout, used two files with two workers, ordered fork output by line position, and treated completion under a generous budget as off-budget evidence. FN-9140 replaces it with a connectionless temporary fixture that appends JSONL ledger events, repeats every cell, and applies a falsifying four-arm budget matrix.
|
||||
|
||||
## Scoped runner and matrix
|
||||
|
||||
The attempted run used Vitest **4.1.10**, Node **v26.3.0**, `pool: "forks"`, 2 workers, 6 files, 3 repeats, both isolate modes, and a 50ms cross-process ordering margin. `D=4000ms`; `SMALL=1000ms`; `LARGE=20000ms`; each child had a derived 74000ms cap. Arm A uses both SMALL budgets, B only test SMALL, C only hook SMALL, and R both LARGE. A pass counts only with an `end - start` ledger duration of at least 3600ms.
|
||||
|
||||
The original recorded table was invalidated during code review: it retained only arm R's summary, so it could not prove that arm A's ordering or granularity agreed across repeats. The corrected survey retains every arm's outcome, ordering, and granularity; setup callbacks now retain Vitest's active test path, while a setup boundary without a supported file identity is explicitly `indeterminate` rather than inferred from recycled worker PIDs. Runner-resolution failures likewise become retained `failed-unclassified` calibration/cell evidence and an `insufficient-data` report. Its one permitted full execution exceeded the hosted 720-second command boundary before it wrote the JSON report. No complete fixed-instrument calibration or boundary cell is therefore available; the prior table is deliberately not repeated here as evidence.
|
||||
|
||||
| Boundary | Isolate | A/B/C/R | Ownership | Granularity | Ordering | Deterministic |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| global setup | true/false | unavailable — fixed-instrument survey did not complete | unavailable | unavailable | unavailable | unavailable |
|
||||
| setup top-level await | true/false | unavailable — fixed-instrument survey did not complete | unavailable | unavailable | unavailable | unavailable |
|
||||
| setup beforeAll | true/false | unavailable — fixed-instrument survey did not complete | unavailable | unavailable | unavailable | unavailable |
|
||||
| per-file beforeAll | true/false | unavailable — fixed-instrument survey did not complete | unavailable | unavailable | unavailable | unavailable |
|
||||
|
||||
## Terminal verdict: `insufficient-data`
|
||||
|
||||
The corrected instrument has no complete calibrated result, so it cannot establish an off-budget, before-first-test boundary. The PostgreSQL pre-admission probe remains out of scope and campaign execution remains unauthorized. A successor may not reinterpret the prior report, rerun this survey toward a preferred result, or change timeouts, retries, quarantine entries, worker caps, or harness wiring as a workaround.
|
||||
|
||||
No repository timeout, retry, quarantine, worker cap, or harness wiring changed. The SMALL/LARGE values exist exclusively in the generated temporary Vitest config.
|
||||
@@ -39,7 +39,7 @@ The initial harness wiring made a 27-worker PostgreSQL directory run worse, even
|
||||
|
||||
`FUSION_PG_TEST_PREADMISSION_PROBE=1` is reserved for a default-off, observation-only measurement probe and is consulted only by a participating lane. A boundary probe must publish no token, lease, registry, singleton, or environment value that the harness can consume: coupling it would be an unmeasured admission rewiring, not a boundary experiment. `FUSION_PG_TEST_TEARDOWN_DIAGNOSTICS=1` enables the existing JSONL measurement channel in both arms.
|
||||
|
||||
Use `scripts/pg-setup-boundary-probe.mjs` to survey setup boundaries before changing a shared setup hook. Its fixture reports boundary granularity and ordering; it never runs a repository suite. The loaded campaign is driven only by `scripts/pg-preadmission-campaign.mjs`, which alternates control and candidate arms, samples one shell's diagnostics, and enforces a five-minute run limit and sixty-minute campaign limit. Collect at least five valid runs per arm. The candidate is affordable only when its median wall time is no worse than the control maximum, its maximum failed-file count is no worse than control, its peak backend footprint is at most one above control, and it records no probe degradation. Fewer than five valid samples, invalid candidate probe records, a capped campaign, or an unavailable environment is deterministically `insufficient-data`, not a reason to tune timeouts or repeat unfavorable results.
|
||||
Use `scripts/pg-setup-boundary-probe.mjs` to survey setup boundaries before changing a shared setup hook. Its temporary, connectionless fixture uses a falsifying four-arm budget matrix: only a duration-verified `both-small` pass proves off-budget work; a reference-only pass is `completed-within-budget`. Every cell is repeated, calibration controls must pass, and unclassified, timed-out, flapping, or indeterminate data is `insufficient-data`. The ledger compares monotonic clocks only within a process and requires an epoch separation margin across processes. See [the recorded ownership survey](solutions/test-failures/vitest-setup-boundary-timeout-ownership.md). The campaign driver is executed only by a successor after that prerequisite is established; its pure unit suite is always safe to run. The loaded campaign is driven only by `scripts/pg-preadmission-campaign.mjs`, which alternates control and candidate arms, samples one shell's diagnostics, and enforces a five-minute run limit and sixty-minute campaign limit. Collect at least five valid runs per arm. The candidate is affordable only when its median wall time is no worse than the control maximum, its maximum failed-file count is no worse than control, its peak backend footprint is at most one above control, and it records no probe degradation. Fewer than five valid samples, invalid candidate probe records, a capped campaign, or an unavailable environment is deterministically `insufficient-data`, not a reason to tune timeouts or repeat unfavorable results.
|
||||
|
||||
<!-- FNXC:EngineTests 2026-07-08-03:00: FN-7667 decouples the engine-core gate's module graph from full-barrel growth so new feature modules don't silently inflate every gate fork's transform/import cost. -->
|
||||
**Gate-safe `@fusion/core` barrel:** the `engine-core` project resolves `@fusion/core` to `packages/core/src/index.gate.ts` (a project-scoped `resolve.alias`, not the root map), not the full `packages/core/src/index.ts` barrel. `index.gate.ts` is a byte-for-byte copy of the full barrel minus the `export ... from` statements for modules added to the barrel after the last re-audit baseline — i.e. it re-exports everything the full barrel does except genuinely new, gate-irrelevant feature modules (diffed against the prior baseline commit's barrel, not hand-picked from what gate *test* files import — production modules under test pull in far more of the barrel transitively than their own imports suggest). `engine-default`/`engine-reliability`/`engine-slow` are unaffected and keep resolving the full barrel. `@fusion/engine` is untouched (no gate file imports it). When adding a new barrel module that no gate test needs, mirror the exclusion in `index.gate.ts` rather than letting gate wall-time grow — see the FNXC comment at the top of `index.gate.ts` and `packages/engine/vitest.config.ts`'s `engine-core` project for the audit procedure.
|
||||
|
||||
@@ -1,92 +1,153 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import test from "node:test";
|
||||
import { URL } from "node:url";
|
||||
|
||||
import {
|
||||
classifyVitestOutput,
|
||||
classifyArm,
|
||||
decideSurveyVerdict,
|
||||
deriveBudgets,
|
||||
fixtureFiles,
|
||||
parseLedger,
|
||||
parseProbeArgs,
|
||||
parseProbeEvents,
|
||||
summarizeBoundary,
|
||||
resolveOwnership,
|
||||
resolveVitestRunner,
|
||||
runFixtureCell,
|
||||
runSurvey,
|
||||
summarizeLedger,
|
||||
summarizeRepeats,
|
||||
} from "../pg-setup-boundary-probe.mjs";
|
||||
|
||||
const event = (overrides = {}) => ({ type: "boundary", boundary: "sample", phase: "start", pid: 1, file: "one", epochMs: 1000, monotonicNs: "100", ...overrides });
|
||||
const options = { workers: 2, files: 6, repeats: 3, isolate: true, delayMs: 4000, orderingMarginMs: 50, runTimeoutMs: 60_000 };
|
||||
|
||||
/*
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-03:25:
|
||||
FN-9139 keeps this report-only survey's unit proof connectionless: fixture strings validate the
|
||||
parser and summary contract without spawning Vitest or making a PostgreSQL admission attempt.
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-23:40:
|
||||
FN-9140 keeps survey verification connectionless by using synthetic ledgers for ownership and
|
||||
ordering. The opt-in smoke is the only test that starts Vitest, so normal script checks remain fast.
|
||||
*/
|
||||
|
||||
test("parseProbeArgs uses bounded defaults and accepts explicit positive integers", () => {
|
||||
assert.deepEqual(parseProbeArgs([]), { delayMs: 40, timeoutMs: 15, workers: 2 });
|
||||
assert.deepEqual(
|
||||
parseProbeArgs(["--delay-ms", "25", "--timeout-ms", "12", "--workers", "3"]),
|
||||
{ delayMs: 25, timeoutMs: 12, workers: 3 },
|
||||
);
|
||||
assert.deepEqual(parseProbeArgs(["--help"]), { help: true });
|
||||
test("arguments derive a discriminating matrix and reject undecidable file/worker counts", () => {
|
||||
assert.deepEqual(deriveBudgets(4000), { D: 4000, SMALL: 1000, LARGE: 20000 });
|
||||
assert.deepEqual(parseProbeArgs([]), { ...options, isolate: "both", json: undefined, runTimeoutMs: 74_000 });
|
||||
assert.throws(() => parseProbeArgs(["--files", "2", "--workers", "2"]), /indeterminate configuration/);
|
||||
assert.throws(() => parseProbeArgs(["--isolate", "maybe"]), /isolate/);
|
||||
});
|
||||
|
||||
test("parseProbeArgs rejects unknown, missing, and non-positive values", () => {
|
||||
assert.throws(() => parseProbeArgs(["--unexpected"]), /Unknown argument/);
|
||||
assert.throws(() => parseProbeArgs(["--workers"]), /positive integer/);
|
||||
assert.throws(() => parseProbeArgs(["--delay-ms", "0"]), /positive integer/);
|
||||
assert.throws(() => parseProbeArgs(["--timeout-ms", "1.5"]), /positive integer/);
|
||||
test("ledger parser ignores malformed or truncated final records", () => {
|
||||
const text = `${JSON.stringify(event())}\n{"type":"boundary"`;
|
||||
assert.deepEqual(parseLedger(text), [{ ...event(), ledgerIndex: 0 }]);
|
||||
});
|
||||
|
||||
test("parseProbeEvents retains only well-formed prefixed fixture events", () => {
|
||||
const output = [
|
||||
"ordinary Vitest output",
|
||||
'PG_SETUP_BOUNDARY_EVENT:{"type":"boundary","boundary":"global-setup","phase":"start","pid":42}',
|
||||
"PG_SETUP_BOUNDARY_EVENT:not-json",
|
||||
'PG_SETUP_BOUNDARY_EVENT:{"type":"test","boundary":"test","phase":"start","pid":42,"file":"one"}',
|
||||
].join("\n");
|
||||
|
||||
assert.deepEqual(parseProbeEvents(output), [
|
||||
{ type: "boundary", boundary: "global-setup", phase: "start", pid: 42, outputIndex: 1 },
|
||||
{ type: "test", boundary: "test", phase: "start", pid: 42, file: "one", outputIndex: 3 },
|
||||
]);
|
||||
test("ledger granularity distinguishes worker and file execution with six files", () => {
|
||||
const workerEvents = ["0", "1"].flatMap((file, pid) => [event({ pid: pid + 1, file }), event({ pid: pid + 1, file, phase: "end", epochMs: 5000, monotonicNs: "500" })]);
|
||||
assert.equal(summarizeLedger("sample", workerEvents, options).granularity, "per worker");
|
||||
const fileEvents = Array.from({ length: 6 }, (_, index) => event({ pid: index % 2 + 1, file: String(index) }));
|
||||
assert.equal(summarizeLedger("sample", fileEvents, options).granularity, "per file");
|
||||
});
|
||||
|
||||
test("classifyVitestOutput separates hook and test budgets from off-budget success", () => {
|
||||
assert.equal(classifyVitestOutput("Error: Hook timed out in 15ms.", 1), "hook-timeout");
|
||||
assert.equal(classifyVitestOutput("Error: Test timed out in 15ms.", 1), "test-timeout");
|
||||
assert.equal(classifyVitestOutput("all tests passed", 0), "off-budget");
|
||||
assert.equal(classifyVitestOutput("unexpected fixture failure", 1), "unknown-failure");
|
||||
test("setup-file granularity requires a real test-file identity across worker lifecycles", () => {
|
||||
/*
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-05:34:
|
||||
Persistent workers can execute several files; each setup callback must retain the active path.
|
||||
*/
|
||||
const persistentWorkers = [event({ pid: 1, file: "a.test.mjs" }), event({ pid: 1, file: "b.test.mjs" }), event({ pid: 2, file: "c.test.mjs" })];
|
||||
assert.equal(summarizeLedger("sample", persistentWorkers, { ...options, files: 3 }).granularity, "per file");
|
||||
/*
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-05:34:
|
||||
Recycled workers may produce as many PIDs as files. Empty setup-file paths must not fake per-file proof.
|
||||
*/
|
||||
const recycledWorkers = [event({ pid: 11, file: "" }), event({ pid: 12, file: "" }), event({ pid: 13, file: "" })];
|
||||
assert.equal(summarizeLedger("sample", recycledWorkers, { ...options, files: 3 }).granularity, "indeterminate");
|
||||
});
|
||||
|
||||
test("summarizeBoundary reports per-file execution and ordering before the first test", () => {
|
||||
const normalRun = {
|
||||
exitCode: 0,
|
||||
output: [
|
||||
'PG_SETUP_BOUNDARY_EVENT:{"type":"boundary","boundary":"per-file-before-all","phase":"start","pid":11,"file":"one"}',
|
||||
'PG_SETUP_BOUNDARY_EVENT:{"type":"boundary","boundary":"per-file-before-all","phase":"start","pid":12,"file":"two"}',
|
||||
'PG_SETUP_BOUNDARY_EVENT:{"type":"test","boundary":"test","phase":"start","pid":11,"file":"one"}',
|
||||
].join("\n"),
|
||||
test("ordering uses same-process monotonic time and cross-process epoch margin", () => {
|
||||
const samePid = [event(), event({ type: "test", boundary: "test", pid: 1, phase: "start", monotonicNs: "200", epochMs: 900 })];
|
||||
assert.equal(summarizeLedger("sample", samePid, options).ordering, "before-first-test");
|
||||
const near = [event({ pid: 1, epochMs: 1000 }), event({ type: "test", boundary: "test", pid: 2, phase: "start", epochMs: 1020, monotonicNs: "1" })];
|
||||
assert.equal(summarizeLedger("sample", near, options).ordering, "indeterminate");
|
||||
const before = [event({ pid: 1, epochMs: 1000 }), event({ type: "test", boundary: "test", pid: 2, phase: "start", epochMs: 1100, monotonicNs: "1" })];
|
||||
assert.equal(summarizeLedger("sample", before, options).ordering, "before-first-test");
|
||||
assert.equal(summarizeLedger("sample", [event()], options).ordering, "not-observed");
|
||||
});
|
||||
|
||||
test("unrecognized failures retain their diagnostic and never become off-budget", () => {
|
||||
const result = classifyArm({ exitCode: 1, output: "Error: unrelated fixture failure", events: [], boundary: "sample", delayMs: 4000 });
|
||||
assert.deepEqual(result, { outcome: "failed-unclassified", diagnostic: "Error: unrelated fixture failure" });
|
||||
assert.equal(resolveOwnership({ A: result, B: { outcome: "passed" }, C: { outcome: "passed" }, R: { outcome: "passed" } }), "failed-unclassified");
|
||||
});
|
||||
|
||||
test("only a duration-verified both-small pass establishes falsifying off-budget evidence", () => {
|
||||
const passed = { outcome: "passed" };
|
||||
assert.equal(resolveOwnership({ A: { outcome: "timed-out-test" }, B: passed, C: passed, R: passed }), "failed-unclassified");
|
||||
assert.equal(resolveOwnership({ R: passed }), "completed-within-budget");
|
||||
assert.equal(resolveOwnership({ A: passed, B: passed, C: passed, R: passed }), "off-budget");
|
||||
const short = classifyArm({ exitCode: 0, events: [event(), event({ phase: "end", epochMs: 2000, monotonicNs: "200" })], boundary: "sample", delayMs: 4000 });
|
||||
assert.equal(short.outcome, "failed-unclassified");
|
||||
assert.equal(resolveOwnership({ A: passed, B: { outcome: "timed-out-test" }, C: passed, R: passed }), "failed-unclassified");
|
||||
});
|
||||
|
||||
test("repeat disagreement, bad calibration, and indeterminate ordering force insufficient data", () => {
|
||||
const stable = {
|
||||
ownership: "off-budget",
|
||||
arms: { A: { outcome: "passed" }, B: { outcome: "passed" }, C: { outcome: "passed" }, R: { outcome: "passed" } },
|
||||
summaries: Object.fromEntries(["A", "B", "C", "R"].map((arm) => [arm, { ordering: "before-first-test", granularity: "per worker", durationMs: 4000 }])),
|
||||
};
|
||||
const stressedRun = { exitCode: 1, output: "Error: Hook timed out in 15ms." };
|
||||
|
||||
assert.deepEqual(summarizeBoundary("per-file-before-all", normalRun, stressedRun), {
|
||||
boundary: "per-file-before-all",
|
||||
granularity: "per file",
|
||||
executions: 2,
|
||||
workersObserved: 2,
|
||||
filesObserved: 2,
|
||||
timeoutCharge: "hook-timeout",
|
||||
orderingRelativeToFirstTest: "before-first-test",
|
||||
});
|
||||
const flapping = summarizeRepeats("sample", [stable, { ...stable, ownership: "test-timeout" }]);
|
||||
assert.equal(flapping.deterministic, false);
|
||||
assert.equal(decideSurveyVerdict({ calibration: "passed", cells: [flapping] }), "insufficient-data");
|
||||
assert.equal(decideSurveyVerdict({ calibration: "failed", cells: [{ ...stable, deterministic: true, ordering: "before-first-test", ownership: "off-budget" }] }), "insufficient-data");
|
||||
assert.equal(decideSurveyVerdict({ calibration: "passed", cells: [{ ...stable, deterministic: true, ordering: "indeterminate", ownership: "off-budget" }] }), "insufficient-data");
|
||||
assert.equal(decideSurveyVerdict({ calibration: "passed", cells: [{ ...stable, deterministic: true, ordering: "after-first-test", ownership: "completed-within-budget" }] }), "prerequisite-not-established");
|
||||
assert.equal(decideSurveyVerdict({ calibration: "passed", cells: [{ ...stable, deterministic: true, ordering: "before-first-test", ownership: "off-budget" }] }), "prerequisite-established");
|
||||
});
|
||||
|
||||
test("summarizeBoundary uses global invocation semantics and reports absent ordering honestly", () => {
|
||||
const normalRun = {
|
||||
exitCode: 0,
|
||||
output: 'PG_SETUP_BOUNDARY_EVENT:{"type":"boundary","boundary":"global-setup","phase":"start","pid":99}',
|
||||
test("arm-level ordering and outcomes must agree across repeats", () => {
|
||||
const repeat = {
|
||||
ownership: "off-budget",
|
||||
arms: { A: { outcome: "passed" }, B: { outcome: "passed" }, C: { outcome: "passed" }, R: { outcome: "passed" } },
|
||||
summaries: Object.fromEntries(["A", "B", "C", "R"].map((arm) => [arm, { ordering: "before-first-test", granularity: "per invocation", durationMs: 4000 }])),
|
||||
};
|
||||
const stressedRun = { exitCode: 0, output: "passed" };
|
||||
|
||||
assert.deepEqual(summarizeBoundary("global-setup", normalRun, stressedRun), {
|
||||
boundary: "global-setup",
|
||||
granularity: "per invocation",
|
||||
executions: 1,
|
||||
workersObserved: 1,
|
||||
filesObserved: 0,
|
||||
timeoutCharge: "off-budget",
|
||||
orderingRelativeToFirstTest: "not-observed",
|
||||
});
|
||||
const aAfterTest = JSON.parse(JSON.stringify(repeat));
|
||||
aAfterTest.summaries.A.ordering = "after-first-test";
|
||||
const armFlap = JSON.parse(JSON.stringify(repeat));
|
||||
armFlap.arms.B.outcome = "timed-out-test";
|
||||
assert.equal(summarizeRepeats("sample", [repeat, aAfterTest]).deterministic, false);
|
||||
assert.equal(summarizeRepeats("sample", [repeat, armFlap]).deterministic, false);
|
||||
});
|
||||
|
||||
test("fixture-local config contains derived budgets and survey remains repository-inert", () => {
|
||||
const fixture = fixtureFiles({ boundary: "setup-top-level-await", arm: "A", options, ledgerPath: "/tmp/ledger", vitestApiUrl: "vitest", control: undefined });
|
||||
assert.match(fixture["vitest.config.mjs"], /testTimeout:1000/);
|
||||
assert.match(fixture["vitest.config.mjs"], /hookTimeout:1000/);
|
||||
const perFileFixture = fixtureFiles({ boundary: "setup-before-all", arm: "A", options, ledgerPath: "/tmp/ledger", vitestApiUrl: "vitest", control: undefined });
|
||||
assert.match(perFileFixture["setup.mjs"], /expect\.getState\(\)\.testPath/);
|
||||
const source = readFileSync(new URL("../pg-setup-boundary-probe.mjs", import.meta.url), "utf8");
|
||||
assert.doesNotMatch(source, /pg-test-harness|pg-preadmission-campaign|packages\/core\/src\/__test-utils__|FUSION_PG_TEST_/);
|
||||
assert.match(source, /mkdtempSync\(join\(tmpdir\(\)/);
|
||||
});
|
||||
|
||||
test("unresolvable Vitest becomes a retained insufficient-data report", async () => {
|
||||
const runner = resolveVitestRunner(() => { throw new Error("test runner unavailable"); });
|
||||
assert.equal(runner.vitestVersion, "unresolvable");
|
||||
const result = await runFixtureCell({ boundary: "global-setup", arm: "A", options, runner });
|
||||
assert.equal(result.arm.outcome, "failed-unclassified");
|
||||
assert.match(result.arm.diagnostic, /Vitest resolution failed: test runner unavailable/);
|
||||
const report = await runSurvey({ ...options, repeats: 1, isolate: "true" }, runner);
|
||||
assert.equal(report.calibration, "failed");
|
||||
assert.equal(report.calibrationFailureReason, "classifier-uncalibrated");
|
||||
assert.equal(report.verdict, "insufficient-data");
|
||||
assert.match(report.runnerResolutionError, /test runner unavailable/);
|
||||
assert.ok(report.cells.every((cell) => cell.ownership === "failed-unclassified"));
|
||||
});
|
||||
|
||||
test("live fixture cells preserve setup-file identities on the installed runner", { skip: process.env.FUSION_BOUNDARY_SURVEY_LIVE !== "1" }, async () => {
|
||||
const liveOptions = { ...options, files: 3, workers: 2, delayMs: 100, runTimeoutMs: 60_000, isolate: false };
|
||||
const global = await runFixtureCell({ boundary: "global-setup", arm: "A", options: liveOptions });
|
||||
assert.ok(global.events.length > 0);
|
||||
assert.equal(global.arm.outcome, "passed");
|
||||
const setup = await runFixtureCell({ boundary: "setup-before-all", arm: "R", options: liveOptions });
|
||||
const summary = summarizeLedger("setup-before-all", setup.events, liveOptions);
|
||||
assert.equal(setup.arm.outcome, "passed");
|
||||
assert.equal(summary.granularity, "per file");
|
||||
assert.equal(summary.filesObserved, 3);
|
||||
});
|
||||
|
||||
@@ -1,182 +1,247 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { appendFileSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { createRequire } from "node:module";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
/*
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-03:25:
|
||||
FN-9139 must establish whether Vitest lifecycle setup can host a future PostgreSQL admission
|
||||
without changing the PostgreSQL harness. This isolated fixture observes only lifecycle timing and
|
||||
never loads a repository config, a database client, or a project test suite.
|
||||
*/
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
const BOUNDARIES = ["global-setup", "setup-top-level-await", "setup-before-all", "per-file-before-all"];
|
||||
const DEFAULTS = Object.freeze({ delayMs: 40, timeoutMs: 15, workers: 2 });
|
||||
export const BOUNDARIES = ["global-setup", "setup-top-level-await", "setup-before-all", "per-file-before-all"];
|
||||
export const ARMS = ["A", "B", "C", "R"];
|
||||
const DIAGNOSTIC_LIMIT = 4000;
|
||||
|
||||
/*
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-23:40:
|
||||
FN-9140 replaces stdout scraping and one-shot, non-falsifying observations with an append-only
|
||||
ledger, repeated fixture-local budget arms, and calibration. Parent and fork clocks have different
|
||||
monotonic origins, so only same-pid ordering uses monotonicNs; cross-pid ordering requires an epoch
|
||||
separation margin. SMALL and LARGE are measurement parameters written only to the temp fixture.
|
||||
*/
|
||||
export function deriveBudgets(delayMs) {
|
||||
return { D: delayMs, SMALL: Math.floor(delayMs / 4), LARGE: delayMs * 5 };
|
||||
}
|
||||
|
||||
export function parseProbeArgs(argv) {
|
||||
const options = { ...DEFAULTS };
|
||||
const names = new Map([
|
||||
["--delay-ms", "delayMs"],
|
||||
["--timeout-ms", "timeoutMs"],
|
||||
["--workers", "workers"],
|
||||
]);
|
||||
|
||||
const result = { workers: 2, files: 6, repeats: 3, isolate: "both", delayMs: 4000, orderingMarginMs: 50, json: undefined, runTimeoutMs: undefined };
|
||||
const names = new Map([["--workers", "workers"], ["--files", "files"], ["--repeats", "repeats"], ["--delay-ms", "delayMs"], ["--ordering-margin-ms", "orderingMarginMs"], ["--run-timeout-ms", "runTimeoutMs"]]);
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const argument = argv[index];
|
||||
if (argument === "--help") return { help: true };
|
||||
const name = names.get(argument);
|
||||
if (!name) throw new Error(`Unknown argument: ${argument}`);
|
||||
const arg = argv[index];
|
||||
if (arg === "--help") return { help: true };
|
||||
if (arg === "--json") { result.json = argv[++index]; if (!result.json) throw new Error("--json requires a path"); continue; }
|
||||
if (arg === "--isolate") { result.isolate = argv[++index]; if (!["both", "true", "false"].includes(result.isolate)) throw new Error("--isolate must be both, true, or false"); continue; }
|
||||
const name = names.get(arg);
|
||||
if (!name) throw new Error(`Unknown argument: ${arg}`);
|
||||
const value = Number(argv[++index]);
|
||||
if (!Number.isSafeInteger(value) || value <= 0) {
|
||||
throw new Error(`${argument} must be a positive integer`);
|
||||
}
|
||||
options[name] = value;
|
||||
if (!Number.isSafeInteger(value) || value <= 0) throw new Error(`${arg} must be a positive integer`);
|
||||
result[name] = value;
|
||||
}
|
||||
return options;
|
||||
if (result.files <= result.workers) throw new Error("indeterminate configuration: --files must be strictly greater than --workers");
|
||||
const budgets = deriveBudgets(result.delayMs);
|
||||
if (budgets.SMALL >= budgets.D || budgets.LARGE <= budgets.D) throw new Error("delay does not produce discriminating budgets");
|
||||
result.runTimeoutMs ??= Math.max(60_000, budgets.LARGE + result.delayMs * result.files + 30_000);
|
||||
return result;
|
||||
}
|
||||
|
||||
/** Extract deliberately prefixed fixture events without coupling classification to Vitest's reporter. */
|
||||
export function parseProbeEvents(output) {
|
||||
return output.split(/\r?\n/).flatMap((line, index) => {
|
||||
const marker = line.indexOf("PG_SETUP_BOUNDARY_EVENT:");
|
||||
if (marker < 0) return [];
|
||||
/** Parse append-only JSONL, intentionally ignoring an interrupted final write. */
|
||||
export function parseLedger(text) {
|
||||
const lines = text.split(/\r?\n/);
|
||||
return lines.flatMap((line, index) => {
|
||||
if (!line.trim()) return [];
|
||||
try {
|
||||
const event = JSON.parse(line.slice(marker + "PG_SETUP_BOUNDARY_EVENT:".length));
|
||||
return [{ ...event, outputIndex: index }];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
const event = JSON.parse(line);
|
||||
if (!event || typeof event !== "object" || !Number.isFinite(event.pid) || !Number.isFinite(event.epochMs) || typeof event.monotonicNs !== "string") return [];
|
||||
return [{ ...event, ledgerIndex: index }];
|
||||
} catch { return []; }
|
||||
});
|
||||
}
|
||||
|
||||
/** Classify timeout ownership from stable Vitest diagnostic phrases, preserving unknown output. */
|
||||
export function classifyVitestOutput(output, exitCode = 0) {
|
||||
if (/Hook timed out in \d+ms\./i.test(output)) return "hook-timeout";
|
||||
if (/Test timed out in \d+ms\./i.test(output)) return "test-timeout";
|
||||
if (exitCode === 0) return "off-budget";
|
||||
return "unknown-failure";
|
||||
function eventDuration(events, boundary) {
|
||||
const start = events.find((event) => event.type === "boundary" && event.boundary === boundary && event.phase === "start");
|
||||
const end = events.find((event) => event.type === "boundary" && event.boundary === boundary && event.phase === "end" && event.pid === start?.pid && event.file === start?.file);
|
||||
return start && end ? end.epochMs - start.epochMs : null;
|
||||
}
|
||||
|
||||
export function summarizeBoundary(boundary, normalRun, stressedRun) {
|
||||
const events = parseProbeEvents(normalRun.output);
|
||||
const starts = events.filter((event) => event.boundary === boundary && event.phase === "start");
|
||||
const firstTest = events.find((event) => event.type === "test" && event.phase === "start");
|
||||
const lastBoundary = starts.at(-1);
|
||||
const files = new Set(starts.map((event) => event.file).filter(Boolean));
|
||||
const pids = new Set(starts.map((event) => event.pid).filter(Boolean));
|
||||
const granularity = boundary === "global-setup"
|
||||
? "per invocation"
|
||||
: files.size > 1
|
||||
? "per file"
|
||||
: pids.size > 1
|
||||
? "per worker"
|
||||
: "per invocation";
|
||||
export function summarizeLedger(boundary, events, { files, orderingMarginMs }) {
|
||||
const starts = events.filter((event) => event.type === "boundary" && event.boundary === boundary && event.phase === "start");
|
||||
const pairs = new Set(starts.map((event) => `${event.pid}:${event.file ?? ""}`));
|
||||
const pidCount = new Set(starts.map((event) => event.pid)).size;
|
||||
const fileCount = new Set(starts.map((event) => event.file).filter(Boolean)).size;
|
||||
/*
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-05:34:
|
||||
A setup-file callback can be evaluated repeatedly in one worker. Without a test-file identity,
|
||||
pid/file pairs cannot distinguish that from worker-scoped setup, even when PIDs are recycled.
|
||||
*/
|
||||
const hasFileIdentity = starts.every((event) => typeof event.file === "string" && event.file.length > 0);
|
||||
const granularity = starts.length === 1 ? "per invocation"
|
||||
: !hasFileIdentity ? "indeterminate"
|
||||
: pairs.size === files && fileCount === files ? "per file"
|
||||
: pairs.size === pidCount ? "per worker"
|
||||
: "indeterminate";
|
||||
const tests = events.filter((event) => event.type === "test" && event.phase === "start");
|
||||
let ordering = "not-observed";
|
||||
if (starts.length && tests.length) {
|
||||
const last = starts.toSorted((a, b) => a.pid === b.pid ? Number(BigInt(a.monotonicNs) - BigInt(b.monotonicNs)) : a.epochMs - b.epochMs).at(-1);
|
||||
const first = tests.toSorted((a, b) => a.pid === b.pid ? Number(BigInt(a.monotonicNs) - BigInt(b.monotonicNs)) : a.epochMs - b.epochMs)[0];
|
||||
if (last.pid === first.pid) ordering = BigInt(last.monotonicNs) < BigInt(first.monotonicNs) ? "before-first-test" : "after-first-test";
|
||||
else if (Math.abs(first.epochMs - last.epochMs) < orderingMarginMs) ordering = "indeterminate";
|
||||
else ordering = last.epochMs < first.epochMs ? "before-first-test" : "after-first-test";
|
||||
}
|
||||
return { boundary, executions: starts.length, distinctPidFiles: pairs.size, workersObserved: pidCount, filesObserved: fileCount, granularity, ordering, durationMs: eventDuration(events, boundary) };
|
||||
}
|
||||
|
||||
function excerpt(output) { return output.slice(-DIAGNOSTIC_LIMIT); }
|
||||
export function classifyArm({ exitCode, output = "", events = [], boundary, delayMs, timedOut = false }) {
|
||||
if (timedOut) return { outcome: "run-timeout" };
|
||||
const summary = summarizeLedger(boundary, events, { files: Number.MAX_SAFE_INTEGER, orderingMarginMs: 1 });
|
||||
if (exitCode === 0) {
|
||||
if (summary.durationMs == null || summary.durationMs < delayMs * 0.9) return { outcome: "failed-unclassified", diagnostic: "boundary end missing or duration below tolerance" };
|
||||
return { outcome: "passed", durationMs: summary.durationMs };
|
||||
}
|
||||
if (/Test timed out in \d+ms\./i.test(output)) return { outcome: "timed-out-test" };
|
||||
if (/Hook timed out in \d+ms\./i.test(output)) return { outcome: "timed-out-hook" };
|
||||
return { outcome: "failed-unclassified", diagnostic: excerpt(output) };
|
||||
}
|
||||
|
||||
export function resolveOwnership(arms) {
|
||||
const A = arms.A?.outcome; const B = arms.B?.outcome; const C = arms.C?.outcome; const R = arms.R?.outcome;
|
||||
if ([A, B, C, R].some((outcome) => outcome === "run-timeout")) return "run-timeout";
|
||||
if ([A, B, C, R].some((outcome) => outcome === "failed-unclassified")) return "failed-unclassified";
|
||||
if (A === "passed" && R === "passed" && B === "passed" && C === "passed") return "off-budget";
|
||||
if (A === "timed-out-test" && B === "timed-out-test" && C === "passed" && R === "passed") return "test-timeout";
|
||||
if (A === "timed-out-hook" && C === "timed-out-hook" && B === "passed" && R === "passed") return "hook-timeout";
|
||||
if (A !== "passed" && B === "timed-out-test" && C === "timed-out-hook" && R === "passed") return "both-budgets";
|
||||
// Arm R alone is an intentionally inconclusive calibration observation, never a survey fact.
|
||||
if (A === undefined && B === undefined && C === undefined && R === "passed") return "completed-within-budget";
|
||||
return "failed-unclassified";
|
||||
}
|
||||
|
||||
/*
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-05:12:
|
||||
FN-9140's qualifying off-budget evidence belongs to arm A, not the last arm executed. Preserve every
|
||||
arm summary and require it to agree across repeats so an after-test A observation or arm-level flap
|
||||
cannot be masked by a stable reference arm.
|
||||
*/
|
||||
export function summarizeRepeats(boundary, repeats) {
|
||||
const values = repeats.map((repeat) => ({
|
||||
ownership: repeat.ownership,
|
||||
arms: Object.fromEntries(ARMS.map((arm) => [arm, repeat.arms[arm]?.outcome])),
|
||||
summaries: Object.fromEntries(ARMS.map((arm) => [arm, {
|
||||
ordering: repeat.summaries[arm]?.ordering,
|
||||
granularity: repeat.summaries[arm]?.granularity,
|
||||
}])),
|
||||
}));
|
||||
const deterministic = values.every((value) => JSON.stringify(value) === JSON.stringify(values[0]));
|
||||
const evidence = values[0]?.summaries.A;
|
||||
return {
|
||||
boundary,
|
||||
granularity,
|
||||
executions: starts.length,
|
||||
workersObserved: pids.size,
|
||||
filesObserved: files.size,
|
||||
timeoutCharge: classifyVitestOutput(stressedRun.output, stressedRun.exitCode),
|
||||
orderingRelativeToFirstTest: !firstTest || !lastBoundary
|
||||
? "not-observed"
|
||||
: lastBoundary.outputIndex < firstTest.outputIndex
|
||||
? "before-first-test"
|
||||
: "after-first-test",
|
||||
repeats,
|
||||
deterministic,
|
||||
ownership: deterministic ? values[0]?.ownership : "failed-unclassified",
|
||||
ordering: deterministic ? evidence?.ordering : "indeterminate",
|
||||
granularity: deterministic ? evidence?.granularity : "indeterminate",
|
||||
};
|
||||
}
|
||||
|
||||
function eventSource(type, boundary, phase, file = "") {
|
||||
return `console.log("PG_SETUP_BOUNDARY_EVENT:" + JSON.stringify({type:${JSON.stringify(type)},boundary:${JSON.stringify(boundary)},phase:${JSON.stringify(phase)},pid:process.pid,file:${JSON.stringify(file)}}));`;
|
||||
export function decideSurveyVerdict({ calibration, cells }) {
|
||||
if (calibration !== "passed") return "insufficient-data";
|
||||
if (cells.some((cell) => !cell.deterministic || ["failed-unclassified", "run-timeout"].includes(cell.ownership) || cell.ordering === "indeterminate")) return "insufficient-data";
|
||||
return cells.some((cell) => cell.ownership === "off-budget" && cell.ordering === "before-first-test") ? "prerequisite-established" : "prerequisite-not-established";
|
||||
}
|
||||
|
||||
function fixtureFiles(boundary, delayMs, timeoutMs, workers, vitestApiUrl) {
|
||||
const delay = `await new Promise((resolve) => setTimeout(resolve, ${delayMs}));`;
|
||||
// The fixture is outside the repository, so its Vitest API import must be absolute.
|
||||
const config = `export default { test: { include: ["tests/**/*.test.mjs"], pool: "forks", maxWorkers: ${workers}, minWorkers: ${workers}, testTimeout: ${timeoutMs}, hookTimeout: ${timeoutMs}, setupFiles: ["./setup.mjs"]${boundary === "global-setup" ? ', globalSetup: ["./global-setup.mjs"]' : ""} } };\n`;
|
||||
const setup = boundary === "setup-top-level-await"
|
||||
? `${eventSource("boundary", boundary, "start")}${delay}${eventSource("boundary", boundary, "end")}`
|
||||
: boundary === "setup-before-all"
|
||||
? `import { beforeAll } from ${JSON.stringify(vitestApiUrl)};\nbeforeAll(async () => { ${eventSource("boundary", boundary, "start")}${delay}${eventSource("boundary", boundary, "end")} });`
|
||||
: "";
|
||||
const global = boundary === "global-setup"
|
||||
? `export default async function setup() { ${eventSource("boundary", boundary, "start")}${delay}${eventSource("boundary", boundary, "end")} }`
|
||||
: "export default async function setup() {}";
|
||||
const test = (file) => `import { beforeAll, test } from ${JSON.stringify(vitestApiUrl)};\n${boundary === "per-file-before-all" ? `beforeAll(async () => { ${eventSource("boundary", boundary, "start", file)}${delay}${eventSource("boundary", boundary, "end", file)} });` : ""}\ntest(${JSON.stringify(file)}, () => { ${eventSource("test", "test", "start", file)} });\n`;
|
||||
return {
|
||||
"vitest.config.mjs": config,
|
||||
"setup.mjs": setup,
|
||||
"global-setup.mjs": global,
|
||||
"tests/one.test.mjs": test("one"),
|
||||
"tests/two.test.mjs": test("two"),
|
||||
};
|
||||
export function fixtureFiles({ boundary, arm, options, ledgerPath, vitestApiUrl, control }) {
|
||||
const budgets = deriveBudgets(options.delayMs);
|
||||
const budget = arm === "A" ? [budgets.SMALL, budgets.SMALL] : arm === "B" ? [budgets.SMALL, budgets.LARGE] : arm === "C" ? [budgets.LARGE, budgets.SMALL] : [budgets.LARGE, budgets.LARGE];
|
||||
const emit = (type, name, phase, fileExpression = '""') => `appendFileSync(process.env.FUSION_BOUNDARY_LEDGER, JSON.stringify({type:${JSON.stringify(type)},boundary:${JSON.stringify(name)},phase:${JSON.stringify(phase)},pid:process.pid,file:${fileExpression},isolate:${JSON.stringify(String(options.isolate))},arm:${JSON.stringify(arm)},epochMs:Date.now(),monotonicNs:process.hrtime.bigint().toString()})+'\\n');`;
|
||||
const sleep = `await new Promise(resolve => setTimeout(resolve, ${options.delayMs}));`;
|
||||
const boundaryBody = (fileExpression) => `${emit("boundary", boundary, "start", fileExpression)} ${sleep} ${emit("boundary", boundary, "end", fileExpression)}`;
|
||||
const testBody = control === "test" ? `${emit("boundary", "control-test", "start")} ${sleep} ${emit("boundary", "control-test", "end")}` : "";
|
||||
const config = `export default { test: { include:["tests/**/*.test.mjs"], pool:"forks", isolate:${options.isolate}, maxWorkers:${options.workers}, minWorkers:${options.workers}, fileParallelism:true, testTimeout:${budget[0]}, hookTimeout:${budget[1]}, setupFiles:["./setup.mjs"]${boundary === "global-setup" ? ',globalSetup:["./global-setup.mjs"]' : ""} } };\n`;
|
||||
/*
|
||||
FNXC:PgSetupBoundaryProbe 2026-08-17-05:34:
|
||||
FN-9140 must not infer per-file setup execution from worker PIDs. beforeAll can read Vitest's
|
||||
active test path; top-level setup has no supported file context and is deliberately indeterminate.
|
||||
*/
|
||||
const setup = `import { appendFileSync } from "node:fs"; import { beforeAll, expect } from ${JSON.stringify(vitestApiUrl)}; ${boundary === "setup-top-level-await" ? boundaryBody() : ""} ${boundary === "setup-before-all" || control === "hook" ? `beforeAll(async()=>{${control === "hook" ? `${emit("boundary", "control-hook", "start")} ${sleep} ${emit("boundary", "control-hook", "end")}` : boundaryBody('expect.getState().testPath ?? ""')}});` : ""}`;
|
||||
const global = `import { appendFileSync } from "node:fs"; export default async function(){ ${boundary === "global-setup" ? boundaryBody() : ""} }`;
|
||||
const files = { "vitest.config.mjs": config, "setup.mjs": setup, "global-setup.mjs": global };
|
||||
for (let index = 0; index < options.files; index += 1) files[`tests/${index}.test.mjs`] = `import { appendFileSync } from "node:fs"; import { beforeAll,test } from ${JSON.stringify(vitestApiUrl)}; ${boundary === "per-file-before-all" ? `beforeAll(async()=>{${boundaryBody(JSON.stringify(String(index)))}});` : ""} test(${JSON.stringify(String(index))}, async()=>{${testBody} ${emit("test", "test", "start", JSON.stringify(String(index)))} ${emit("test", "test", "end", JSON.stringify(String(index)))}});`;
|
||||
return files;
|
||||
}
|
||||
|
||||
async function writeFixture(directory, files) {
|
||||
await Promise.all(Object.entries(files).map(async ([file, content]) => {
|
||||
const path = join(directory, file);
|
||||
await mkdir(dirname(path), { recursive: true });
|
||||
await writeFile(path, content, "utf8");
|
||||
}));
|
||||
}
|
||||
function writeFixture(directory, files) { for (const [file, content] of Object.entries(files)) { const target = join(directory, file); mkdirSync(dirname(target), { recursive: true }); writeFileSync(target, content); } }
|
||||
function vitestEntry() { return join(dirname(require.resolve("vitest", { paths: [resolve(process.cwd(), "packages/core")] })), "vitest.mjs"); }
|
||||
|
||||
function runVitest(directory) {
|
||||
const vitest = join(dirname(require.resolve("vitest", { paths: [resolve(process.cwd(), "packages/core")] })), "vitest.mjs");
|
||||
return new Promise((resolve, reject) => {
|
||||
const child = spawn(process.execPath, [vitest, "run", "--config", "vitest.config.mjs", "--reporter=verbose"], {
|
||||
cwd: directory,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let output = "";
|
||||
child.stdout.on("data", (chunk) => { output += chunk; });
|
||||
child.stderr.on("data", (chunk) => { output += chunk; });
|
||||
child.on("error", reject);
|
||||
child.on("close", (exitCode) => resolve({ output, exitCode: exitCode ?? 1 }));
|
||||
});
|
||||
}
|
||||
|
||||
async function runBoundary(boundary, options, stressed) {
|
||||
const directory = await mkdtemp(join(tmpdir(), "fusion-pg-setup-boundary-"));
|
||||
export function resolveVitestRunner(resolveEntry = vitestEntry) {
|
||||
try {
|
||||
const delayMs = stressed ? Math.max(options.delayMs, options.timeoutMs + 20) : Math.min(options.delayMs, Math.max(1, options.timeoutMs - 1));
|
||||
const vitestApiUrl = pathToFileURL(require.resolve("vitest", { paths: [resolve(process.cwd(), "packages/core")] })).href;
|
||||
await writeFixture(directory, fixtureFiles(boundary, delayMs, options.timeoutMs, options.workers, vitestApiUrl));
|
||||
return await runVitest(directory);
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
export async function runProbe(options) {
|
||||
const summaries = [];
|
||||
for (const boundary of BOUNDARIES) {
|
||||
const normalRun = await runBoundary(boundary, options, false);
|
||||
const stressedRun = await runBoundary(boundary, options, true);
|
||||
summaries.push(summarizeBoundary(boundary, normalRun, stressedRun));
|
||||
}
|
||||
return summaries;
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
console.log("Usage: node scripts/pg-setup-boundary-probe.mjs [--delay-ms N] [--timeout-ms N] [--workers N]");
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
try {
|
||||
const options = parseProbeArgs(process.argv.slice(2));
|
||||
if (options.help) {
|
||||
printHelp();
|
||||
} else {
|
||||
console.table(await runProbe(options));
|
||||
}
|
||||
const entry = resolveEntry();
|
||||
return { entry, vitestVersion: JSON.parse(readFileSync(join(dirname(entry), "package.json"), "utf8")).version };
|
||||
} catch (error) {
|
||||
console.error(`[pg-setup-boundary-probe] ${error.message}`);
|
||||
process.exitCode = 1;
|
||||
return { entry: undefined, vitestVersion: "unresolvable", resolutionError: error instanceof Error ? error.message : String(error) };
|
||||
}
|
||||
}
|
||||
|
||||
export async function runFixtureCell({ boundary, arm, options, control, runner = resolveVitestRunner() }) {
|
||||
const directory = mkdtempSync(join(tmpdir(), "fusion-pg-setup-boundary-")); const ledgerPath = join(directory, "ledger.jsonl");
|
||||
try {
|
||||
if (!runner.entry) {
|
||||
const run = { exitCode: 1, output: `Vitest resolution failed: ${runner.resolutionError}`, timedOut: false, events: [] };
|
||||
return { ...run, arm: classifyArm({ ...run, boundary: control ? `control-${control}` : boundary, delayMs: options.delayMs }) };
|
||||
}
|
||||
try {
|
||||
const api = pathToFileURL(join(dirname(runner.entry), "dist", "index.js")).href;
|
||||
writeFixture(directory, fixtureFiles({ boundary, arm, options, ledgerPath, vitestApiUrl: api, control }));
|
||||
const run = await new Promise((done) => {
|
||||
const child = spawn(process.execPath, [runner.entry, "run", "--config", "vitest.config.mjs", "--reporter=verbose"], { cwd: directory, env: { ...process.env, FUSION_BOUNDARY_LEDGER: ledgerPath }, stdio: ["ignore", "pipe", "pipe"] }); let output = ""; let timedOut = false;
|
||||
child.stdout.on("data", (chunk) => { output += chunk; }); child.stderr.on("data", (chunk) => { output += chunk; });
|
||||
child.once("error", (error) => { output += `Vitest spawn failed: ${error.message}`; });
|
||||
const timer = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, options.runTimeoutMs);
|
||||
child.on("close", (exitCode) => { clearTimeout(timer); done({ exitCode: exitCode ?? 1, output, timedOut }); });
|
||||
});
|
||||
let events = [];
|
||||
try { events = parseLedger(readFileSync(ledgerPath, "utf8")); } catch { /* A setup failure may occur before the first ledger append. */ }
|
||||
return { ...run, events, arm: classifyArm({ ...run, events, boundary: control ? `control-${control}` : boundary, delayMs: options.delayMs }) };
|
||||
} catch (error) {
|
||||
const run = { exitCode: 1, output: `Vitest fixture failed: ${error instanceof Error ? error.message : String(error)}`, timedOut: false, events: [] };
|
||||
return { ...run, arm: classifyArm({ ...run, boundary: control ? `control-${control}` : boundary, delayMs: options.delayMs }) };
|
||||
}
|
||||
} finally { rmSync(directory, { recursive: true, force: true }); }
|
||||
}
|
||||
|
||||
export async function runSurvey(options, runner = resolveVitestRunner()) {
|
||||
const budgets = deriveBudgets(options.delayMs); const isolateModes = options.isolate === "both" ? [true, false] : [options.isolate === "true"];
|
||||
const calibrationResults = [];
|
||||
for (const [control, arm, expected] of [["hook", "C", "hook-timeout"], ["test", "B", "test-timeout"], ["test", "R", "completed-within-budget"]]) {
|
||||
const run = await runFixtureCell({ boundary: "setup-before-all", arm, options: { ...options, isolate: true }, control, runner });
|
||||
const outcome = control === "test" && arm === "R" && run.arm.outcome === "passed" ? "completed-within-budget" : run.arm.outcome === "timed-out-hook" ? "hook-timeout" : run.arm.outcome === "timed-out-test" ? "test-timeout" : run.arm.outcome;
|
||||
calibrationResults.push({ control, arm, expected, actual: outcome });
|
||||
}
|
||||
const calibration = calibrationResults.every((row) => row.actual === row.expected) ? "passed" : "failed";
|
||||
const cells = [];
|
||||
for (const isolate of isolateModes) for (const boundary of BOUNDARIES) {
|
||||
const repeats = [];
|
||||
for (let repeat = 0; repeat < options.repeats; repeat += 1) {
|
||||
const arms = {}; const summaries = {};
|
||||
for (const arm of ARMS) {
|
||||
const run = await runFixtureCell({ boundary, arm, options: { ...options, isolate }, runner });
|
||||
arms[arm] = run.arm;
|
||||
summaries[arm] = summarizeLedger(boundary, run.events, options);
|
||||
}
|
||||
repeats.push({ repeat, arms, summaries, ownership: resolveOwnership(arms) });
|
||||
}
|
||||
cells.push({ isolate, ...summarizeRepeats(boundary, repeats) });
|
||||
}
|
||||
return { vitestVersion: runner.vitestVersion, nodeVersion: process.version, pool: "forks", isolateModes, workers: options.workers, files: options.files, repeats: options.repeats, budgetMatrix: { ...budgets, orderingMarginMs: options.orderingMarginMs, runTimeoutMs: options.runTimeoutMs }, calibration, calibrationFailureReason: calibration === "failed" ? "classifier-uncalibrated" : undefined, runnerResolutionError: runner.resolutionError, calibrationResults, cells, verdict: decideSurveyVerdict({ calibration, cells }) };
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const options = parseProbeArgs(process.argv.slice(2));
|
||||
if (options.help) return console.log("Usage: node scripts/pg-setup-boundary-probe.mjs --workers 2 --files 6 --repeats 3 --delay-ms 4000 [--json path]");
|
||||
const isolateModes = options.isolate === "both" ? [true, false] : [options.isolate === "true"];
|
||||
console.log(`[pg-setup-boundary-probe] per-cell bound ${options.runTimeoutMs}ms; total bounded cells ${(3 + BOUNDARIES.length * isolateModes.length * ARMS.length) * options.repeats}`);
|
||||
const report = await runSurvey(options);
|
||||
if (options.json) writeFileSync(options.json, `${JSON.stringify(report, null, 2)}\n`);
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
}
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) main().catch((error) => { console.error(`[pg-setup-boundary-probe] ${error.message}`); process.exitCode = 1; });
|
||||
|
||||
Reference in New Issue
Block a user