From fb77eef7a985a82f804d04583d99aed810618f56 Mon Sep 17 00:00:00 2001 From: gsxdsm Date: Wed, 19 Aug 2026 10:10:39 -0700 Subject: [PATCH] FN-9150: Capture abandoned PostgreSQL timeout boundaries Capture elapsed and join evidence for PostgreSQL test boundaries that Vitest abandons before settlement. - emit unique boundary join keys, progress ladders, terminal records, and payload-free watchdog breaches - classify ladder-only, breach, and unobservable lifecycle evidence in the loaded-failure census - cover abandoned and superseded body windows and document the corrective campaign findings Files changed: ...res-loaded-lane-unrelated-failure-population.md | 33 ++++++- docs/testing.md | 14 ++- .../__test-utils__/pg-timeout-boundary-observer.ts | 108 ++++++++++++++++++--- .../__tests__/pg-timeout-boundary-observer.test.ts | 98 ++++++++++++++++++- .../__tests__/pg-loaded-failure-census.test.mjs | 30 ++++++ scripts/pg-loaded-failure-census.mjs | 71 ++++++++++++-- 6 files changed, 327 insertions(+), 27 deletions(-) Fusion-Task-Id: FN-9150 Fusion-Task-Lineage: fc2e9f99-7739-4d86-99f4-713a78f53e57 Co-authored-by: Fusion (runfusion.ai) --- ...oaded-lane-unrelated-failure-population.md | 33 +++++- docs/testing.md | 14 ++- .../pg-timeout-boundary-observer.ts | 108 +++++++++++++++--- .../pg-timeout-boundary-observer.test.ts | 98 +++++++++++++++- .../pg-loaded-failure-census.test.mjs | 30 +++++ scripts/pg-loaded-failure-census.mjs | 71 +++++++++++- 6 files changed, 327 insertions(+), 27 deletions(-) diff --git a/docs/solutions/test-failures/postgres-loaded-lane-unrelated-failure-population.md b/docs/solutions/test-failures/postgres-loaded-lane-unrelated-failure-population.md index 346522f648..767af62225 100644 --- a/docs/solutions/test-failures/postgres-loaded-lane-unrelated-failure-population.md +++ b/docs/solutions/test-failures/postgres-loaded-lane-unrelated-failure-population.md @@ -72,7 +72,36 @@ The superseded 12s clean/dirty arm remains retained but cannot decide M5. The re | M2 DDL serialization | still undecided (missing evidence: join coverage) | Twelve J watchdog joins cannot correlate DDL/locks to 127 failures. | | M3 golden-template/advisory convoy | still undecided (missing evidence: joined golden-lock waiters) | The probe now records granted holders and non-granted golden advisory waiters; no sufficient joined timeout population exists. | | M4 host CPU/event-loop starvation | still undecided (missing evidence: join coverage) | The observer now records watchdog scheduling drift rather than a fixed zero, but the campaign cannot distinguish idle-cluster host starvation from blocked SQL at required body coverage. | -| M5 dirty-cluster carryover | affirmed | The amended interleaved J04–J07 arm covaried at 0 leftovers → 29/36 failures and 1 test + 2 schema-template (one golden) + 1 pool leftover → 43/50 failures. FN-9151 owns identification and a regression-proven structural remedy; no remedy is implemented here. | +| M5 dirty-cluster carryover | still undecided (underpowered descriptive covariation) | J04–J07 covaried at 0 leftovers → 29/36 failures and 1 test + 2 schema-template (one golden) + 1 pool leftover → 43/50 failures. FN-9150's prospectively configured clean/dirty pair was 60/42 failures, respectively, but n=1/arm is below the 7/arm Δ=20 target. Neither result is an M5 verdict; FN-9152 owns the powered arm. | + +## FN-9150 timeout-boundary coverage diagnosis + +M01 was a clean-start 27-worker measure-first run with a 1,000ms ladder, threshold 0, one probe maximum, and the 12,000ms watchdog. It reported 29/176 failing files at peak 70, leaving 27 ordinary slots of headroom. It is elapsed-distribution evidence, not attribution. Setup/body/teardown records were 3,101/804/2,579; terminal maxima were 20,448/1,895/15,013ms and progress-only maxima were 20,099/1,001/14,065ms. Progress rows are lower bounds at 1,000ms resolution and cannot be pooled with terminal elapsed measurements. + +The 12,000ms watchdog has at most 3,000ms before the inherited 15,000ms Vitest limit. Its 1,500ms probe timeout and non-negative single-flight queue delay leave at most 1,500ms before scheduling delay and the unobservable Vitest-to-harness offset. M01 captured 30 watchdog results and seven cap suppressions. Of 29 failing files, 14 (48.3%) were consumer `afterEach` failures outside the shared-harness bracket; they are position-unobservable, not observer misses. Four failures (13.8%) had ladder bounds; 11 (37.9%) remained observable-position unjoined. Vitest JSON exposed 1,386 per-test durations but no hook duration or hook failure position; file-level pairing produced invalid negative differences down to -13,971.94ms. The hook-clock offset is therefore unmeasurable on this reporter version, not a point estimate. + +FN-9150's two-phase breach row preserves a keyed elapsed record before a probe can be abandoned, while the ladder covers boundaries that never reach the watchdog. Neither row may affirm M2–M4. The real shared-harness gate passed normal, ladder, and forced runs: forced setup/body/teardown records were 10/3/13, with enriched non-suppressed payloads 3/1/4 and a reporter file join. The forced gate is wiring evidence only. + +A post-campaign review found that shared-harness body windows reused a file-level key. The observer now emits a unique `joinKey` for every open body window and retains that file-level value only as `supersessionKey`: a terminal record for a later healthy body can no longer mask an earlier abandoned body's progress-only ladder records. The census already groups terminal filtering by `joinKey`; a production-shaped multiple-body regression covers the separation. This is a coverage-integrity correction, not a new sample or an attribution verdict. + +The prospective power rule uses A01–A05 σ=12.6 and `2(1.96+0.84)^2σ²/Δ²`: Δ=20 requires seven runs/arm and Δ=10 requires 25. The 12-run self-imposed budget (46.16 minutes at 230.8s/run) cannot power either enabled-vs-unset or M5 after one measure-first and three gate-equivalent runs. The retained J enabled/control and J04–J07 clean/dirty samples are consequently descriptive; no absence or covariation from them is a powered perturbation or M5 verdict. + +## FN-9150 corrective campaign — retained outcomes + +The fixed campaign stack was pre-registered before its runs: observer threshold 0, ladder 1,000ms, 12,000ms setup/body/teardown watchdogs, 1,500/1,400/3,000ms probe/statement/drain bounds, one concurrent probe, queue timeout zero, and four probes. Every run used 27 workers, teardown diagnostics, JSON reporter capture, and a clean starting cluster unless labelled dirty. The intended two-run-per-arm allocation could not approach the pre-registered n=7/arm Δ=20 requirement; all comparisons below are explicitly **undetermined (underpowered)**. + +| lane | failing files | peak / headroom | positions | coverage yield (joined / ladder / position-unobservable / unjoined) | outcome | +|---|---:|---:|---|---|---| +| enabled C1 | 34 | 66 / 31 | afterEach 8; afterAll 9; beforeAll 1; body 16 | 0 / 2 / 8 / 24 | Coverage floor missed; no attribution. | +| unset U1 | 39 | 70 / 27 | beforeAll 5; afterAll 4; afterEach 15; body 15 | 0 / 0 / 15 / 24 | Control retained. | +| enabled C2 | 35 | 70 / 27 | afterEach 15; afterAll 5; beforeAll 1; body 14 | 2 / 0 / 15 / 18 | Coverage floor missed; two joined rows do not identify a mechanism. | +| unset U2 | 66 | 66 / 31 | afterEach 28; afterAll 11; beforeAll 8; body 19 | 0 / 0 / 28 / 38 | Control retained. | +| M5 clean | 60 | 60 / 37 | beforeAll 6; afterEach 31; afterAll 8; body 14; setup 1 | 2 / 5 / 31 / 22 | Clean arm, one `cap` suppression. | +| M5 dirty | 42 | 78 / 19 | body 18; afterAll 5; beforeAll 5; afterEach 14 | 2 / 1 / 14 / 25 | Dirty arm began with one test, one schema-template, and one pool database. | + +Enabled mean failure count was 34.5 versus unset 52.5 (difference −18), but n=2/arm is below seven and the pre-declared Welch/rank comparison is not inferential. The 50% joined-plus-ladder coverage floor was missed in both enabled samples (2/34 and 2/35), so enabled rows are a coverage finding, not M2–M4 attribution. No observer probe was concurrency-suppressed in the perturbation lanes; one M5-clean record was cap-suppressed. The M5 clean/dirty difference is 18 failures at n=1/arm and is likewise **undetermined (underpowered)**. Final hygiene after every run was restored to seven backends and zero matching test/template/pool databases. + +Payload-free breach and ladder rows remain location-only evidence: they cannot affirm a cluster-state mechanism. The new campaign again keeps every observed peak below the 97 ordinary-slot ceiling, which preserves M1's generic-form elimination. M2–M4 remain undecided because coverage is below floor; the successor needs an executing-test/hook-position join identity or a bracket for the remaining unowned positions, while M5 needs seven interleaved runs per arm under an allocated host budget. ## Remedies disqualified by this evidence @@ -84,4 +113,4 @@ not a resolution. ## Successor measurement seam -FN-9151 owns the affirmed dirty-carryover structural seam: identify why retained `fusion_test_%`, `fusion_schema_template%` (including golden), and `fusion_pool_%` state covaries with the 27-worker failure population, then prove a remedy with a regression without changing timeouts, retries, skips, worker caps, quarantine, DDL, connection-budget, or admission behavior. A separate successor may improve timeout-boundary join coverage for M2–M4; it must preserve the default-off observer and repeat the perturbation control. Generic ordinary-slot exhaustion remains disqualified. +FN-9152 owns the powered dirty-carryover measurement arm: obtain the host budget for seven interleaved clean and seven dirty runs before judging the retained covariation, without changing timeouts, retries, skips, worker caps, quarantine, DDL, connection-budget, or admission behavior. A separate successor may improve timeout-boundary join coverage for M2–M4; it must preserve the default-off observer and repeat the perturbation control. Generic ordinary-slot exhaustion remains disqualified. diff --git a/docs/testing.md b/docs/testing.md index 8f3d24eaf7..9a0605e2e1 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -549,9 +549,17 @@ A run is `insufficient-data` when its runner log lacks a complete `Test Files` s ### PostgreSQL timeout-boundary observer -`FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER=1` enables a diagnostic-only JSONL channel; when unset it creates no timer, probe connection, sink write, listener, or environment mutation. `observeBoundary()` wraps harness-owned setup and teardown work. `openBoundary()`/`closeBoundary()` span the shared harness's separate `beforeEach` and `afterEach` hooks, so `shared.body` measures test bodies without editing consumers. Closing is idempotent; an abandoned window is superseded by a later open or disposed without inventing a completion. +`FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER=1` enables a diagnostic-only JSONL channel; when unset it creates no timer, probe connection, sink write, listener, or environment mutation. `observeBoundary()` wraps harness-owned setup and teardown work. `openBoundary()`/`closeBoundary()` span the shared harness's separate `beforeEach` and `afterEach` hooks, so `shared.body` measures test bodies without editing consumers. Each open body window receives a unique record `joinKey`; its stable file-level `supersessionKey` only closes a stale handle. Closing is idempotent; an abandoned window is superseded by a later open or disposed without inventing a completion. -The observer arms an unref'd watchdog at boundary start. Only a pending watchdog dispatches a maintenance-connection probe; completion records are host-only and emit when elapsed time meets `FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_THRESHOLD_MS` (default 2000; `0` measures every completion). A dispatched probe survives boundary settlement and appends when it resolves, with `settledDuringProbe`, `probeLatencyMs`, and optionally `probeStartDelayMs`; records can therefore be out of order and consumers must join by pid, worker, file, boundary, and timestamp rather than JSONL order. `flush()`/`dispose()` drain bounded pending and queued probes; a missed drain becomes `probeSuppressed:"drain-timeout"`. The enabled-only `beforeExit` hook performs the same best-effort drain. +The observer arms an unref'd watchdog at boundary start. Only a pending watchdog dispatches a maintenance-connection probe; completion records are host-only and emit when elapsed time meets `FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_THRESHOLD_MS` (default 2000; `0` measures every completion). With `FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_LADDER_MS` above zero, an independent unref'd checkpoint timer synchronously emits keyed payload-free `kind:"progress"` rows and rearms at that spacing. The last progress row is an elapsed lower bound for an abandoned boundary, with the configured spacing as its resolution; it is never a PostgreSQL-state snapshot. A settle/reject emits a keyed `kind:"terminal"` outcome without error text and clears both timers exactly once. + +The watchdog first synchronously appends a keyed `kind:"breach"`, `payloadFree:true` record, then appends its enriched `kind:"watchdog"` probe result. This two-phase form preserves a joinable elapsed boundary when Vitest abandons work during the watchdog-to-probe residual window; a payload-free breach never affirms a cluster-state mechanism. A dispatched probe survives boundary settlement and appends when it resolves, with `settledDuringProbe`, `probeLatencyMs`, and optionally `probeStartDelayMs`; records can therefore be out of order and consumers must join by the per-window `joinKey`, never by JSONL order or the shared `supersessionKey`. `flush()`/`dispose()` drain bounded pending and queued probes; a missed drain becomes `probeSuppressed:"drain-timeout"`. The enabled-only `beforeExit` hook performs the same best-effort drain. + + + + + + The independent, tighten-only bounds are `statement timeout < probe timeout < inherited budget`, `queue timeout < probe timeout`, and `threshold <= per-boundary watchdog < inherited budget`. A probe timeout is deliberately **not** tied to a short watchdog: forced wiring runs need a fast watchdog and a real cluster round trip. Floors are `MIN_WATCHDOG_MS`, `MIN_PROBE_TIMEOUT_MS`, and `MIN_STATEMENT_TIMEOUT_MS`; an impossible floor yields `bounds-floor`, never a wider budget. @@ -570,7 +578,7 @@ At concurrency 1 the cost profile matches the older single-flight observer. `pro For an enabled-wiring check, first run disabled and normal enabled bounds to compare duration, then threshold 0 to verify setup, `shared.body`, and teardown completion records, then force per-boundary watchdogs below measured elapsed times while independently retaining a probe/drain timeout above cluster round-trip and raising probe cap/concurrency to at least 3. Every forced boundary must have a non-suppressed payload; `cap`, `concurrency`, `bounds-floor`, `drain-timeout`, and `error` require diagnosis, not acceptance. Files using the shared harness are body-observable; direct `createTaskStoreForTest` files deliberately have no harness-owned body bracket and must be reported as `body-unobservable`. -Pass `--boundary-observer `, `--body-unobservable-files `, and `--fully-unobservable-files ` to `scripts/pg-loaded-failure-census.mjs`. It tolerates malformed lines and classifies each failing boundary as `cluster-implicated`, `host-implicated`, `template-convoy`, `body-unobservable`, or `unjoined`; failures from fully-unobservable files remain `unjoined` but are counted and listed separately so the known no-harness limitation is never mistaken for a missing observer join. Missing/empty input stays explicitly absent rather than a measured zero. +Pass `--boundary-observer `, `--vitest-json `, `--body-unobservable-files `, and `--fully-unobservable-files ` to `scripts/pg-loaded-failure-census.mjs`. Capture the reporter per run with `--reporter=dot --reporter=json --outputFile=`; it provides per-test duration and file identifiers but Vitest 3 provides no hook duration or failure position, so a hook-clock offset is unmeasurable rather than a point estimate. The census tolerates malformed and out-of-order rows, joins ladder/breach/enriched rows by key, and reports joined, `attributed-by-ladder`, `body-unobservable`, `position-unobservable`, and `unjoined` coverage counts plus probe suppression and latency distributions. `afterEach` consumer hooks are position-unobservable because the shared-harness body bracket closes before those hooks run. Failures from fully-unobservable files remain `unjoined` but are counted and listed separately so the known no-harness limitation is never mistaken for a missing observer join. Missing/empty input stays explicitly absent rather than a measured zero. ### PostgreSQL DDL loaded-lane acceptance metric diff --git a/packages/core/src/__test-utils__/pg-timeout-boundary-observer.ts b/packages/core/src/__test-utils__/pg-timeout-boundary-observer.ts index db60f69c29..a76a4af86a 100644 --- a/packages/core/src/__test-utils__/pg-timeout-boundary-observer.ts +++ b/packages/core/src/__test-utils__/pg-timeout-boundary-observer.ts @@ -8,6 +8,7 @@ export const MAX_CONCURRENT_PROBES_CEILING = 8; export type PgTimeoutBoundary = "setup" | "body" | "teardown"; export type PgTimeoutBoundaryTrigger = "boundary-complete" | "boundary-watchdog"; +export type PgTimeoutBoundaryRecordKind = "progress" | "terminal" | "breach" | "watchdog"; export type PgTimeoutBoundarySuppression = "cap" | "concurrency" | "bounds-floor" | "drain-timeout" | "error"; type TimerHandle = ReturnType; @@ -41,9 +42,18 @@ export interface PgTimeoutBoundaryRecord { readonly testName?: string; readonly boundary: PgTimeoutBoundary; readonly phase: string; + /** Stable lifecycle label used to join reporter failures without line order. */ + readonly position: string; + /** Per-window identity; progress-only keys prove an abandoned boundary. */ + readonly joinKey: string; + /** Caller-owned lifecycle identity used only to supersede stale open windows. */ + readonly supersessionKey?: string; + readonly kind: PgTimeoutBoundaryRecordKind; readonly trigger: PgTimeoutBoundaryTrigger; readonly elapsedMs: number; readonly boundaryIncomplete: boolean; + /** A breach was synchronously recorded before its optional probe begins. */ + readonly payloadFree?: boolean; readonly settledDuringProbe?: boolean; readonly probeLatencyMs?: number; readonly probeStartDelayMs?: number; @@ -106,6 +116,14 @@ function envNumber(env: NodeJS.ProcessEnv, key: string, fallback: number, minimu return Number.isFinite(value) && value >= minimum ? Math.trunc(value) : fallback; } +/** Keep observer and census file joins stable across absolute and Windows paths. */ +function normalizeTestFile(value: string): string { + const normalized = value.replaceAll("\\", "/"); + const root = normalized.indexOf("src/__tests__/"); + if (root >= 0) return normalized.slice(root); + return normalized; +} + function defaultHostSample(eventLoopLagMs = 0): PgTimeoutBoundaryHostSample { const [loadavg1, loadavg5, loadavg15] = loadavg(); const workers = Number(process.env.VITEST_MAX_WORKERS); @@ -129,6 +147,8 @@ interface Bounds { readonly maxConcurrentProbes: number; readonly probeQueueTimeoutMs: number; readonly maxProbes: number; + /** Zero preserves the pre-FN-9150 settle/watchdog-only observer. */ + readonly ladderMs: number; readonly probeAllowed: boolean; readonly clamped: boolean; } @@ -139,6 +159,7 @@ function resolveBounds(env: NodeJS.ProcessEnv, hookBudget: number, bodyBudget: n const requestedProbe = envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_PROBE_TIMEOUT_MS", 1_500, 1); const requestedStatement = envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_STATEMENT_TIMEOUT_MS", Math.max(MIN_STATEMENT_TIMEOUT_MS, requestedProbe - 100), 1); const requestedDrain = envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_PROBE_DRAIN_TIMEOUT_MS", 3_000, 1); + const ladderMs = envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_LADDER_MS", 0); const capBelowHook = Math.max(0, hookBudget - 1); // Floors make malformed tiny values observable, but never widen an inherited // budget: a budget below the floor disables probing for that boundary. @@ -180,6 +201,7 @@ function resolveBounds(env: NodeJS.ProcessEnv, hookBudget: number, bodyBudget: n maxConcurrentProbes, probeQueueTimeoutMs, maxProbes: envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_MAX_PROBES", 4, 1), + ladderMs, probeAllowed, clamped, }; @@ -189,11 +211,15 @@ interface BoundaryState { readonly handle: PgTimeoutBoundaryHandle; readonly boundary: PgTimeoutBoundary; readonly phase: string; - readonly key?: string; + /** Unique emitted-record key; it must never be shared by adjacent test bodies. */ + readonly joinKey: string; + /** Stable caller key that only controls stale-window supersession. */ + readonly supersessionKey?: string; readonly startedAt: number; readonly timestamp: string; readonly host: PgTimeoutBoundaryHostSample; timer?: TimerHandle; + ladderTimer?: TimerHandle; settled: boolean; watchdogFired: boolean; outcome?: "resolved" | "rejected"; @@ -237,6 +263,19 @@ Once probes survive their boundary, strict single-flight would suppress body and teardown behind setup. The configurable limiter defaults to one (the inherited cost profile) but is raisable for the forced wiring gate; its ceiling prevents a bad environment from flooding PostgreSQL. + +FNXC:PgTimeoutBoundaryObserver 2026-08-19-15:43: +FN-9150 needs elapsed evidence for boundaries Vitest abandons before they +settle. A payload-free checkpoint ladder is independent of probes, unref'd, and +writes synchronously so its last progress record is a lower bound rather than a +claim about PostgreSQL state. Terminal records make progress-only join keys the +explicit abandoned-boundary signature. + +FNXC:PgTimeoutBoundaryObserver 2026-08-19-16:06: +A watchdog must append its keyed payload-free breach before scheduling a probe. +Vitest can abandon the boundary during that residual window, so waiting for the +probe result censors the timeout population this default-off observer measures. +The breach locates a boundary but never asserts PostgreSQL state. */ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserverOptions = {}): PgTimeoutBoundaryObserver { const env = options.env ?? process.env; @@ -270,15 +309,19 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv const unref = (timer: TimerHandle | undefined): void => { (timer as unknown as { unref?: () => void } | undefined)?.unref?.(); }; - const emit = (state: BoundaryState, trigger: PgTimeoutBoundaryTrigger, fields: Partial): void => { + const emit = (state: BoundaryState, trigger: PgTimeoutBoundaryTrigger, kind: PgTimeoutBoundaryRecordKind, fields: Partial, writeStderr = true): void => { const record: PgTimeoutBoundaryRecord = { timestamp: state.timestamp, pid: process.pid, ...(env.VITEST_WORKER_ID ? { workerId: env.VITEST_WORKER_ID } : {}), - ...(options.testFile ? { testFile: options.testFile } : {}), + ...(options.testFile ? { testFile: normalizeTestFile(options.testFile) } : {}), ...(options.testName?.() ? { testName: options.testName() } : {}), boundary: state.boundary, phase: state.phase, + position: state.phase, + joinKey: state.joinKey, + ...(state.supersessionKey ? { supersessionKey: state.supersessionKey } : {}), + kind, trigger, elapsedMs: Math.max(0, now() - state.startedAt), boundaryIncomplete: !state.settled, @@ -296,7 +339,9 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv if (sink) { try { append(sink, `${JSON.stringify(record)}\n`); } catch { /* sink failures must not affect a test */ } } - try { writeError(`[pg-timeout-boundary-observer] ${trigger} ${state.boundary}/${state.phase}`); } catch { /* diagnostic stderr is best effort */ } + if (writeStderr) { + try { writeError(`[pg-timeout-boundary-observer] ${trigger} ${state.boundary}/${state.phase}`); } catch { /* diagnostic stderr is best effort */ } + } }; const finalize = (pendingProbe: PendingProbe, suppression?: PgTimeoutBoundarySuppression, payload?: PgTimeoutBoundaryProbePayload): void => { if (pendingProbe.finalized) return; @@ -309,7 +354,7 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv admitQueued(); } const state = pendingProbe.state; - emit(state, "boundary-watchdog", { + emit(state, "boundary-watchdog", "watchdog", { elapsedMs: bounds.watchdog[state.boundary], boundaryIncomplete: true, settledDuringProbe: state.settled, @@ -354,7 +399,16 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv }; const requestProbe = (state: BoundaryState, eventLoopLagMs: number): void => { const watchdogAt = now(); - const pendingProbe: PendingProbe = { state, watchdogAt, host: hostSample(eventLoopLagMs), controller: new AbortController(), started: false, finalized: false }; + const host = hostSample(eventLoopLagMs); + // This synchronous first phase survives disposal/worker termination while a + // maintenance connection is still queued or executing. + emit(state, "boundary-watchdog", "breach", { + elapsedMs: bounds.watchdog[state.boundary], + boundaryIncomplete: true, + payloadFree: true, + host, + }); + const pendingProbe: PendingProbe = { state, watchdogAt, host, controller: new AbortController(), started: false, finalized: false }; if (!options.probe) return finalize(pendingProbe, "error"); if (!bounds.probeAllowed) return finalize(pendingProbe, "bounds-floor"); pending.add(pendingProbe); @@ -373,20 +427,35 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv // as the host/cluster snapshot, without adding another timer or I/O path. requestProbe(state, Math.max(0, now() - (state.startedAt + bounds.watchdog[state.boundary]))); }; + const armLadder = (state: BoundaryState): void => { + if (bounds.ladderMs === 0) return; + const checkpoint = (): void => { + if (state.ladderTimer) clearTimer(state.ladderTimer); + if (disposed || state.settled) return; + emit(state, "boundary-complete", "progress", { boundaryIncomplete: true }, false); + state.ladderTimer = setTimer(checkpoint, bounds.ladderMs); + unref(state.ladderTimer); + }; + state.ladderTimer = setTimer(checkpoint, bounds.ladderMs); + unref(state.ladderTimer); + }; const arm = (state: BoundaryState): void => { state.timer = setTimer(() => fireWatchdog(state), bounds.watchdog[state.boundary]); unref(state.timer); + armLadder(state); }; const complete = (state: BoundaryState, outcome: "resolved" | "rejected"): void => { if (state.settled) return; state.settled = true; state.outcome = outcome; if (state.timer) clearTimer(state.timer); + if (state.ladderTimer) clearTimer(state.ladderTimer); states.delete(state.handle.id); - if (state.key) windows.delete(state.key); - if (!state.watchdogFired && Math.max(0, now() - state.startedAt) >= bounds.thresholdMs) { - emit(state, "boundary-complete", { outcome, boundaryIncomplete: false }); - } + if (state.supersessionKey && windows.get(state.supersessionKey) === state) windows.delete(state.supersessionKey); + // A terminal record is emitted regardless of threshold. Progress-only keys + // are therefore the durable, resolution-bounded signature of abandonment. + const reportCompletion = !state.watchdogFired && Math.max(0, now() - state.startedAt) >= bounds.thresholdMs; + emit(state, "boundary-complete", "terminal", { outcome, boundaryIncomplete: false }, reportCompletion); }; const beforeExit = (): void => { void flush(); }; process.on("beforeExit", beforeExit); @@ -412,7 +481,8 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv return { enabled: true, observeBoundary(boundary: PgTimeoutBoundary, phase: string, action: () => Promise): Promise { - const state: BoundaryState = { handle: { id: nextHandle++ }, boundary, phase, startedAt: now(), timestamp: new Date(wallNow()).toISOString(), host: hostSample(), settled: false, watchdogFired: false }; + const handle = { id: nextHandle++ }; + const state: BoundaryState = { handle, boundary, phase, joinKey: `${process.pid}:${env.VITEST_WORKER_ID ?? "main"}:${normalizeTestFile(options.testFile ?? "unknown")}:${phase}:${handle.id}`, startedAt: now(), timestamp: new Date(wallNow()).toISOString(), host: hostSample(), settled: false, watchdogFired: false }; states.set(state.handle.id, state); arm(state); let result: Promise; @@ -424,10 +494,19 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv const superseded = Boolean(prior && !prior.settled); if (prior && !prior.settled) { if (prior.timer) clearTimer(prior.timer); + if (prior.ladderTimer) clearTimer(prior.ladderTimer); prior.settled = true; states.delete(prior.handle.id); } - const state: BoundaryState = { handle: { id: nextHandle++ }, boundary, phase, key, startedAt: now(), timestamp: new Date(wallNow()).toISOString(), host: hostSample(), settled: false, watchdogFired: false, ...(superseded ? { supersededOpenWindow: true } : {}) }; + const handle = { id: nextHandle++ }; + /* + FNXC:PgTimeoutBoundaryObserver 2026-08-19-16:56: + Shared-harness body hooks reuse a file-level lifecycle key across tests. + That key may supersede a stale handle, but every body window needs its own + emitted join key: a later healthy terminal must not mask an earlier + abandoned body's ladder records in the failure census. + */ + const state: BoundaryState = { handle, boundary, phase, joinKey: `${key}:${handle.id}`, supersessionKey: key, startedAt: now(), timestamp: new Date(wallNow()).toISOString(), host: hostSample(), settled: false, watchdogFired: false, ...(superseded ? { supersededOpenWindow: true } : {}) }; windows.set(key, state); states.set(state.handle.id, state); arm(state); @@ -441,7 +520,10 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv async dispose(): Promise { if (disposed) return; disposed = true; - for (const state of states.values()) if (state.timer) clearTimer(state.timer); + for (const state of states.values()) { + if (state.timer) clearTimer(state.timer); + if (state.ladderTimer) clearTimer(state.ladderTimer); + } states.clear(); windows.clear(); await flush(); diff --git a/packages/core/src/__tests__/pg-timeout-boundary-observer.test.ts b/packages/core/src/__tests__/pg-timeout-boundary-observer.test.ts index 7cea050412..9b26d61f76 100644 --- a/packages/core/src/__tests__/pg-timeout-boundary-observer.test.ts +++ b/packages/core/src/__tests__/pg-timeout-boundary-observer.test.ts @@ -69,13 +69,105 @@ describe("PG timeout boundary observer", () => { await observed; pending.resolve(payload); await vi.advanceTimersByTimeAsync(0); - expect(records(lines)).toEqual([expect.objectContaining({ - trigger: "boundary-watchdog", boundary: "body", phase: "shared.body", boundaryIncomplete: true, + expect(records(lines)).toContainEqual(expect.objectContaining({ + trigger: "boundary-watchdog", kind: "watchdog", boundary: "body", phase: "shared.body", boundaryIncomplete: true, settledDuringProbe: true, cluster: payload.cluster, template: payload.template, - })]); + })); await observer.dispose(); }); + it("records progress-only ladder bounds for an abandoned boundary and clears unref'd timers", async () => { + const lines: string[] = []; + let now = 0; + const timers: Array<{ callback: () => void; ms: number; unref: ReturnType }> = []; + const clearTimer = vi.fn(); + const observer = createPgTimeoutBoundaryObserver({ + env: { ...env, FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_LADDER_MS: "5" }, + now: () => now, + setTimer: ((callback, ms) => { + const timer = { callback, ms, unref: vi.fn() }; + timers.push(timer); + return timer as unknown as ReturnType; + }) as typeof setTimeout, + clearTimer: clearTimer as typeof clearTimeout, + append: (_path, line) => lines.push(line), + writeError: () => {}, + }); + observer.openBoundary("body", "shared.body", "abandoned"); + now = 5; + timers.find((timer) => timer.ms === 5)?.callback(); + now = 10; + timers.at(-1)?.callback(); + await observer.dispose(); + + const progress = records(lines).filter((record) => record.kind === "progress"); + expect(progress).toEqual([ + expect.objectContaining({ joinKey: "abandoned:1", elapsedMs: 5, boundaryIncomplete: true }), + expect.objectContaining({ joinKey: "abandoned:1", elapsedMs: 10, boundaryIncomplete: true }), + ]); + expect(records(lines).some((record) => record.kind === "terminal")).toBe(false); + expect(timers.every((timer) => timer.unref.mock.calls.length === 1)).toBe(true); + expect(clearTimer).toHaveBeenCalledTimes(4); + }); + + it("keeps an abandoned shared-harness body window distinct from a later healthy body", async () => { + const lines: string[] = []; + let now = 0; + const timers: Array<{ callback: () => void; ms: number; unref: ReturnType }> = []; + const observer = createPgTimeoutBoundaryObserver({ + env: { ...env, FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_LADDER_MS: "5" }, + now: () => now, + testFile: "/repo/packages/core/src/__tests__/postgres/shared-bodies.pg.test.ts", + setTimer: ((callback, ms) => { + const timer = { callback, ms, unref: vi.fn() }; + timers.push(timer); + return timer as unknown as ReturnType; + }) as typeof setTimeout, + clearTimer: () => {}, + append: (_path, line) => lines.push(line), + writeError: () => {}, + }); + // This is the file-level key passed by createSharedPgTaskStoreTestHarness + // for consecutive beforeEach/afterEach body windows. + const sharedHarnessKey = `${process.pid}:main:/repo/packages/core/src/__tests__/postgres/shared-bodies.pg.test.ts`; + observer.openBoundary("body", "shared.body", sharedHarnessKey); + now = 5; + timers.find((timer) => timer.ms === 5)?.callback(); + const healthyBody = observer.openBoundary("body", "shared.body", sharedHarnessKey); + observer.closeBoundary(healthyBody); + + const emitted = records(lines); + const abandonedProgress = emitted.find((record) => record.kind === "progress"); + const healthyTerminal = emitted.find((record) => record.kind === "terminal"); + expect(abandonedProgress).toMatchObject({ supersessionKey: sharedHarnessKey, boundary: "body", elapsedMs: 5 }); + expect(healthyTerminal).toMatchObject({ supersessionKey: sharedHarnessKey, boundary: "body" }); + expect(abandonedProgress?.joinKey).not.toBe(healthyTerminal?.joinKey); + await observer.dispose(); + }); + + it("records a payload-free breach before an abandoned probe can resolve", async () => { + vi.useFakeTimers(); + const lines: string[] = []; + const pending = deferred(); + const observer = createPgTimeoutBoundaryObserver({ + env, + probe: () => pending.promise, + append: (_path, line) => lines.push(line), + writeError: () => {}, + }); + observer.openBoundary("setup", "database.clone", "abandoned-watchdog"); + await vi.advanceTimersByTimeAsync(10); + expect(records(lines)).toContainEqual(expect.objectContaining({ + kind: "breach", + payloadFree: true, + joinKey: "abandoned-watchdog:1", + boundary: "setup", + })); + const disposing = observer.dispose(); + await vi.advanceTimersByTimeAsync(100); + await disposing; + }); + it("records watchdog scheduling drift as event-loop lag for host attribution", async () => { const lines: string[] = []; let now = 0; diff --git a/scripts/__tests__/pg-loaded-failure-census.test.mjs b/scripts/__tests__/pg-loaded-failure-census.test.mjs index f4c40919a8..b8dda70400 100644 --- a/scripts/__tests__/pg-loaded-failure-census.test.mjs +++ b/scripts/__tests__/pg-loaded-failure-census.test.mjs @@ -10,6 +10,7 @@ import { parseDiagnosticsJsonl, parseBoundaryObserverJsonl, classifyBoundaryAttribution, + parseVitestJson, stripAnsi, } from "../pg-loaded-failure-census.mjs"; @@ -58,6 +59,33 @@ test("joins out-of-order watchdog payloads by file and boundary, not line order" assert.equal(classifyBoundaryAttribution(failure, parsed.rows).classification, "cluster-implicated"); }); +test("attributes a progress-only observer key without inventing a cluster payload", () => { + const failure = { file: "src/__tests__/postgres/abandoned.test.ts", lifecyclePosition: "test body" }; + const observer = [ + // Consecutive shared-harness bodies retain a common supersession identity, + // but their emitted record join keys must remain per-window. + { testFile: failure.file, boundary: "body", kind: "progress", joinKey: "body-1", supersessionKey: "shared-file", elapsedMs: 5000 }, + { testFile: failure.file, boundary: "body", kind: "progress", joinKey: "body-1", supersessionKey: "shared-file", elapsedMs: 10000 }, + { testFile: failure.file, boundary: "body", kind: "terminal", joinKey: "body-2", supersessionKey: "shared-file", elapsedMs: 10 }, + ]; + const attribution = classifyBoundaryAttribution(failure, observer); + assert.equal(attribution.classification, "attributed-by-ladder"); + assert.equal(attribution.elapsedLowerBoundMs, 10000); + const settledProgress = [ + { testFile: failure.file, boundary: "body", kind: "progress", joinKey: "settled-body", elapsedMs: 5000 }, + { testFile: failure.file, boundary: "body", kind: "terminal", joinKey: "settled-body", elapsedMs: 6000 }, + ]; + assert.equal(classifyBoundaryAttribution(failure, settledProgress).classification, "unjoined"); + const breachOnly = [{ testFile: failure.file, boundary: "body", kind: "breach", payloadFree: true, joinKey: "body-2", trigger: "boundary-watchdog", host: { loadavg1: 0, cpuCount: 8, eventLoopLagMs: 0 } }]; + assert.equal(classifyBoundaryAttribution(failure, breachOnly).classification, "joined"); + + const reporter = parseVitestJson(JSON.stringify({ testResults: [{ name: `/repo/${failure.file}`, assertionResults: [{ fullName: "body timeout", duration: 15000, status: "failed", failureMessages: ["Test timed out in 15000ms"] }] }] })); + assert.equal(reporter.malformed, false); + assert.equal(reporter.rows[0].testFile, failure.file); + assert.equal(reporter.rows[0].durationMs, 15000); + assert.equal(parseVitestJson("truncated").malformed, true); +}); + test("keeps explicit unobservable sets and suppressed watchdog failures distinct from joined attribution", () => { const body = { file: "src/__tests__/postgres/direct.test.ts", lifecyclePosition: "test body" }; assert.equal(classifyBoundaryAttribution(body, [], [body.file]).classification, "body-unobservable"); @@ -71,6 +99,8 @@ test("keeps explicit unobservable sets and suppressed watchdog failures distinct assert.equal(census.fullyUnobservableFailingFileCount, 1); assert.deepEqual(census.fullyUnobservableFailingFiles, [fully.file]); assert.equal(census.attributions[0].boundaryAttribution.classification, "unjoined"); + const afterEach = { file: "src/__tests__/postgres/shared.test.ts", lifecyclePosition: "afterEach" }; + assert.equal(classifyBoundaryAttribution(afterEach, []).classification, "position-unobservable"); }); test("requires a golden advisory waiter, not a holder, for template convoy attribution", () => { diff --git a/scripts/pg-loaded-failure-census.mjs b/scripts/pg-loaded-failure-census.mjs index 3d05ac12f7..b9e35959b0 100644 --- a/scripts/pg-loaded-failure-census.mjs +++ b/scripts/pg-loaded-failure-census.mjs @@ -33,7 +33,7 @@ export function parseDiagnosticsJsonl(text) { return { rows, malformedLines }; } -function normalizeFile(value) { +export function normalizeFile(value) { const normalized = String(value).replaceAll("\\", "/"); // Vitest diagnostics use absolute paths while runner failures use repo paths. // Canonicalize at the test-root segment before any key-based observer join. @@ -102,6 +102,25 @@ export function parseBoundaryObserverJsonl(text) { return parseDiagnosticsJsonl(text); } +/** Vitest JSON exposes test durations; hook duration is not available in v4 output. */ +export function parseVitestJson(text) { + try { + const report = JSON.parse(String(text)); + const rows = Array.isArray(report?.testResults) ? report.testResults : []; + return { + rows: rows.flatMap((file) => (Array.isArray(file?.assertionResults) ? file.assertionResults : []).map((test) => ({ + testFile: normalizeFile(file?.name), + position: classifyLifecyclePosition([test?.fullName, ...(test?.failureMessages ?? [])].join("\n")), + durationMs: Number.isFinite(test?.duration) ? test.duration : null, + status: test?.status ?? null, + }))), + malformed: false, + }; + } catch { + return { rows: [], malformed: true }; + } +} + function boundaryForLifecycle(lifecyclePosition) { if (lifecyclePosition === "beforeAll hook" || lifecyclePosition === "in-test setup") return "setup"; if (lifecyclePosition === "afterEach" || lifecyclePosition === "afterAll hook" || lifecyclePosition === "global setup-teardown") return "teardown"; @@ -129,8 +148,23 @@ export function classifyBoundaryAttribution(failure, observerRecords, bodyUnobse if (boundary === "body" && bodyUnobservableFiles.includes(failure.file)) { return { classification: "body-unobservable", boundary, record: null, hostOnly: false, fullyUnobservable: false }; } + // Shared-harness afterEach only closes the body bracket; consumer afterEach + // hooks are outside it, so an absent record is an explicit position limit. + if (failure.lifecyclePosition === "afterEach") { + return { classification: "position-unobservable", boundary, record: null, hostOnly: false, fullyUnobservable: false }; + } + // A settled boundary can legitimately have earlier checkpoints. Only a key + // with progress and no terminal record is an abandoned-boundary lower bound. + const terminalKeys = new Set(sameFile.filter((record) => record.kind === "terminal" && typeof record.joinKey === "string").map((record) => record.joinKey)); + const abandonedProgress = sameFile.filter((record) => record.kind === "progress" && typeof record.joinKey === "string" && !terminalKeys.has(record.joinKey)); + if (abandonedProgress.length > 0) { + const withElapsed = abandonedProgress.map((record) => ({ record, elapsedMs: Number(record.elapsedMs) })).filter(({ elapsedMs }) => Number.isFinite(elapsedMs)); + const latestBound = withElapsed.length > 0 ? withElapsed.reduce((latest, candidate) => candidate.elapsedMs > latest.elapsedMs ? candidate : latest) : null; + return { classification: "attributed-by-ladder", boundary, record: latestBound?.record ?? abandonedProgress[0], hostOnly: true, elapsedLowerBoundMs: latestBound?.elapsedMs ?? null }; + } const watchdog = sameFile.filter((record) => record.trigger === "boundary-watchdog"); const record = watchdog.find((candidate) => !candidate.probeSuppressed && candidate.cluster && candidate.template) + ?? watchdog.find((candidate) => candidate.kind === "watchdog") ?? watchdog[0] ?? sameFile[0] ?? null; @@ -140,7 +174,15 @@ export function classifyBoundaryAttribution(failure, observerRecords, bodyUnobse const load = Number(record?.host?.loadavg1); const cpus = Number(record?.host?.cpuCount); const lag = Number(record?.host?.eventLoopLagMs); - return { classification: (Number.isFinite(load) && Number.isFinite(cpus) && load >= cpus) || lag >= 100 ? "host-implicated" : "unjoined", boundary, record, hostOnly: true }; + if ((Number.isFinite(load) && Number.isFinite(cpus) && load >= cpus) || lag >= 100) { + return { classification: "host-implicated", boundary, record, hostOnly: true }; + } + // A two-phase breach is a real boundary join even when it has no cluster + // payload. Keep it visible as coverage rather than misreporting no record. + if (record?.kind === "breach" && record?.payloadFree === true) { + return { classification: "joined", boundary, record, hostOnly: true }; + } + return { classification: "unjoined", boundary, record, hostOnly: true }; } const template = record.template ?? {}; // A holder alone is not a convoy: only a non-owner waiter proves the @@ -167,9 +209,17 @@ export function summarizeBoundaryObserver(records, failures, bodyUnobservableFil const reason = row?.probeSuppressed === "single-flight" ? "concurrency" : row?.probeSuppressed; if (reason) suppression[reason] = (suppression[reason] ?? 0) + 1; } + const attributionCounts = Object.fromEntries(Object.entries(Object.groupBy(attributions, (row) => row.boundaryAttribution.classification)).map(([key, values]) => [key, values.length])); return { boundaryObserver: rows.length ? "present" : "absent", - boundaryAttributionHistogram: Object.fromEntries(Object.entries(Object.groupBy(attributions, (row) => row.boundaryAttribution.classification)).map(([key, values]) => [key, values.length])), + boundaryAttributionHistogram: attributionCounts, + joinedCoverageYield: { + joined: (attributionCounts.joined ?? 0) + (attributionCounts["cluster-implicated"] ?? 0) + (attributionCounts["host-implicated"] ?? 0) + (attributionCounts["template-convoy"] ?? 0), + attributedByLadder: attributionCounts["attributed-by-ladder"] ?? 0, + bodyUnobservable: attributionCounts["body-unobservable"] ?? 0, + positionUnobservable: attributionCounts["position-unobservable"] ?? 0, + unjoined: attributionCounts.unjoined ?? 0, + }, observerProbeSuppression: suppression, settledDuringProbeCount: rows.filter((row) => row?.settledDuringProbe === true).length, fullyUnobservableFailingFiles: attributions.filter((row) => row.boundaryAttribution.fullyUnobservable).map((row) => row.file), @@ -211,7 +261,7 @@ export function summarizeDiagnostics(diagnostics) { }; } -export function buildCensus({ log, diagnostics = [], boundaryObserver = [], bodyUnobservableFiles = [], fullyUnobservableFiles = [], ordinarySlotCeiling = null, subjects = [] }) { +export function buildCensus({ log, diagnostics = [], boundaryObserver = [], vitestJson = [], bodyUnobservableFiles = [], fullyUnobservableFiles = [], ordinarySlotCeiling = null, subjects = [] }) { const summary = parseFileSummary(log); if (!summary.complete) { return { status: "insufficient-data", reason: "missing Test Files summary", totalFiles: null, failingFiles: [], failingFileCount: null }; @@ -222,6 +272,12 @@ export function buildCensus({ log, diagnostics = [], boundaryObserver = [], body } const diagnosticSummary = summarizeDiagnostics(diagnostics); const observerSummary = summarizeBoundaryObserver(boundaryObserver, failingFiles, bodyUnobservableFiles, fullyUnobservableFiles); + const reporterFiles = new Set(vitestJson.map((row) => row.testFile).filter(Boolean)); + const observerFiles = new Set(boundaryObserver.map(observerFile).filter(Boolean)); + const reporterJoin = { + observerFilesWithoutReporter: [...observerFiles].filter((file) => !reporterFiles.has(file)), + reporterFilesWithoutObserver: [...reporterFiles].filter((file) => !observerFiles.has(file)), + }; const ceiling = Number.isFinite(ordinarySlotCeiling) && ordinarySlotCeiling >= 0 ? ordinarySlotCeiling : null; return { status: "measured", @@ -235,16 +291,18 @@ export function buildCensus({ log, diagnostics = [], boundaryObserver = [], body backendHeadroom: ceiling != null && diagnosticSummary.peakBackends != null ? ceiling - diagnosticSummary.peakBackends : null, ...diagnosticSummary, ...observerSummary, + reporterJoin, }; } function parseArgs(args) { - const result = { log: undefined, diagnostics: undefined, boundaryObserver: undefined, bodyUnobservableFiles: undefined, fullyUnobservableFiles: undefined, ordinarySlotCeiling: null, subjects: [] }; + const result = { log: undefined, diagnostics: undefined, boundaryObserver: undefined, vitestJson: undefined, bodyUnobservableFiles: undefined, fullyUnobservableFiles: undefined, ordinarySlotCeiling: null, subjects: [] }; for (let index = 0; index < args.length; index += 1) { const argument = args[index]; if (argument === "--log") result.log = args[++index]; else if (argument === "--diagnostics") result.diagnostics = args[++index]; else if (argument === "--boundary-observer") result.boundaryObserver = args[++index]; + else if (argument === "--vitest-json") result.vitestJson = args[++index]; else if (argument === "--body-unobservable-files") result.bodyUnobservableFiles = args[++index]; else if (argument === "--fully-unobservable-files") result.fullyUnobservableFiles = args[++index]; else if (argument === "--ordinary-slot-ceiling") result.ordinarySlotCeiling = Number(args[++index]); @@ -259,7 +317,8 @@ if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) const args = parseArgs(process.argv.slice(2)); const parsed = args.diagnostics ? parseDiagnosticsJsonl(readFileSync(args.diagnostics, "utf8")) : { rows: [], malformedLines: 0 }; const observer = args.boundaryObserver ? parseBoundaryObserverJsonl(readFileSync(args.boundaryObserver, "utf8")) : { rows: [], malformedLines: 0 }; + const reporter = args.vitestJson ? parseVitestJson(readFileSync(args.vitestJson, "utf8")) : { rows: [], malformed: false }; const bodyUnobservableFiles = args.bodyUnobservableFiles ? readFileSync(args.bodyUnobservableFiles, "utf8").split(/\\r?\\n/).map(normalizeFile).filter(Boolean) : []; const fullyUnobservableFiles = args.fullyUnobservableFiles ? readFileSync(args.fullyUnobservableFiles, "utf8").split(/\\r?\\n/).map(normalizeFile).filter(Boolean) : []; - console.log(JSON.stringify({ ...buildCensus({ log: readFileSync(args.log, "utf8"), diagnostics: parsed.rows, boundaryObserver: observer.rows, bodyUnobservableFiles, fullyUnobservableFiles, ordinarySlotCeiling: args.ordinarySlotCeiling, subjects: args.subjects }), malformedDiagnosticLines: parsed.malformedLines, malformedBoundaryObserverLines: observer.malformedLines }, null, 2)); + console.log(JSON.stringify({ ...buildCensus({ log: readFileSync(args.log, "utf8"), diagnostics: parsed.rows, boundaryObserver: observer.rows, vitestJson: reporter.rows, bodyUnobservableFiles, fullyUnobservableFiles, ordinarySlotCeiling: args.ordinarySlotCeiling, subjects: args.subjects }), malformedDiagnosticLines: parsed.malformedLines, malformedBoundaryObserverLines: observer.malformedLines, malformedVitestJson: reporter.malformed }, null, 2)); }