FN-9150: Capture abandoned PostgreSQL timeout boundaries
Capture elapsed and join evidence for PostgreSQL test boundaries that Vitest abandons before settlement. - emit unique boundary join keys, progress ladders, terminal records, and payload-free watchdog breaches - classify ladder-only, breach, and unobservable lifecycle evidence in the loaded-failure census - cover abandoned and superseded body windows and document the corrective campaign findings Files changed: ...res-loaded-lane-unrelated-failure-population.md | 33 ++++++- docs/testing.md | 14 ++- .../__test-utils__/pg-timeout-boundary-observer.ts | 108 ++++++++++++++++++--- .../__tests__/pg-timeout-boundary-observer.test.ts | 98 ++++++++++++++++++- .../__tests__/pg-loaded-failure-census.test.mjs | 30 ++++++ scripts/pg-loaded-failure-census.mjs | 71 ++++++++++++-- 6 files changed, 327 insertions(+), 27 deletions(-) Fusion-Task-Id: FN-9150 Fusion-Task-Lineage: fc2e9f99-7739-4d86-99f4-713a78f53e57 Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
@@ -72,7 +72,36 @@ The superseded 12s clean/dirty arm remains retained but cannot decide M5. The re
|
||||
| M2 DDL serialization | still undecided (missing evidence: join coverage) | Twelve J watchdog joins cannot correlate DDL/locks to 127 failures. |
|
||||
| M3 golden-template/advisory convoy | still undecided (missing evidence: joined golden-lock waiters) | The probe now records granted holders and non-granted golden advisory waiters; no sufficient joined timeout population exists. |
|
||||
| M4 host CPU/event-loop starvation | still undecided (missing evidence: join coverage) | The observer now records watchdog scheduling drift rather than a fixed zero, but the campaign cannot distinguish idle-cluster host starvation from blocked SQL at required body coverage. |
|
||||
| M5 dirty-cluster carryover | affirmed | The amended interleaved J04–J07 arm covaried at 0 leftovers → 29/36 failures and 1 test + 2 schema-template (one golden) + 1 pool leftover → 43/50 failures. FN-9151 owns identification and a regression-proven structural remedy; no remedy is implemented here. |
|
||||
| M5 dirty-cluster carryover | still undecided (underpowered descriptive covariation) | J04–J07 covaried at 0 leftovers → 29/36 failures and 1 test + 2 schema-template (one golden) + 1 pool leftover → 43/50 failures. FN-9150's prospectively configured clean/dirty pair was 60/42 failures, respectively, but n=1/arm is below the 7/arm Δ=20 target. Neither result is an M5 verdict; FN-9152 owns the powered arm. |
|
||||
|
||||
## FN-9150 timeout-boundary coverage diagnosis
|
||||
|
||||
M01 was a clean-start 27-worker measure-first run with a 1,000ms ladder, threshold 0, one probe maximum, and the 12,000ms watchdog. It reported 29/176 failing files at peak 70, leaving 27 ordinary slots of headroom. It is elapsed-distribution evidence, not attribution. Setup/body/teardown records were 3,101/804/2,579; terminal maxima were 20,448/1,895/15,013ms and progress-only maxima were 20,099/1,001/14,065ms. Progress rows are lower bounds at 1,000ms resolution and cannot be pooled with terminal elapsed measurements.
|
||||
|
||||
The 12,000ms watchdog has at most 3,000ms before the inherited 15,000ms Vitest limit. Its 1,500ms probe timeout and non-negative single-flight queue delay leave at most 1,500ms before scheduling delay and the unobservable Vitest-to-harness offset. M01 captured 30 watchdog results and seven cap suppressions. Of 29 failing files, 14 (48.3%) were consumer `afterEach` failures outside the shared-harness bracket; they are position-unobservable, not observer misses. Four failures (13.8%) had ladder bounds; 11 (37.9%) remained observable-position unjoined. Vitest JSON exposed 1,386 per-test durations but no hook duration or hook failure position; file-level pairing produced invalid negative differences down to -13,971.94ms. The hook-clock offset is therefore unmeasurable on this reporter version, not a point estimate.
|
||||
|
||||
FN-9150's two-phase breach row preserves a keyed elapsed record before a probe can be abandoned, while the ladder covers boundaries that never reach the watchdog. Neither row may affirm M2–M4. The real shared-harness gate passed normal, ladder, and forced runs: forced setup/body/teardown records were 10/3/13, with enriched non-suppressed payloads 3/1/4 and a reporter file join. The forced gate is wiring evidence only.
|
||||
|
||||
A post-campaign review found that shared-harness body windows reused a file-level key. The observer now emits a unique `joinKey` for every open body window and retains that file-level value only as `supersessionKey`: a terminal record for a later healthy body can no longer mask an earlier abandoned body's progress-only ladder records. The census already groups terminal filtering by `joinKey`; a production-shaped multiple-body regression covers the separation. This is a coverage-integrity correction, not a new sample or an attribution verdict.
|
||||
|
||||
The prospective power rule uses A01–A05 σ=12.6 and `2(1.96+0.84)^2σ²/Δ²`: Δ=20 requires seven runs/arm and Δ=10 requires 25. The 12-run self-imposed budget (46.16 minutes at 230.8s/run) cannot power either enabled-vs-unset or M5 after one measure-first and three gate-equivalent runs. The retained J enabled/control and J04–J07 clean/dirty samples are consequently descriptive; no absence or covariation from them is a powered perturbation or M5 verdict.
|
||||
|
||||
## FN-9150 corrective campaign — retained outcomes
|
||||
|
||||
The fixed campaign stack was pre-registered before its runs: observer threshold 0, ladder 1,000ms, 12,000ms setup/body/teardown watchdogs, 1,500/1,400/3,000ms probe/statement/drain bounds, one concurrent probe, queue timeout zero, and four probes. Every run used 27 workers, teardown diagnostics, JSON reporter capture, and a clean starting cluster unless labelled dirty. The intended two-run-per-arm allocation could not approach the pre-registered n=7/arm Δ=20 requirement; all comparisons below are explicitly **undetermined (underpowered)**.
|
||||
|
||||
| lane | failing files | peak / headroom | positions | coverage yield (joined / ladder / position-unobservable / unjoined) | outcome |
|
||||
|---|---:|---:|---|---|---|
|
||||
| enabled C1 | 34 | 66 / 31 | afterEach 8; afterAll 9; beforeAll 1; body 16 | 0 / 2 / 8 / 24 | Coverage floor missed; no attribution. |
|
||||
| unset U1 | 39 | 70 / 27 | beforeAll 5; afterAll 4; afterEach 15; body 15 | 0 / 0 / 15 / 24 | Control retained. |
|
||||
| enabled C2 | 35 | 70 / 27 | afterEach 15; afterAll 5; beforeAll 1; body 14 | 2 / 0 / 15 / 18 | Coverage floor missed; two joined rows do not identify a mechanism. |
|
||||
| unset U2 | 66 | 66 / 31 | afterEach 28; afterAll 11; beforeAll 8; body 19 | 0 / 0 / 28 / 38 | Control retained. |
|
||||
| M5 clean | 60 | 60 / 37 | beforeAll 6; afterEach 31; afterAll 8; body 14; setup 1 | 2 / 5 / 31 / 22 | Clean arm, one `cap` suppression. |
|
||||
| M5 dirty | 42 | 78 / 19 | body 18; afterAll 5; beforeAll 5; afterEach 14 | 2 / 1 / 14 / 25 | Dirty arm began with one test, one schema-template, and one pool database. |
|
||||
|
||||
Enabled mean failure count was 34.5 versus unset 52.5 (difference −18), but n=2/arm is below seven and the pre-declared Welch/rank comparison is not inferential. The 50% joined-plus-ladder coverage floor was missed in both enabled samples (2/34 and 2/35), so enabled rows are a coverage finding, not M2–M4 attribution. No observer probe was concurrency-suppressed in the perturbation lanes; one M5-clean record was cap-suppressed. The M5 clean/dirty difference is 18 failures at n=1/arm and is likewise **undetermined (underpowered)**. Final hygiene after every run was restored to seven backends and zero matching test/template/pool databases.
|
||||
|
||||
Payload-free breach and ladder rows remain location-only evidence: they cannot affirm a cluster-state mechanism. The new campaign again keeps every observed peak below the 97 ordinary-slot ceiling, which preserves M1's generic-form elimination. M2–M4 remain undecided because coverage is below floor; the successor needs an executing-test/hook-position join identity or a bracket for the remaining unowned positions, while M5 needs seven interleaved runs per arm under an allocated host budget.
|
||||
|
||||
## Remedies disqualified by this evidence
|
||||
|
||||
@@ -84,4 +113,4 @@ not a resolution.
|
||||
|
||||
## Successor measurement seam
|
||||
|
||||
FN-9151 owns the affirmed dirty-carryover structural seam: identify why retained `fusion_test_%`, `fusion_schema_template%` (including golden), and `fusion_pool_%` state covaries with the 27-worker failure population, then prove a remedy with a regression without changing timeouts, retries, skips, worker caps, quarantine, DDL, connection-budget, or admission behavior. A separate successor may improve timeout-boundary join coverage for M2–M4; it must preserve the default-off observer and repeat the perturbation control. Generic ordinary-slot exhaustion remains disqualified.
|
||||
FN-9152 owns the powered dirty-carryover measurement arm: obtain the host budget for seven interleaved clean and seven dirty runs before judging the retained covariation, without changing timeouts, retries, skips, worker caps, quarantine, DDL, connection-budget, or admission behavior. A separate successor may improve timeout-boundary join coverage for M2–M4; it must preserve the default-off observer and repeat the perturbation control. Generic ordinary-slot exhaustion remains disqualified.
|
||||
|
||||
@@ -549,9 +549,17 @@ A run is `insufficient-data` when its runner log lacks a complete `Test Files` s
|
||||
|
||||
### PostgreSQL timeout-boundary observer
|
||||
|
||||
`FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER=1` enables a diagnostic-only JSONL channel; when unset it creates no timer, probe connection, sink write, listener, or environment mutation. `observeBoundary()` wraps harness-owned setup and teardown work. `openBoundary()`/`closeBoundary()` span the shared harness's separate `beforeEach` and `afterEach` hooks, so `shared.body` measures test bodies without editing consumers. Closing is idempotent; an abandoned window is superseded by a later open or disposed without inventing a completion.
|
||||
`FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER=1` enables a diagnostic-only JSONL channel; when unset it creates no timer, probe connection, sink write, listener, or environment mutation. `observeBoundary()` wraps harness-owned setup and teardown work. `openBoundary()`/`closeBoundary()` span the shared harness's separate `beforeEach` and `afterEach` hooks, so `shared.body` measures test bodies without editing consumers. Each open body window receives a unique record `joinKey`; its stable file-level `supersessionKey` only closes a stale handle. Closing is idempotent; an abandoned window is superseded by a later open or disposed without inventing a completion.
|
||||
|
||||
The observer arms an unref'd watchdog at boundary start. Only a pending watchdog dispatches a maintenance-connection probe; completion records are host-only and emit when elapsed time meets `FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_THRESHOLD_MS` (default 2000; `0` measures every completion). A dispatched probe survives boundary settlement and appends when it resolves, with `settledDuringProbe`, `probeLatencyMs`, and optionally `probeStartDelayMs`; records can therefore be out of order and consumers must join by pid, worker, file, boundary, and timestamp rather than JSONL order. `flush()`/`dispose()` drain bounded pending and queued probes; a missed drain becomes `probeSuppressed:"drain-timeout"`. The enabled-only `beforeExit` hook performs the same best-effort drain.
|
||||
The observer arms an unref'd watchdog at boundary start. Only a pending watchdog dispatches a maintenance-connection probe; completion records are host-only and emit when elapsed time meets `FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_THRESHOLD_MS` (default 2000; `0` measures every completion). With `FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_LADDER_MS` above zero, an independent unref'd checkpoint timer synchronously emits keyed payload-free `kind:"progress"` rows and rearms at that spacing. The last progress row is an elapsed lower bound for an abandoned boundary, with the configured spacing as its resolution; it is never a PostgreSQL-state snapshot. A settle/reject emits a keyed `kind:"terminal"` outcome without error text and clears both timers exactly once.
|
||||
|
||||
The watchdog first synchronously appends a keyed `kind:"breach"`, `payloadFree:true` record, then appends its enriched `kind:"watchdog"` probe result. This two-phase form preserves a joinable elapsed boundary when Vitest abandons work during the watchdog-to-probe residual window; a payload-free breach never affirms a cluster-state mechanism. A dispatched probe survives boundary settlement and appends when it resolves, with `settledDuringProbe`, `probeLatencyMs`, and optionally `probeStartDelayMs`; records can therefore be out of order and consumers must join by the per-window `joinKey`, never by JSONL order or the shared `supersessionKey`. `flush()`/`dispose()` drain bounded pending and queued probes; a missed drain becomes `probeSuppressed:"drain-timeout"`. The enabled-only `beforeExit` hook performs the same best-effort drain.
|
||||
|
||||
<!-- FNXC:PgTimeoutBoundaryObserver 2026-08-19-16:11: FN-9150 records checkpoint progress because settle-only telemetry censors exactly the boundaries Vitest abandons. Checkpoints and breach rows establish elapsed coverage, never cluster-state attribution or a reason to extend a timeout. -->
|
||||
|
||||
<!-- FNXC:PgTimeoutBoundaryObserver 2026-08-19-16:06: FN-9150 records a payload-free breach before probe scheduling because an inherited 15-second Vitest budget can abandon a boundary before the maintenance probe resolves. The record is coverage evidence only, never a cluster-state attribution. -->
|
||||
|
||||
<!-- FNXC:PgTimeoutBoundaryObserver 2026-08-19-16:56: Consecutive shared-harness bodies reuse a lifecycle supersession key, but each emitted window has a unique join key so a later healthy terminal cannot hide an earlier abandoned ladder boundary. -->
|
||||
|
||||
The independent, tighten-only bounds are `statement timeout < probe timeout < inherited budget`, `queue timeout < probe timeout`, and `threshold <= per-boundary watchdog < inherited budget`. A probe timeout is deliberately **not** tied to a short watchdog: forced wiring runs need a fast watchdog and a real cluster round trip. Floors are `MIN_WATCHDOG_MS`, `MIN_PROBE_TIMEOUT_MS`, and `MIN_STATEMENT_TIMEOUT_MS`; an impossible floor yields `bounds-floor`, never a wider budget.
|
||||
|
||||
@@ -570,7 +578,7 @@ At concurrency 1 the cost profile matches the older single-flight observer. `pro
|
||||
|
||||
For an enabled-wiring check, first run disabled and normal enabled bounds to compare duration, then threshold 0 to verify setup, `shared.body`, and teardown completion records, then force per-boundary watchdogs below measured elapsed times while independently retaining a probe/drain timeout above cluster round-trip and raising probe cap/concurrency to at least 3. Every forced boundary must have a non-suppressed payload; `cap`, `concurrency`, `bounds-floor`, `drain-timeout`, and `error` require diagnosis, not acceptance. Files using the shared harness are body-observable; direct `createTaskStoreForTest` files deliberately have no harness-owned body bracket and must be reported as `body-unobservable`.
|
||||
|
||||
Pass `--boundary-observer <jsonl>`, `--body-unobservable-files <list>`, and `--fully-unobservable-files <list>` to `scripts/pg-loaded-failure-census.mjs`. It tolerates malformed lines and classifies each failing boundary as `cluster-implicated`, `host-implicated`, `template-convoy`, `body-unobservable`, or `unjoined`; failures from fully-unobservable files remain `unjoined` but are counted and listed separately so the known no-harness limitation is never mistaken for a missing observer join. Missing/empty input stays explicitly absent rather than a measured zero.
|
||||
Pass `--boundary-observer <jsonl>`, `--vitest-json <reporter.json>`, `--body-unobservable-files <list>`, and `--fully-unobservable-files <list>` to `scripts/pg-loaded-failure-census.mjs`. Capture the reporter per run with `--reporter=dot --reporter=json --outputFile=<reporter.json>`; it provides per-test duration and file identifiers but Vitest 3 provides no hook duration or failure position, so a hook-clock offset is unmeasurable rather than a point estimate. The census tolerates malformed and out-of-order rows, joins ladder/breach/enriched rows by key, and reports joined, `attributed-by-ladder`, `body-unobservable`, `position-unobservable`, and `unjoined` coverage counts plus probe suppression and latency distributions. `afterEach` consumer hooks are position-unobservable because the shared-harness body bracket closes before those hooks run. Failures from fully-unobservable files remain `unjoined` but are counted and listed separately so the known no-harness limitation is never mistaken for a missing observer join. Missing/empty input stays explicitly absent rather than a measured zero.
|
||||
|
||||
### PostgreSQL DDL loaded-lane acceptance metric
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ export const MAX_CONCURRENT_PROBES_CEILING = 8;
|
||||
|
||||
export type PgTimeoutBoundary = "setup" | "body" | "teardown";
|
||||
export type PgTimeoutBoundaryTrigger = "boundary-complete" | "boundary-watchdog";
|
||||
export type PgTimeoutBoundaryRecordKind = "progress" | "terminal" | "breach" | "watchdog";
|
||||
export type PgTimeoutBoundarySuppression = "cap" | "concurrency" | "bounds-floor" | "drain-timeout" | "error";
|
||||
|
||||
type TimerHandle = ReturnType<typeof setTimeout>;
|
||||
@@ -41,9 +42,18 @@ export interface PgTimeoutBoundaryRecord {
|
||||
readonly testName?: string;
|
||||
readonly boundary: PgTimeoutBoundary;
|
||||
readonly phase: string;
|
||||
/** Stable lifecycle label used to join reporter failures without line order. */
|
||||
readonly position: string;
|
||||
/** Per-window identity; progress-only keys prove an abandoned boundary. */
|
||||
readonly joinKey: string;
|
||||
/** Caller-owned lifecycle identity used only to supersede stale open windows. */
|
||||
readonly supersessionKey?: string;
|
||||
readonly kind: PgTimeoutBoundaryRecordKind;
|
||||
readonly trigger: PgTimeoutBoundaryTrigger;
|
||||
readonly elapsedMs: number;
|
||||
readonly boundaryIncomplete: boolean;
|
||||
/** A breach was synchronously recorded before its optional probe begins. */
|
||||
readonly payloadFree?: boolean;
|
||||
readonly settledDuringProbe?: boolean;
|
||||
readonly probeLatencyMs?: number;
|
||||
readonly probeStartDelayMs?: number;
|
||||
@@ -106,6 +116,14 @@ function envNumber(env: NodeJS.ProcessEnv, key: string, fallback: number, minimu
|
||||
return Number.isFinite(value) && value >= minimum ? Math.trunc(value) : fallback;
|
||||
}
|
||||
|
||||
/** Keep observer and census file joins stable across absolute and Windows paths. */
|
||||
function normalizeTestFile(value: string): string {
|
||||
const normalized = value.replaceAll("\\", "/");
|
||||
const root = normalized.indexOf("src/__tests__/");
|
||||
if (root >= 0) return normalized.slice(root);
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function defaultHostSample(eventLoopLagMs = 0): PgTimeoutBoundaryHostSample {
|
||||
const [loadavg1, loadavg5, loadavg15] = loadavg();
|
||||
const workers = Number(process.env.VITEST_MAX_WORKERS);
|
||||
@@ -129,6 +147,8 @@ interface Bounds {
|
||||
readonly maxConcurrentProbes: number;
|
||||
readonly probeQueueTimeoutMs: number;
|
||||
readonly maxProbes: number;
|
||||
/** Zero preserves the pre-FN-9150 settle/watchdog-only observer. */
|
||||
readonly ladderMs: number;
|
||||
readonly probeAllowed: boolean;
|
||||
readonly clamped: boolean;
|
||||
}
|
||||
@@ -139,6 +159,7 @@ function resolveBounds(env: NodeJS.ProcessEnv, hookBudget: number, bodyBudget: n
|
||||
const requestedProbe = envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_PROBE_TIMEOUT_MS", 1_500, 1);
|
||||
const requestedStatement = envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_STATEMENT_TIMEOUT_MS", Math.max(MIN_STATEMENT_TIMEOUT_MS, requestedProbe - 100), 1);
|
||||
const requestedDrain = envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_PROBE_DRAIN_TIMEOUT_MS", 3_000, 1);
|
||||
const ladderMs = envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_LADDER_MS", 0);
|
||||
const capBelowHook = Math.max(0, hookBudget - 1);
|
||||
// Floors make malformed tiny values observable, but never widen an inherited
|
||||
// budget: a budget below the floor disables probing for that boundary.
|
||||
@@ -180,6 +201,7 @@ function resolveBounds(env: NodeJS.ProcessEnv, hookBudget: number, bodyBudget: n
|
||||
maxConcurrentProbes,
|
||||
probeQueueTimeoutMs,
|
||||
maxProbes: envNumber(env, "FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_MAX_PROBES", 4, 1),
|
||||
ladderMs,
|
||||
probeAllowed,
|
||||
clamped,
|
||||
};
|
||||
@@ -189,11 +211,15 @@ interface BoundaryState {
|
||||
readonly handle: PgTimeoutBoundaryHandle;
|
||||
readonly boundary: PgTimeoutBoundary;
|
||||
readonly phase: string;
|
||||
readonly key?: string;
|
||||
/** Unique emitted-record key; it must never be shared by adjacent test bodies. */
|
||||
readonly joinKey: string;
|
||||
/** Stable caller key that only controls stale-window supersession. */
|
||||
readonly supersessionKey?: string;
|
||||
readonly startedAt: number;
|
||||
readonly timestamp: string;
|
||||
readonly host: PgTimeoutBoundaryHostSample;
|
||||
timer?: TimerHandle;
|
||||
ladderTimer?: TimerHandle;
|
||||
settled: boolean;
|
||||
watchdogFired: boolean;
|
||||
outcome?: "resolved" | "rejected";
|
||||
@@ -237,6 +263,19 @@ Once probes survive their boundary, strict single-flight would suppress body and
|
||||
teardown behind setup. The configurable limiter defaults to one (the inherited
|
||||
cost profile) but is raisable for the forced wiring gate; its ceiling prevents a
|
||||
bad environment from flooding PostgreSQL.
|
||||
|
||||
FNXC:PgTimeoutBoundaryObserver 2026-08-19-15:43:
|
||||
FN-9150 needs elapsed evidence for boundaries Vitest abandons before they
|
||||
settle. A payload-free checkpoint ladder is independent of probes, unref'd, and
|
||||
writes synchronously so its last progress record is a lower bound rather than a
|
||||
claim about PostgreSQL state. Terminal records make progress-only join keys the
|
||||
explicit abandoned-boundary signature.
|
||||
|
||||
FNXC:PgTimeoutBoundaryObserver 2026-08-19-16:06:
|
||||
A watchdog must append its keyed payload-free breach before scheduling a probe.
|
||||
Vitest can abandon the boundary during that residual window, so waiting for the
|
||||
probe result censors the timeout population this default-off observer measures.
|
||||
The breach locates a boundary but never asserts PostgreSQL state.
|
||||
*/
|
||||
export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserverOptions = {}): PgTimeoutBoundaryObserver {
|
||||
const env = options.env ?? process.env;
|
||||
@@ -270,15 +309,19 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv
|
||||
const unref = (timer: TimerHandle | undefined): void => {
|
||||
(timer as unknown as { unref?: () => void } | undefined)?.unref?.();
|
||||
};
|
||||
const emit = (state: BoundaryState, trigger: PgTimeoutBoundaryTrigger, fields: Partial<PgTimeoutBoundaryRecord>): void => {
|
||||
const emit = (state: BoundaryState, trigger: PgTimeoutBoundaryTrigger, kind: PgTimeoutBoundaryRecordKind, fields: Partial<PgTimeoutBoundaryRecord>, writeStderr = true): void => {
|
||||
const record: PgTimeoutBoundaryRecord = {
|
||||
timestamp: state.timestamp,
|
||||
pid: process.pid,
|
||||
...(env.VITEST_WORKER_ID ? { workerId: env.VITEST_WORKER_ID } : {}),
|
||||
...(options.testFile ? { testFile: options.testFile } : {}),
|
||||
...(options.testFile ? { testFile: normalizeTestFile(options.testFile) } : {}),
|
||||
...(options.testName?.() ? { testName: options.testName() } : {}),
|
||||
boundary: state.boundary,
|
||||
phase: state.phase,
|
||||
position: state.phase,
|
||||
joinKey: state.joinKey,
|
||||
...(state.supersessionKey ? { supersessionKey: state.supersessionKey } : {}),
|
||||
kind,
|
||||
trigger,
|
||||
elapsedMs: Math.max(0, now() - state.startedAt),
|
||||
boundaryIncomplete: !state.settled,
|
||||
@@ -296,7 +339,9 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv
|
||||
if (sink) {
|
||||
try { append(sink, `${JSON.stringify(record)}\n`); } catch { /* sink failures must not affect a test */ }
|
||||
}
|
||||
try { writeError(`[pg-timeout-boundary-observer] ${trigger} ${state.boundary}/${state.phase}`); } catch { /* diagnostic stderr is best effort */ }
|
||||
if (writeStderr) {
|
||||
try { writeError(`[pg-timeout-boundary-observer] ${trigger} ${state.boundary}/${state.phase}`); } catch { /* diagnostic stderr is best effort */ }
|
||||
}
|
||||
};
|
||||
const finalize = (pendingProbe: PendingProbe, suppression?: PgTimeoutBoundarySuppression, payload?: PgTimeoutBoundaryProbePayload): void => {
|
||||
if (pendingProbe.finalized) return;
|
||||
@@ -309,7 +354,7 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv
|
||||
admitQueued();
|
||||
}
|
||||
const state = pendingProbe.state;
|
||||
emit(state, "boundary-watchdog", {
|
||||
emit(state, "boundary-watchdog", "watchdog", {
|
||||
elapsedMs: bounds.watchdog[state.boundary],
|
||||
boundaryIncomplete: true,
|
||||
settledDuringProbe: state.settled,
|
||||
@@ -354,7 +399,16 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv
|
||||
};
|
||||
const requestProbe = (state: BoundaryState, eventLoopLagMs: number): void => {
|
||||
const watchdogAt = now();
|
||||
const pendingProbe: PendingProbe = { state, watchdogAt, host: hostSample(eventLoopLagMs), controller: new AbortController(), started: false, finalized: false };
|
||||
const host = hostSample(eventLoopLagMs);
|
||||
// This synchronous first phase survives disposal/worker termination while a
|
||||
// maintenance connection is still queued or executing.
|
||||
emit(state, "boundary-watchdog", "breach", {
|
||||
elapsedMs: bounds.watchdog[state.boundary],
|
||||
boundaryIncomplete: true,
|
||||
payloadFree: true,
|
||||
host,
|
||||
});
|
||||
const pendingProbe: PendingProbe = { state, watchdogAt, host, controller: new AbortController(), started: false, finalized: false };
|
||||
if (!options.probe) return finalize(pendingProbe, "error");
|
||||
if (!bounds.probeAllowed) return finalize(pendingProbe, "bounds-floor");
|
||||
pending.add(pendingProbe);
|
||||
@@ -373,20 +427,35 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv
|
||||
// as the host/cluster snapshot, without adding another timer or I/O path.
|
||||
requestProbe(state, Math.max(0, now() - (state.startedAt + bounds.watchdog[state.boundary])));
|
||||
};
|
||||
const armLadder = (state: BoundaryState): void => {
|
||||
if (bounds.ladderMs === 0) return;
|
||||
const checkpoint = (): void => {
|
||||
if (state.ladderTimer) clearTimer(state.ladderTimer);
|
||||
if (disposed || state.settled) return;
|
||||
emit(state, "boundary-complete", "progress", { boundaryIncomplete: true }, false);
|
||||
state.ladderTimer = setTimer(checkpoint, bounds.ladderMs);
|
||||
unref(state.ladderTimer);
|
||||
};
|
||||
state.ladderTimer = setTimer(checkpoint, bounds.ladderMs);
|
||||
unref(state.ladderTimer);
|
||||
};
|
||||
const arm = (state: BoundaryState): void => {
|
||||
state.timer = setTimer(() => fireWatchdog(state), bounds.watchdog[state.boundary]);
|
||||
unref(state.timer);
|
||||
armLadder(state);
|
||||
};
|
||||
const complete = (state: BoundaryState, outcome: "resolved" | "rejected"): void => {
|
||||
if (state.settled) return;
|
||||
state.settled = true;
|
||||
state.outcome = outcome;
|
||||
if (state.timer) clearTimer(state.timer);
|
||||
if (state.ladderTimer) clearTimer(state.ladderTimer);
|
||||
states.delete(state.handle.id);
|
||||
if (state.key) windows.delete(state.key);
|
||||
if (!state.watchdogFired && Math.max(0, now() - state.startedAt) >= bounds.thresholdMs) {
|
||||
emit(state, "boundary-complete", { outcome, boundaryIncomplete: false });
|
||||
}
|
||||
if (state.supersessionKey && windows.get(state.supersessionKey) === state) windows.delete(state.supersessionKey);
|
||||
// A terminal record is emitted regardless of threshold. Progress-only keys
|
||||
// are therefore the durable, resolution-bounded signature of abandonment.
|
||||
const reportCompletion = !state.watchdogFired && Math.max(0, now() - state.startedAt) >= bounds.thresholdMs;
|
||||
emit(state, "boundary-complete", "terminal", { outcome, boundaryIncomplete: false }, reportCompletion);
|
||||
};
|
||||
const beforeExit = (): void => { void flush(); };
|
||||
process.on("beforeExit", beforeExit);
|
||||
@@ -412,7 +481,8 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv
|
||||
return {
|
||||
enabled: true,
|
||||
observeBoundary<T>(boundary: PgTimeoutBoundary, phase: string, action: () => Promise<T>): Promise<T> {
|
||||
const state: BoundaryState = { handle: { id: nextHandle++ }, boundary, phase, startedAt: now(), timestamp: new Date(wallNow()).toISOString(), host: hostSample(), settled: false, watchdogFired: false };
|
||||
const handle = { id: nextHandle++ };
|
||||
const state: BoundaryState = { handle, boundary, phase, joinKey: `${process.pid}:${env.VITEST_WORKER_ID ?? "main"}:${normalizeTestFile(options.testFile ?? "unknown")}:${phase}:${handle.id}`, startedAt: now(), timestamp: new Date(wallNow()).toISOString(), host: hostSample(), settled: false, watchdogFired: false };
|
||||
states.set(state.handle.id, state);
|
||||
arm(state);
|
||||
let result: Promise<T>;
|
||||
@@ -424,10 +494,19 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv
|
||||
const superseded = Boolean(prior && !prior.settled);
|
||||
if (prior && !prior.settled) {
|
||||
if (prior.timer) clearTimer(prior.timer);
|
||||
if (prior.ladderTimer) clearTimer(prior.ladderTimer);
|
||||
prior.settled = true;
|
||||
states.delete(prior.handle.id);
|
||||
}
|
||||
const state: BoundaryState = { handle: { id: nextHandle++ }, boundary, phase, key, startedAt: now(), timestamp: new Date(wallNow()).toISOString(), host: hostSample(), settled: false, watchdogFired: false, ...(superseded ? { supersededOpenWindow: true } : {}) };
|
||||
const handle = { id: nextHandle++ };
|
||||
/*
|
||||
FNXC:PgTimeoutBoundaryObserver 2026-08-19-16:56:
|
||||
Shared-harness body hooks reuse a file-level lifecycle key across tests.
|
||||
That key may supersede a stale handle, but every body window needs its own
|
||||
emitted join key: a later healthy terminal must not mask an earlier
|
||||
abandoned body's ladder records in the failure census.
|
||||
*/
|
||||
const state: BoundaryState = { handle, boundary, phase, joinKey: `${key}:${handle.id}`, supersessionKey: key, startedAt: now(), timestamp: new Date(wallNow()).toISOString(), host: hostSample(), settled: false, watchdogFired: false, ...(superseded ? { supersededOpenWindow: true } : {}) };
|
||||
windows.set(key, state);
|
||||
states.set(state.handle.id, state);
|
||||
arm(state);
|
||||
@@ -441,7 +520,10 @@ export function createPgTimeoutBoundaryObserver(options: PgTimeoutBoundaryObserv
|
||||
async dispose(): Promise<void> {
|
||||
if (disposed) return;
|
||||
disposed = true;
|
||||
for (const state of states.values()) if (state.timer) clearTimer(state.timer);
|
||||
for (const state of states.values()) {
|
||||
if (state.timer) clearTimer(state.timer);
|
||||
if (state.ladderTimer) clearTimer(state.ladderTimer);
|
||||
}
|
||||
states.clear();
|
||||
windows.clear();
|
||||
await flush();
|
||||
|
||||
@@ -69,13 +69,105 @@ describe("PG timeout boundary observer", () => {
|
||||
await observed;
|
||||
pending.resolve(payload);
|
||||
await vi.advanceTimersByTimeAsync(0);
|
||||
expect(records(lines)).toEqual([expect.objectContaining({
|
||||
trigger: "boundary-watchdog", boundary: "body", phase: "shared.body", boundaryIncomplete: true,
|
||||
expect(records(lines)).toContainEqual(expect.objectContaining({
|
||||
trigger: "boundary-watchdog", kind: "watchdog", boundary: "body", phase: "shared.body", boundaryIncomplete: true,
|
||||
settledDuringProbe: true, cluster: payload.cluster, template: payload.template,
|
||||
})]);
|
||||
}));
|
||||
await observer.dispose();
|
||||
});
|
||||
|
||||
it("records progress-only ladder bounds for an abandoned boundary and clears unref'd timers", async () => {
|
||||
const lines: string[] = [];
|
||||
let now = 0;
|
||||
const timers: Array<{ callback: () => void; ms: number; unref: ReturnType<typeof vi.fn> }> = [];
|
||||
const clearTimer = vi.fn();
|
||||
const observer = createPgTimeoutBoundaryObserver({
|
||||
env: { ...env, FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_LADDER_MS: "5" },
|
||||
now: () => now,
|
||||
setTimer: ((callback, ms) => {
|
||||
const timer = { callback, ms, unref: vi.fn() };
|
||||
timers.push(timer);
|
||||
return timer as unknown as ReturnType<typeof setTimeout>;
|
||||
}) as typeof setTimeout,
|
||||
clearTimer: clearTimer as typeof clearTimeout,
|
||||
append: (_path, line) => lines.push(line),
|
||||
writeError: () => {},
|
||||
});
|
||||
observer.openBoundary("body", "shared.body", "abandoned");
|
||||
now = 5;
|
||||
timers.find((timer) => timer.ms === 5)?.callback();
|
||||
now = 10;
|
||||
timers.at(-1)?.callback();
|
||||
await observer.dispose();
|
||||
|
||||
const progress = records(lines).filter((record) => record.kind === "progress");
|
||||
expect(progress).toEqual([
|
||||
expect.objectContaining({ joinKey: "abandoned:1", elapsedMs: 5, boundaryIncomplete: true }),
|
||||
expect.objectContaining({ joinKey: "abandoned:1", elapsedMs: 10, boundaryIncomplete: true }),
|
||||
]);
|
||||
expect(records(lines).some((record) => record.kind === "terminal")).toBe(false);
|
||||
expect(timers.every((timer) => timer.unref.mock.calls.length === 1)).toBe(true);
|
||||
expect(clearTimer).toHaveBeenCalledTimes(4);
|
||||
});
|
||||
|
||||
it("keeps an abandoned shared-harness body window distinct from a later healthy body", async () => {
|
||||
const lines: string[] = [];
|
||||
let now = 0;
|
||||
const timers: Array<{ callback: () => void; ms: number; unref: ReturnType<typeof vi.fn> }> = [];
|
||||
const observer = createPgTimeoutBoundaryObserver({
|
||||
env: { ...env, FUSION_PG_TEST_TIMEOUT_BOUNDARY_OBSERVER_LADDER_MS: "5" },
|
||||
now: () => now,
|
||||
testFile: "/repo/packages/core/src/__tests__/postgres/shared-bodies.pg.test.ts",
|
||||
setTimer: ((callback, ms) => {
|
||||
const timer = { callback, ms, unref: vi.fn() };
|
||||
timers.push(timer);
|
||||
return timer as unknown as ReturnType<typeof setTimeout>;
|
||||
}) as typeof setTimeout,
|
||||
clearTimer: () => {},
|
||||
append: (_path, line) => lines.push(line),
|
||||
writeError: () => {},
|
||||
});
|
||||
// This is the file-level key passed by createSharedPgTaskStoreTestHarness
|
||||
// for consecutive beforeEach/afterEach body windows.
|
||||
const sharedHarnessKey = `${process.pid}:main:/repo/packages/core/src/__tests__/postgres/shared-bodies.pg.test.ts`;
|
||||
observer.openBoundary("body", "shared.body", sharedHarnessKey);
|
||||
now = 5;
|
||||
timers.find((timer) => timer.ms === 5)?.callback();
|
||||
const healthyBody = observer.openBoundary("body", "shared.body", sharedHarnessKey);
|
||||
observer.closeBoundary(healthyBody);
|
||||
|
||||
const emitted = records(lines);
|
||||
const abandonedProgress = emitted.find((record) => record.kind === "progress");
|
||||
const healthyTerminal = emitted.find((record) => record.kind === "terminal");
|
||||
expect(abandonedProgress).toMatchObject({ supersessionKey: sharedHarnessKey, boundary: "body", elapsedMs: 5 });
|
||||
expect(healthyTerminal).toMatchObject({ supersessionKey: sharedHarnessKey, boundary: "body" });
|
||||
expect(abandonedProgress?.joinKey).not.toBe(healthyTerminal?.joinKey);
|
||||
await observer.dispose();
|
||||
});
|
||||
|
||||
it("records a payload-free breach before an abandoned probe can resolve", async () => {
|
||||
vi.useFakeTimers();
|
||||
const lines: string[] = [];
|
||||
const pending = deferred<PgTimeoutBoundaryProbePayload>();
|
||||
const observer = createPgTimeoutBoundaryObserver({
|
||||
env,
|
||||
probe: () => pending.promise,
|
||||
append: (_path, line) => lines.push(line),
|
||||
writeError: () => {},
|
||||
});
|
||||
observer.openBoundary("setup", "database.clone", "abandoned-watchdog");
|
||||
await vi.advanceTimersByTimeAsync(10);
|
||||
expect(records(lines)).toContainEqual(expect.objectContaining({
|
||||
kind: "breach",
|
||||
payloadFree: true,
|
||||
joinKey: "abandoned-watchdog:1",
|
||||
boundary: "setup",
|
||||
}));
|
||||
const disposing = observer.dispose();
|
||||
await vi.advanceTimersByTimeAsync(100);
|
||||
await disposing;
|
||||
});
|
||||
|
||||
it("records watchdog scheduling drift as event-loop lag for host attribution", async () => {
|
||||
const lines: string[] = [];
|
||||
let now = 0;
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
parseDiagnosticsJsonl,
|
||||
parseBoundaryObserverJsonl,
|
||||
classifyBoundaryAttribution,
|
||||
parseVitestJson,
|
||||
stripAnsi,
|
||||
} from "../pg-loaded-failure-census.mjs";
|
||||
|
||||
@@ -58,6 +59,33 @@ test("joins out-of-order watchdog payloads by file and boundary, not line order"
|
||||
assert.equal(classifyBoundaryAttribution(failure, parsed.rows).classification, "cluster-implicated");
|
||||
});
|
||||
|
||||
test("attributes a progress-only observer key without inventing a cluster payload", () => {
|
||||
const failure = { file: "src/__tests__/postgres/abandoned.test.ts", lifecyclePosition: "test body" };
|
||||
const observer = [
|
||||
// Consecutive shared-harness bodies retain a common supersession identity,
|
||||
// but their emitted record join keys must remain per-window.
|
||||
{ testFile: failure.file, boundary: "body", kind: "progress", joinKey: "body-1", supersessionKey: "shared-file", elapsedMs: 5000 },
|
||||
{ testFile: failure.file, boundary: "body", kind: "progress", joinKey: "body-1", supersessionKey: "shared-file", elapsedMs: 10000 },
|
||||
{ testFile: failure.file, boundary: "body", kind: "terminal", joinKey: "body-2", supersessionKey: "shared-file", elapsedMs: 10 },
|
||||
];
|
||||
const attribution = classifyBoundaryAttribution(failure, observer);
|
||||
assert.equal(attribution.classification, "attributed-by-ladder");
|
||||
assert.equal(attribution.elapsedLowerBoundMs, 10000);
|
||||
const settledProgress = [
|
||||
{ testFile: failure.file, boundary: "body", kind: "progress", joinKey: "settled-body", elapsedMs: 5000 },
|
||||
{ testFile: failure.file, boundary: "body", kind: "terminal", joinKey: "settled-body", elapsedMs: 6000 },
|
||||
];
|
||||
assert.equal(classifyBoundaryAttribution(failure, settledProgress).classification, "unjoined");
|
||||
const breachOnly = [{ testFile: failure.file, boundary: "body", kind: "breach", payloadFree: true, joinKey: "body-2", trigger: "boundary-watchdog", host: { loadavg1: 0, cpuCount: 8, eventLoopLagMs: 0 } }];
|
||||
assert.equal(classifyBoundaryAttribution(failure, breachOnly).classification, "joined");
|
||||
|
||||
const reporter = parseVitestJson(JSON.stringify({ testResults: [{ name: `/repo/${failure.file}`, assertionResults: [{ fullName: "body timeout", duration: 15000, status: "failed", failureMessages: ["Test timed out in 15000ms"] }] }] }));
|
||||
assert.equal(reporter.malformed, false);
|
||||
assert.equal(reporter.rows[0].testFile, failure.file);
|
||||
assert.equal(reporter.rows[0].durationMs, 15000);
|
||||
assert.equal(parseVitestJson("truncated").malformed, true);
|
||||
});
|
||||
|
||||
test("keeps explicit unobservable sets and suppressed watchdog failures distinct from joined attribution", () => {
|
||||
const body = { file: "src/__tests__/postgres/direct.test.ts", lifecyclePosition: "test body" };
|
||||
assert.equal(classifyBoundaryAttribution(body, [], [body.file]).classification, "body-unobservable");
|
||||
@@ -71,6 +99,8 @@ test("keeps explicit unobservable sets and suppressed watchdog failures distinct
|
||||
assert.equal(census.fullyUnobservableFailingFileCount, 1);
|
||||
assert.deepEqual(census.fullyUnobservableFailingFiles, [fully.file]);
|
||||
assert.equal(census.attributions[0].boundaryAttribution.classification, "unjoined");
|
||||
const afterEach = { file: "src/__tests__/postgres/shared.test.ts", lifecyclePosition: "afterEach" };
|
||||
assert.equal(classifyBoundaryAttribution(afterEach, []).classification, "position-unobservable");
|
||||
});
|
||||
|
||||
test("requires a golden advisory waiter, not a holder, for template convoy attribution", () => {
|
||||
|
||||
@@ -33,7 +33,7 @@ export function parseDiagnosticsJsonl(text) {
|
||||
return { rows, malformedLines };
|
||||
}
|
||||
|
||||
function normalizeFile(value) {
|
||||
export function normalizeFile(value) {
|
||||
const normalized = String(value).replaceAll("\\", "/");
|
||||
// Vitest diagnostics use absolute paths while runner failures use repo paths.
|
||||
// Canonicalize at the test-root segment before any key-based observer join.
|
||||
@@ -102,6 +102,25 @@ export function parseBoundaryObserverJsonl(text) {
|
||||
return parseDiagnosticsJsonl(text);
|
||||
}
|
||||
|
||||
/** Vitest JSON exposes test durations; hook duration is not available in v4 output. */
|
||||
export function parseVitestJson(text) {
|
||||
try {
|
||||
const report = JSON.parse(String(text));
|
||||
const rows = Array.isArray(report?.testResults) ? report.testResults : [];
|
||||
return {
|
||||
rows: rows.flatMap((file) => (Array.isArray(file?.assertionResults) ? file.assertionResults : []).map((test) => ({
|
||||
testFile: normalizeFile(file?.name),
|
||||
position: classifyLifecyclePosition([test?.fullName, ...(test?.failureMessages ?? [])].join("\n")),
|
||||
durationMs: Number.isFinite(test?.duration) ? test.duration : null,
|
||||
status: test?.status ?? null,
|
||||
}))),
|
||||
malformed: false,
|
||||
};
|
||||
} catch {
|
||||
return { rows: [], malformed: true };
|
||||
}
|
||||
}
|
||||
|
||||
function boundaryForLifecycle(lifecyclePosition) {
|
||||
if (lifecyclePosition === "beforeAll hook" || lifecyclePosition === "in-test setup") return "setup";
|
||||
if (lifecyclePosition === "afterEach" || lifecyclePosition === "afterAll hook" || lifecyclePosition === "global setup-teardown") return "teardown";
|
||||
@@ -129,8 +148,23 @@ export function classifyBoundaryAttribution(failure, observerRecords, bodyUnobse
|
||||
if (boundary === "body" && bodyUnobservableFiles.includes(failure.file)) {
|
||||
return { classification: "body-unobservable", boundary, record: null, hostOnly: false, fullyUnobservable: false };
|
||||
}
|
||||
// Shared-harness afterEach only closes the body bracket; consumer afterEach
|
||||
// hooks are outside it, so an absent record is an explicit position limit.
|
||||
if (failure.lifecyclePosition === "afterEach") {
|
||||
return { classification: "position-unobservable", boundary, record: null, hostOnly: false, fullyUnobservable: false };
|
||||
}
|
||||
// A settled boundary can legitimately have earlier checkpoints. Only a key
|
||||
// with progress and no terminal record is an abandoned-boundary lower bound.
|
||||
const terminalKeys = new Set(sameFile.filter((record) => record.kind === "terminal" && typeof record.joinKey === "string").map((record) => record.joinKey));
|
||||
const abandonedProgress = sameFile.filter((record) => record.kind === "progress" && typeof record.joinKey === "string" && !terminalKeys.has(record.joinKey));
|
||||
if (abandonedProgress.length > 0) {
|
||||
const withElapsed = abandonedProgress.map((record) => ({ record, elapsedMs: Number(record.elapsedMs) })).filter(({ elapsedMs }) => Number.isFinite(elapsedMs));
|
||||
const latestBound = withElapsed.length > 0 ? withElapsed.reduce((latest, candidate) => candidate.elapsedMs > latest.elapsedMs ? candidate : latest) : null;
|
||||
return { classification: "attributed-by-ladder", boundary, record: latestBound?.record ?? abandonedProgress[0], hostOnly: true, elapsedLowerBoundMs: latestBound?.elapsedMs ?? null };
|
||||
}
|
||||
const watchdog = sameFile.filter((record) => record.trigger === "boundary-watchdog");
|
||||
const record = watchdog.find((candidate) => !candidate.probeSuppressed && candidate.cluster && candidate.template)
|
||||
?? watchdog.find((candidate) => candidate.kind === "watchdog")
|
||||
?? watchdog[0]
|
||||
?? sameFile[0]
|
||||
?? null;
|
||||
@@ -140,7 +174,15 @@ export function classifyBoundaryAttribution(failure, observerRecords, bodyUnobse
|
||||
const load = Number(record?.host?.loadavg1);
|
||||
const cpus = Number(record?.host?.cpuCount);
|
||||
const lag = Number(record?.host?.eventLoopLagMs);
|
||||
return { classification: (Number.isFinite(load) && Number.isFinite(cpus) && load >= cpus) || lag >= 100 ? "host-implicated" : "unjoined", boundary, record, hostOnly: true };
|
||||
if ((Number.isFinite(load) && Number.isFinite(cpus) && load >= cpus) || lag >= 100) {
|
||||
return { classification: "host-implicated", boundary, record, hostOnly: true };
|
||||
}
|
||||
// A two-phase breach is a real boundary join even when it has no cluster
|
||||
// payload. Keep it visible as coverage rather than misreporting no record.
|
||||
if (record?.kind === "breach" && record?.payloadFree === true) {
|
||||
return { classification: "joined", boundary, record, hostOnly: true };
|
||||
}
|
||||
return { classification: "unjoined", boundary, record, hostOnly: true };
|
||||
}
|
||||
const template = record.template ?? {};
|
||||
// A holder alone is not a convoy: only a non-owner waiter proves the
|
||||
@@ -167,9 +209,17 @@ export function summarizeBoundaryObserver(records, failures, bodyUnobservableFil
|
||||
const reason = row?.probeSuppressed === "single-flight" ? "concurrency" : row?.probeSuppressed;
|
||||
if (reason) suppression[reason] = (suppression[reason] ?? 0) + 1;
|
||||
}
|
||||
const attributionCounts = Object.fromEntries(Object.entries(Object.groupBy(attributions, (row) => row.boundaryAttribution.classification)).map(([key, values]) => [key, values.length]));
|
||||
return {
|
||||
boundaryObserver: rows.length ? "present" : "absent",
|
||||
boundaryAttributionHistogram: Object.fromEntries(Object.entries(Object.groupBy(attributions, (row) => row.boundaryAttribution.classification)).map(([key, values]) => [key, values.length])),
|
||||
boundaryAttributionHistogram: attributionCounts,
|
||||
joinedCoverageYield: {
|
||||
joined: (attributionCounts.joined ?? 0) + (attributionCounts["cluster-implicated"] ?? 0) + (attributionCounts["host-implicated"] ?? 0) + (attributionCounts["template-convoy"] ?? 0),
|
||||
attributedByLadder: attributionCounts["attributed-by-ladder"] ?? 0,
|
||||
bodyUnobservable: attributionCounts["body-unobservable"] ?? 0,
|
||||
positionUnobservable: attributionCounts["position-unobservable"] ?? 0,
|
||||
unjoined: attributionCounts.unjoined ?? 0,
|
||||
},
|
||||
observerProbeSuppression: suppression,
|
||||
settledDuringProbeCount: rows.filter((row) => row?.settledDuringProbe === true).length,
|
||||
fullyUnobservableFailingFiles: attributions.filter((row) => row.boundaryAttribution.fullyUnobservable).map((row) => row.file),
|
||||
@@ -211,7 +261,7 @@ export function summarizeDiagnostics(diagnostics) {
|
||||
};
|
||||
}
|
||||
|
||||
export function buildCensus({ log, diagnostics = [], boundaryObserver = [], bodyUnobservableFiles = [], fullyUnobservableFiles = [], ordinarySlotCeiling = null, subjects = [] }) {
|
||||
export function buildCensus({ log, diagnostics = [], boundaryObserver = [], vitestJson = [], bodyUnobservableFiles = [], fullyUnobservableFiles = [], ordinarySlotCeiling = null, subjects = [] }) {
|
||||
const summary = parseFileSummary(log);
|
||||
if (!summary.complete) {
|
||||
return { status: "insufficient-data", reason: "missing Test Files summary", totalFiles: null, failingFiles: [], failingFileCount: null };
|
||||
@@ -222,6 +272,12 @@ export function buildCensus({ log, diagnostics = [], boundaryObserver = [], body
|
||||
}
|
||||
const diagnosticSummary = summarizeDiagnostics(diagnostics);
|
||||
const observerSummary = summarizeBoundaryObserver(boundaryObserver, failingFiles, bodyUnobservableFiles, fullyUnobservableFiles);
|
||||
const reporterFiles = new Set(vitestJson.map((row) => row.testFile).filter(Boolean));
|
||||
const observerFiles = new Set(boundaryObserver.map(observerFile).filter(Boolean));
|
||||
const reporterJoin = {
|
||||
observerFilesWithoutReporter: [...observerFiles].filter((file) => !reporterFiles.has(file)),
|
||||
reporterFilesWithoutObserver: [...reporterFiles].filter((file) => !observerFiles.has(file)),
|
||||
};
|
||||
const ceiling = Number.isFinite(ordinarySlotCeiling) && ordinarySlotCeiling >= 0 ? ordinarySlotCeiling : null;
|
||||
return {
|
||||
status: "measured",
|
||||
@@ -235,16 +291,18 @@ export function buildCensus({ log, diagnostics = [], boundaryObserver = [], body
|
||||
backendHeadroom: ceiling != null && diagnosticSummary.peakBackends != null ? ceiling - diagnosticSummary.peakBackends : null,
|
||||
...diagnosticSummary,
|
||||
...observerSummary,
|
||||
reporterJoin,
|
||||
};
|
||||
}
|
||||
|
||||
function parseArgs(args) {
|
||||
const result = { log: undefined, diagnostics: undefined, boundaryObserver: undefined, bodyUnobservableFiles: undefined, fullyUnobservableFiles: undefined, ordinarySlotCeiling: null, subjects: [] };
|
||||
const result = { log: undefined, diagnostics: undefined, boundaryObserver: undefined, vitestJson: undefined, bodyUnobservableFiles: undefined, fullyUnobservableFiles: undefined, ordinarySlotCeiling: null, subjects: [] };
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const argument = args[index];
|
||||
if (argument === "--log") result.log = args[++index];
|
||||
else if (argument === "--diagnostics") result.diagnostics = args[++index];
|
||||
else if (argument === "--boundary-observer") result.boundaryObserver = args[++index];
|
||||
else if (argument === "--vitest-json") result.vitestJson = args[++index];
|
||||
else if (argument === "--body-unobservable-files") result.bodyUnobservableFiles = args[++index];
|
||||
else if (argument === "--fully-unobservable-files") result.fullyUnobservableFiles = args[++index];
|
||||
else if (argument === "--ordinary-slot-ceiling") result.ordinarySlotCeiling = Number(args[++index]);
|
||||
@@ -259,7 +317,8 @@ if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href)
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const parsed = args.diagnostics ? parseDiagnosticsJsonl(readFileSync(args.diagnostics, "utf8")) : { rows: [], malformedLines: 0 };
|
||||
const observer = args.boundaryObserver ? parseBoundaryObserverJsonl(readFileSync(args.boundaryObserver, "utf8")) : { rows: [], malformedLines: 0 };
|
||||
const reporter = args.vitestJson ? parseVitestJson(readFileSync(args.vitestJson, "utf8")) : { rows: [], malformed: false };
|
||||
const bodyUnobservableFiles = args.bodyUnobservableFiles ? readFileSync(args.bodyUnobservableFiles, "utf8").split(/\\r?\\n/).map(normalizeFile).filter(Boolean) : [];
|
||||
const fullyUnobservableFiles = args.fullyUnobservableFiles ? readFileSync(args.fullyUnobservableFiles, "utf8").split(/\\r?\\n/).map(normalizeFile).filter(Boolean) : [];
|
||||
console.log(JSON.stringify({ ...buildCensus({ log: readFileSync(args.log, "utf8"), diagnostics: parsed.rows, boundaryObserver: observer.rows, bodyUnobservableFiles, fullyUnobservableFiles, ordinarySlotCeiling: args.ordinarySlotCeiling, subjects: args.subjects }), malformedDiagnosticLines: parsed.malformedLines, malformedBoundaryObserverLines: observer.malformedLines }, null, 2));
|
||||
console.log(JSON.stringify({ ...buildCensus({ log: readFileSync(args.log, "utf8"), diagnostics: parsed.rows, boundaryObserver: observer.rows, vitestJson: reporter.rows, bodyUnobservableFiles, fullyUnobservableFiles, ordinarySlotCeiling: args.ordinarySlotCeiling, subjects: args.subjects }), malformedDiagnosticLines: parsed.malformedLines, malformedBoundaryObserverLines: observer.malformedLines, malformedVitestJson: reporter.malformed }, null, 2));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user