root/fusion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

FN-9148: add PostgreSQL loaded-failure census

Browse Source 
Add retained-log census tooling and evidence for PostgreSQL loaded-lane timeout investigations.

- Parse complete Vitest logs and optional diagnostics without opening PostgreSQL or running tests.
- Classify failing files, lifecycle positions, failure shapes, backend headroom, waits, and watchdog data.
- Cover high-failure, healthy, malformed-diagnostics, and truncated-log cases with fixtures.
- Document the reproduced population, unsupported remedies, and successor measurement requirements.

Files changed:
 ...res-loaded-lane-unrelated-failure-population.md |  80 ++++++++++
 docs/testing.md                                    |   8 +
 .../fixtures/pg-loaded-failure-census/high-run.txt |  77 +++++++++
 .../fixtures/pg-loaded-failure-census/high.jsonl   |   4 +
 .../fixtures/pg-loaded-failure-census/low-run.txt  |   4 +
 .../pg-loaded-failure-census/truncated-run.txt     |   2 +
 .../__tests__/pg-loaded-failure-census.test.mjs    |  65 ++++++++
 scripts/pg-loaded-failure-census.mjs               | 172 +++++++++++++++++++++
 8 files changed, 412 insertions(+)

Fusion-Task-Id: FN-9148

Fusion-Task-Lineage: c632a9d0-b823-4416-ab46-0d834e850007

Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
This commit is contained in:
gsxdsm
2026-08-19 06:24:01 -07:00
parent 687990c0a6
commit 161edaa694
8 changed files with 412 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

80
docs/solutions/test-failures/postgres-loaded-lane-unrelated-failure-population.md Normal file
View File

@@ -0,0 +1,80 @@
---
category: test-failures
module: testing
problem_type: loaded_postgresql_timeout_population
applies_when:
- "The 27-worker core PostgreSQL directory lane reports unrelated hook or test timeouts"
- "A PostgreSQL loaded-lane remedy is proposed from runner-log impressions"
tags:
- postgres
- vitest
- diagnostics
- timeout
- census
---
# PostgreSQL loaded-lane unrelated failure population
## Verdict: reproduced but unattributed
FN-9148 reproduced the unrelated population in three of five pre-registered,
diagnostics-enabled 27-worker directory runs. A03/A04/A05 reported 45, 35, and
32 failed files respectively, while their observed peaks were 63, 75, and 71
backends below the 97 ordinary-slot ceiling. This establishes the fan-out
symptom; it does not establish a cause or authorize a harness remedy.
## Method
`scripts/pg-loaded-failure-census.mjs` is a cluster-free parser. It reads a
retained Vitest runner log and teardown-diagnostics JSONL, then reports every
failing file, its lifecycle position and shape, snapshot peak/headroom, waits,
phase-duration statistics, and watchdog/probe-degradation counts. It labels
campaign subjects rather than excluding them. A missing `Test Files` summary
is `insufficient-data`; a complete passing summary is a measured zero-failure
run.
The host had 28 CPUs, so requested 27 workers resolved to 27. PostgreSQL was
15.15 with `max_connections=100`, three reserved connections, 128MB
`shared_buffers`, a five-minute checkpoint timeout, and 1GB maximum WAL. Test
databases were enumerated and explicitly reset between primary samples.
| lane | outcome |
|---|---|
| 27-worker directory A01–A05 | red: 13, 24, 45, 35, 32 failed files; peaks 73, 61, 63, 75, 71 |
| 12-worker directory | green, measured zero failures |
| isolated `project-identity.test.ts` | green, measured zero failures |
| configured four-fork PG gate | green, measured zero failures |
| default core lane | green, measured zero failures |
The reproduced runs mixed setup/teardown and body timeouts. A03, for example,
had four beforeAll, 19 afterEach, five afterAll, and 17 body failures; its
watchdog snapshots included checkpoint, ProcSignalBarrier, and object-lock
waits. The checkpointed task document `evidence` is the detailed durable
record.
## Discrimination table
| mechanism | verdict | evidence / missing discriminator |
|---|---|---|
| M1 ordinary backend exhaustion | undecided; generic version contradicted | Peaks stay 22–36 below 97, but watchdog-only snapshots cannot rule out a per-user/per-database limit or a missed transient peak. |
| M2 DDL serialization | undecided | Hook concentration and checkpoint/catalog/object waits are observations, not per-failed-hook DDL correlation. |
| M3 golden-template/advisory convoy | undecided | No template-owner/lock-wait timeline was captured. |
| M4 host CPU/event-loop starvation | undecided | Host load was material, but teardown-only probes cannot show PostgreSQL idle versus in-flight at setup/body timeout time. |
| M5 dirty-cluster carryover | undecided | Clean resets still reproduced, but no controlled clean/dirty covariation measurement was run. |
## Remedies disqualified by this evidence
Do not raise timeouts, add retries or skips, alter worker caps, quarantine core
PostgreSQL files, wire the retained connection-budget/admission primitives, or
change DDL paths. Reducing generic connection demand is specifically unsupported:
the reproduced peaks are below ordinary capacity. A green comparison lane is
not a resolution.
## Successor measurement seam
The successor must design a separately reviewed, default-off observer that can
join a setup/body or teardown timeout to host pressure, the active SQL/lock
state, and template ownership without changing harness execution. It must run
three instrumented reproductions and an unset-environment control before
attribution. A controlled dirty-cluster arm is also required for M5. Reuse the
census tool and retain all logs/JSONL; do not return to impression-based claims.

8
docs/testing.md
View File

@@ -537,6 +537,14 @@ For a pre-registered loaded-flake campaign, record PostgreSQL version, `max_conn
<!-- FNXC:PgDdlLaneMetric 2026-08-17-00:59: FN-9134 requires a pre-registered end-to-end band because teardown watchdogs become structurally meaningless when cleanup leaves the hook. The parser is intentionally report-only and the alternating samples are campaign observations, never Vitest retries. -->
<!-- FNXC:PgLoadedFailureCensus 2026-08-19-12:41: FN-9148 requires a retained loaded-lane failure population to be classified without contacting PostgreSQL, so a complete green log remains evidence rather than being conflated with a truncated capture. -->
### PostgreSQL loaded-failure census
Use `scripts/pg-loaded-failure-census.mjs` to inspect an already-retained Vitest runner log and its optional teardown-diagnostics JSONL. The script never opens a cluster or runs tests. Supply `--log`, `--diagnostics`, and the recorded `--ordinary-slot-ceiling`; repeat `--subject` to label campaign files without dropping them. Its output contains total and failing files, lifecycle and failure-shape breakdowns, observed backend peak/headroom, wait histogram, phase-duration order statistics, and watchdog/probe-degradation counts.
A run is `insufficient-data` when its runner log lacks a complete `Test Files` summary or when the reported failed-file count cannot be reconciled to parsed failure blocks. A complete passing summary instead produces `status: "measured"` with `failingFileCount: 0`; never treat that zero as missing evidence or coerce incomplete input to a healthy result.
### PostgreSQL DDL loaded-lane acceptance metric
Use `scripts/pg-ddl-lane-metric.mjs` before judging a PostgreSQL DDL structural candidate. Run at least seven **interleaved** control/candidate pairs at `VITEST_MAX_WORKERS=12`; preserve one diagnostics JSONL sink and complete runner log per invocation. The exact lane is: