Reorganize the suite-only flake register around its current decision state.
- Separate active observations, gate evictions, and archived closed records while preserving entry anchors.
- Mark entry 1 as an escalated second sighting and restore FN-9126/FN-9131, FN-9128, and FN-9127 ownership.
- Add regression checks for active counts, exact states and owners, and testing-guide anchors.
Files changed:
.../suite-only-flakes-observed-register.md | 189 +++++++++++++--------
scripts/__tests__/observed-flake-register.test.mjs | 65 +++++++
2 files changed, 181 insertions(+), 73 deletions(-)
Fusion-Task-Id: FN-9145
Fusion-Task-Lineage: 39b3292a-db48-4be4-878c-7d2b026abae3
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
**Problem:** Each dashboard/engine project consumer polled
`task_deleted` outbox on a fixed 5s setInterval, so ~44 per-project
consumers thundered together on the same cadence — an idle DB query
storm and CPU hot-spot even when projects were paused/idle.
**Fix:** The outbox consumer reschedules itself from each poll outcome:
an idle poll (zero events) grows the next delay by
`TASK_DELETED_OUTBOX_BACKOFF_STEP_MS` toward `MAX_POLL_MS`, with ±20%
jitter so the consumers de-synchronize; a poll that delivered events
resets to the fast base. A paused/idle project drains its outbox and
backoff alone drops the DB load.
**Includes:** regression test (bounded jitter + idle growth),
performance changeset, solution doc, deploy handoff script.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Performance**
- Reduced unnecessary idle polling by gradually increasing the polling
interval, up to 60 seconds, with bounded timing variation.
- Restored the faster 5-second polling cadence when new events, waits,
or transient errors occur.
- Preserved event ordering, delivery guarantees, acknowledgements, and
independent behavior across concurrent consumers.
- **Documentation**
- Added guidance on polling behavior, deployment verification, and
monitoring targets.
- **Tests**
- Added coverage for backoff growth, jitter limits, event bursts,
concurrent consumers, error handling, retries, and clean shutdown.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Problem:** Fusion had no Prometheus-scrapeable metrics surface, so
production CPU/health/DB/fleet observability had to be inferred from
logs and ad-hoc monitoring.
**Fix:** Expose a Prometheus-text `/metrics` HTTP endpoint on the
dashboard with runtime + domain samplers: request latency quantiles
(p50/p95/max), last-request-age freeze indicator, CPU user/system,
RSS/heap, child-process spawn rates by kind, git children count, PG tps,
active/idle/total projects, running agents, and board task counts.
Scraped at 5s by the existing Prometheus/Grafana stack
(fusion-observability).
**Includes:** full test coverage for the Prometheus text parser and each
sampler.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a public, unauthenticated `/metrics` endpoint with
Prometheus-formatted dashboard, runtime, process, database, and domain
metrics.
* Added request latency, resource usage, child-process, project
activity, agent, and task metrics.
* Metrics remain available in headless mode and provide stable snapshots
without triggering additional data writes or on-demand I/O.
* **Documentation**
* Documented metric families, sampling behavior, and degraded-operation
handling.
* **Tests**
* Added comprehensive coverage for collection, formatting, parsing,
endpoint behavior, and sampler lifecycle.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Fusion <noreply@runfusion.ai>
When the dev child had not reported a bound port within 60s, the wrapper fell
back to the configured port and published a tunnel to it. In the case the
port fix exists for — a container whose own Fusion owns 4040 — that hands out
a dev-looking URL serving a completely different instance, with only a
passing "which may not be it" warning to show for it.
Observed with a dev server stopped on the interactive `Run central db now?
(Y/n)` prompt: it never listens, so it never reports, so the tunnel published
the container's Fusion instead.
A missing tunnel is a visible problem that explains itself; a tunnel to the
wrong app is a silent one. The wrapper now waits, unbounded, printing a
notice once a minute that names the interactive-prompt case. An explicit
--tunnel=PORT still publishes immediately, since it names a target the dev
child knows nothing about.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Terminal PTYs already lived in a server-side registry whose WebSocket attach
supports several viewers, but the tab list is per-browser localStorage and a
browser with no stored tabs skipped the session listing entirely (FN-7686's
cold-open optimization) and spawned its own PTY. Two people on one Fusion —
or one person in a second browser — therefore never saw each other's
terminals and quietly accumulated parallel sessions.
- A zero-tab client adopts the server's sessions, oldest first, so every
client converges on the same set. A client with stored tabs still only
validates them; adopting there would resurrect tabs the user closed here.
- FN-7686's guarantee weakens from "never waits" to "waits at most
ADOPT_LIST_TIMEOUT_MS (1.5s), then behaves exactly as before": auto-create
fires on a 0ms timer, so a background list can never win that race. Its
regression test and comment now state the bounded contract.
- Closing a tab asks which close was meant — detach here (PTY keeps running
for other viewers) or end the session for everyone. alwaysAsk is set: this
gates an informed choice and both wrong answers are destructive. Desktop
and mobile close controls both route through it.
- A footer Reopen control lists sessions the server still runs that this
browser is not showing and reattaches to them, so detaching is not a
one-way door.
Also fixes a multi-viewer data bug: the attach called
getScrollbackAndClearPending(), discarding queued output and deleting a slice
of every already-attached viewer's live stream. It now flushes pending output
to existing viewers first, then reads scrollback for the newcomer, so each
viewer receives it exactly once.
Drive-by: closeTab's dependency array was empty while its body reads
projectId, freezing it at the first render's project.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`pnpm dev --tunnel` published a bare URL under a "public, unauthenticated"
header. That label was wrong for the flag's own default target: --tunnel with
no port aims at the dashboard, which is bearer-token gated, so the recipient
of a shared link got a 401 with no token to supply.
resolveDevTunnelAuth() now classifies the target and the banner says what is
actually true of it:
token dashboard with auth on — prints the token and a ?token= link,
resolved from FUSION_DASHBOARD_TOKEN, FUSION_DAEMON_TOKEN,
then ~/.fusion/settings.json
token-pending first run, token not minted yet — defers to the dashboard's
own startup banner
no-auth --no-auth is on; the dashboard really is open
foreign a non-dashboard port; Fusion has no auth to lend it, the only
genuinely ungated case
Auth resolves at banner time rather than flag-parse time so a token the dev
child mints on a first authenticated run is already readable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator case: someone works inside a remote Fusion (a container, a shared box),
starts a dev server there, and needs to view it from their own browser. The dev
server binds inside that machine, so without a tunnel the only options are port
publishing or a VPN — both needing cooperation from whoever owns the host.
pnpm dev --tunnel # tunnels the dashboard port (PORT, default 4040)
pnpm dev --tunnel=5173 # tunnels a Vite dev server instead
pnpm dev --tunnel dashboard # tunnel the default port AND run the dashboard
FUSION_DEV_TUNNEL=1 pnpm dev
Cloudflare QUICK tunnels are usable here precisely because a dev server is HTTP:
no account, no domain, no card. The TCP endpoints that SSH would have needed
require a card (ngrok) or a domain plus Zero Trust (Cloudflare) — that asymmetry
is why this exists for HTTP only, and it is recorded in the module header so the
next person does not retry the SSH variant.
Design decisions:
- Tunnel failure is NON-FATAL. A missing cloudflared or a tunnel that never
publishes a URL logs and is skipped; losing a preview URL must never cost the
operator their dev loop.
- Watch-mode restarts reuse the existing tunnel. A fresh quick tunnel hands out a
different hostname each time, which would invalidate an already-shared link.
- `--tunnel` consumes a following token only when it is numeric, so
`--tunnel dashboard` forwards `dashboard` to the dev command rather than
tunnelling port NaN. That is the bug this flag shape invites, so it is tested.
Verified end to end in a container: a dev server bound to 127.0.0.1 inside it was
fetched from the public internet through the tunnel (200, correct body). Also
confirmed that tunnelling the DASHBOARD port does not weaken auth — unauthenticated
requests through the tunnel return 401 for /api/tasks, /api/settings and
/api/artifacts, with only /api/health open by design.
Adding two fields to parseDevWrapperArgs' return broke two existing strict toEqual
assertions; those were updated rather than loosened to toMatchObject. 27 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
**Problem:** Scheduler was re-reading each task's
`task_workflow_selection` once per park-resolution (sweep, hold-release,
moved, unpause/wake), causing a nonstop PostgreSQL query storm (~232
idx_scan/s) on idle polling — a major engine CPU hot-spot.
**Fix:** Memoize the workflow selection per scheduler tick/event —
thread a shared, per-event selection cache through
`resolveWorkflowIrForTask` and all park-resolution handlers, then throw
it away. Each task resolves its parked columns with at most one read of
`task_workflow_selection` per tick. A selection write is always observed
on the next event's fresh cache (never a global/infinite LRU).
**Includes:** regression test asserting the once-per-tick read
invariant, performance changeset + per-tick-cache solution doc,
deploy+verify handoff script, and the parallel quarantine-ledger merge
(origin FN-9125 + RUFU-072 OOM entries both retained).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Performance Improvements**
* Reduced repeated workflow-selection reads during scheduler ticks and
related event processing.
* Improved scheduler and health API responsiveness through per-operation
caching and read deduplication.
* Preserved existing behavior, including retry handling for failed reads
and synchronous data-store support.
* **Documentation**
* Added architectural guidance covering workflow-selection performance,
caching behavior, and verification criteria.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Fusion <noreply@runfusion.ai>
Operator report: during a container login there was nowhere obvious to paste the
redirect URL and no sign of what the app was waiting on. The flow was split
across a pre-flight confirm that warned about paste-back and vanished, a card
that shrank to a disabled "Waiting for login…" chip, and the paste field
rendered inline in that card below the fold of a scrolling modal.
ProviderLoginDialog now opens with the flow and stays until it ends: a two-step
progress list, a button to re-open a lost sign-in tab, the paste field, and the
terminal outcome inline instead of a toast that disappears while the operator is
in another browser tab.
Three defects found and fixed while verifying it in a real container:
- It sank behind the onboarding modal and clicks landed on the modal instead.
createPortal relocates the DOM node but NOT the React tree, so pointer events
bubbled to the host FloatingWindow, which raises itself to a fresh
nextFloatingZ() on every pointerdown — each click in the dialog lifted the
window above it. Fixed by rendering the dialog as a sibling of the window,
claiming z once on open (it was calling nextFloatingZ() inline on every render
of a modal that re-renders on a 2s poll), and stopping propagation on the
overlay. Ratcheted for every portaled .modal-overlay.
- Spacing did not match any other dialog: it hand-rolled header/action padding
instead of using .modal-header/.modal-actions, and padded each child
separately. Every row now shares var(--modal-padding) — verified at a uniform
17px inset across header, steps, paste prompt, field, Submit, and actions.
- The paste field was invisible (.form-input fills with var(--surface), and so
does .modal — measured #0c0c0e on #0c0c0e), Submit was a 25px row-density
btn-sm, and both could scroll out of reach. The field now sinks to var(--bg)
with var(--border-strong), Submit takes standard control padding, and the
paste region is pinned outside the scroll area.
Dialog anatomy rules (spacing primitives, portal/stacking) documented in
docs/dashboard-guide.md.
Verified: 441 dashboard tests including 4 new dialog tests, eslint, dashboard
typecheck, and the rendered dialog measured in a container build.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operator report from a containerized dashboard: OpenAI Codex login never opened
a browser window at all, and floating windows still needed the FN-8015 follow-up.
- pi's `AuthPrompt` is a discriminated union — text, secret, select, manual_code —
and FusionAuthStorage.login's interaction shim flattened every variant into
`onPrompt({message, placeholder})`, discarding `type` and a select's `options`.
pi's Codex `login()` OPENS with `prompt({type:"select"})` (Browser vs Device
code) before emitting any auth URL, so the dashboard answered the method picker
with the promise that waits for a pasted code — input the UI never solicits,
because nothing had been surfaced yet. The flow hung until the route's 30s
kickoff timeout: "Login initiation timed out", no window. The route's
onSelect/selectOauthOption has had the right answer since FN-5917, but the
callback was dead code from the moment login moved to pi's ModelRuntime.
Verified against a real container: the login endpoint now returns Codex's
auth.openai.com URL in 0.03s instead of timing out after 30s.
- Promote FN-8766's outboard east/NE/SE resize targets from Task Detail to every
desktop window. With FN-8015's body gutter deleted, a hosted scrollbar sits
flush against the painted edge where those hit zones used to cover it (issue
#2140); moving the targets outside the shell keeps it grabbable without
insetting anything. That needs the host to stop clipping, so the body and its
direct child inherit the corner radius — only 8 of ~30 callers set that
themselves — and phones re-assert clipping since they hide every handle.
- Document the fixed OAuth callback ports (Anthropic 53692, Codex 1455) and
PI_OAUTH_CALLBACK_HOST for Docker: without them the browser callback cannot
reach the container's loopback listener, which is why subscription logins
appeared to fail there.
Verified: 14989 dashboard tests, 58 engine auth-storage tests (4 new, covering
each prompt type), pnpm test:gate, eslint, and both typechecks all pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three defects found while bringing up a container from a clean checkout:
- The dashboard's vite build (~5.7k modules) exceeded V8's default old-space on a
stock Docker Desktop VM and aborted the image build with "Ineffective
mark-compacts near heap limit" (exit 134). Raise the ceiling for that RUN only.
- The documented `-v fusion-home:/home/node/.fusion` mount seeded a root-owned
named volume over a path absent from the image, so embedded Postgres initdb hit
"Permission denied", the supervisor burned its 4 restarts, and the container went
unhealthy on first run. Pre-create the directory node-owned so a fresh named
volume inherits it; document that bind mounts still need a host-side chown.
- Drop the dependency-graph plugin's tsconfig path mapping for the taskStuck module
deleted in 2eae0b2507 / 29d94e0fa3.
Verified: full `docker build` from a clean export of this tree succeeds unpatched,
and a run against brand-new named volumes with no manual chown reaches health=healthy
with /api/health 200.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
**Problem:** Self-healing repeatedly spawned git children
(status/rev-parse/for-each-ref) for paused/idle projects on every sweep,
and certain repair sweeps ran unbounded — a spawn/git storm that spiked
CPU and I/O on the production host.
**Fix:** Bound self-healing git work for paused projects (skip/cooldown)
and cap the repair sweeps so the engine stops churning git processes
when there is nothing actionable. Includes in-process-runtime pause-gate
+ self-healing pause-storm regression tests.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Paused projects no longer trigger Git-intensive self-healing
maintenance.
- Pause and unpause transitions now correctly stop and resume
maintenance scheduling.
- Global and engine-level pauses are handled consistently.
- **Improvements**
- Active-project Git maintenance is limited to an hourly cadence,
reducing unnecessary activity.
- Merge-metadata recovery is capped at 25 items per cycle for more
predictable processing.
- Database and filesystem housekeeping continues during pauses.
- **Documentation**
- Updated architecture and runtime documentation to describe pause-aware
maintenance behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- preserve the gridlock notification wall-clock cooldown across
transient detector clears
- add a regression test for clear-then-rediscover behavior during the
cooldown
- document the cooldown contract and add a patch changeset
## Test plan
- `corepack pnpm --filter @fusion/engine exec vitest run
src/__tests__/notifier.test.ts --project=engine-default
--reporter=verbose -t 'suppresses the same gridlock after a transient
resolution during cooldown'`
- `corepack pnpm --filter @fusion/engine typecheck`
- `corepack pnpm build`
- `corepack pnpm changeset status --output
/tmp/fusion-gridlock-changeset-status.json`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Gridlock notifications now remain suppressed during the 15-minute
cooldown, even if the condition temporarily clears and reappears.
- Prevents repeated notifications caused by transient detector-state
changes.
- **Documentation**
- Updated gridlock notification behavior documentation to reflect the
persistent cooldown.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Preserve the quarantined plugin-runner coverage while recording the inconclusive reproduction campaign.
- Record two loaded-suite runs at each of 2, 6, and 8 workers.
- Keep all 82 plugin-runner tests, the ledger entry, and the default-lane exclusion through 2026-08-30.
- Clarify that opt-in strict checks do not block normal PR lanes.
Files changed:
.../test-failures/suite-only-flakes-observed-register.md | 13 ++++++++++++-
docs/testing.md | 3 +++
2 files changed, 15 insertions(+), 1 deletion(-)
Fusion-Task-Id: FN-9135
Fusion-Task-Lineage: 239425e5-7393-4b8a-a114-0a14f0065847
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
Document the second sighting of the project-identity PostgreSQL timeout and hand off structural diagnosis.
- Record the sterile 27-worker reproduction against the integration SHA
- Preserve pass, failure, and skip counters for both PostgreSQL-directory runs
- Mark quarantine as policy-forbidden and assign root-cause follow-up to FN-9131
Files changed:
.../solutions/test-failures/suite-only-flakes-observed-register.md | 7 +++++++
1 file changed, 7 insertions(+)
Fusion-Task-Id: FN-9126
Fusion-Task-Lineage: 9e31b00f-268b-46fa-af7a-58973c93d8fc
Co-authored-by: Fusion (runfusion.ai) <noreply@runfusion.ai>
