fusion/docs/solutions/workflow-selection-per-tick-cache.md

category, module, date, problem_type, severity, applies_when, component, tags, related_components

category	module	date	problem_type	severity	applies_when	component	tags	related_components
architecture-patterns	packages/engine/src/scheduler.ts	2026-08-12	performance	high	Looping a per-task store read inside a scheduler/engine pass that visits the same task multiple times Seeing Drizzle buildQueryFromSourceParams dominate a CPU profile (~70%) or runaway pg_stat idx_scan counts on a selection/config table Adding any caller-owned per-pass cache to a hot engine loop	scheduler	performance query-storm workflow-selection drizzle per-pass-cache fnxc-workflowscheduling	development_workflow scheduler workflow_resolution

Workflow selection read-once-per-tick in the scheduler (the RUFU-073 query storm)

Symptom

Production CPU sat at 62–70% and the health API took 0.77–2.0s. A cpuprofile showed 70.6% of the node event loop inside Drizzle's buildQueryFromSourceParams (the SQL-string compiler) — the engine was building identical SQL statements nonstop, not waiting on the database. pg_stat_user_tables confirmed the loop was a cache-miss loop on reads:

• project.task_workflow_selection.idx_scan grew ~232 q/s nonstop (121M cumulative index-scans, only 281 inserts).
• project.config workflow_prompt_overrides ~108 q/s (127M scans, 0 inserts).

The DB was not the bottleneck (PG did 5 selects in 46ms, no locks, ~15 connections) — the node process was flooded by per-read Drizzle SQL-string construction.

Root cause

resolveTaskParkedColumns(store, taskId) resolved the task's workflow IR via resolveWorkflowIrForTask(store, taskId) without a selection cache. That is a separate PostgreSQL select of task_workflow_selection (plus a Drizzle SQL build) per call. In a single scheduler event the SAME task is parked-resolved up to ~6×:

task:moved merge reconciliation, task:updated unpause wake, planning-finished wake, approval-cleared wake, task:deleted dependency reconciliation, and the agent-link rollback path. With 22 active projects that composed to ~340 q/s of Drizzle SQL building.

The fix: a caller-owned per-tick selection cache

resolveWorkflowIrForTask(store, taskId, irCache?, selectionCache?) and its resolveWorkflowIrForTaskWithProvenance(..., selectionCache?) already accept a WorkflowSelectionCache = Map<taskId, WorkflowSelection | undefined>. The scheduler now creates a fresh selection cache at the top of each event/loop scope and threads it through every resolveTaskParkedColumns(store, taskId, selectionCache?) call. Note the argument positions differ: the selection cache is the 3rd argument of resolveTaskParkedColumns, whereas it is the 4th argument (after the optional irCache) of resolveWorkflowIrForTask(store, taskId, irCache?, selectionCache?). It is also threaded through the emitHighOverlapFanoutWarnings escalation sweep and the PR-hydration sweep:

• Each task's selection is read at most once per tick (not once per park-resolution).
• The cache is strictly per-call/per-pass: a fresh Map per event/sweep, discarded at scope end.
• A selection WRITE by a later pass is always observed, because the next pass creates a fresh Map.

In-flight read coalescing (race the first cut hit)

The first iteration put the coalescer only in the sync fallback, and concurrent wake closures sharing one cache could each see the cache still-empty (the .has check runs before the first await resolves) and all hit the DB. The fix tracks an in-flight promise per caller-owned cache object in a WeakMap<WorkflowSelectionCache, Map<taskId, Promise>>: only the first closure performs the read; later concurrent closures await the same promise. The weak key binds it to that pass, so it is NOT a global/infinite cache and auto-releases when the pass cache is GC'd.

Invariant (FNXC:WorkflowScheduling)

Selection caches are per-call/per-scheduler-pass only and are strictly invalidated next poll — never a global/infinite LRU. A throwing read is deliberately not cached so transient PostgreSQL failures are retried next pass; therefore instrumentation must count reads, not infer them from cache-key presence.

Verification contract

• A regression test drives one scheduler tick that clears unpause + planning + approval wakes for the same task and asserts the selection is read exactly once (not 3×), that N distinct tasks read ≤ N selections, that an empty task set reads zero, and that a second tick re-reads (proving the cache is pass-scoped, so a selection write between ticks is observed).
• The existing resolver contract tests (workflow-ir-selection-cache, workflow-ir-resolution-provenance) pin the per-call caching semantics; the sync-only store path dedups the same way.
• Production verification: task_workflow_selection idx_scan growth rate must drop from ~232 q/s to below ~30 q/s, health API < 0.5s, CPU < 40%.