fix(backfill): in-process scan timer (cron marker buried behind 444k backlog) #136

root · 2026-06-12T12:14:22+03:00

root commented

2026-06-12 12:14:22 +03:00

Live finding: the BullMQ cron scan marker is a delayed job promoted to the wait HEAD (LPUSH, lifo ignored) while the worker pops the tail → buried behind the ~444k backlog for weeks (ic frozen at 28). Run the hourly scan on an in-process timer instead (NX-locked, prod-host gated); Phase-1 still enqueues lifo init jobs that jump the wait list. Full api suite 272 passed, pnpm build + biome green.

root added 1 commit 2026-06-12 12:14:22 +03:00

fix(backfill): run hourly scan on an in-process timer (cron marker gets buried)

QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled

Details

7d10107f4b

The BullMQ cron scan never fired on prod (ic frozen at 28 for days, 0
scans completed) even after the lifo change. Root cause, confirmed live:
a cron scheduler enqueues a DELAYED marker, and BullMQ promotes delayed
jobs to the wait-list HEAD via LPUSH — `promoteDelayedJobs` ignores lifo —
while the worker pops from the tail. Behind the ~444k deep-drill backlog
the scan marker sat at wait position 1 (head) with ~444k jobs ahead of it
→ unreachable for weeks. So scheduler-level lifo cannot work; the marker
must not go through the queue at all.

Run the scan on an in-process hourly timer in PrefetchWorkerService
instead (kickoff 60s after boot, then hourly), prod-host gated and
single-fired via a Redis NX lock. The scan's Phase-1 still enqueues
NON-delayed `lifo` init jobs, which DO jump the wait list (RPUSH tail →
popped next), so newly-decoded / zero-parts vehicles get onboarded ahead
of the backlog while it drains. jobs.module now removes the old cron
scheduler so stale markers stop being produced; the backfill-scan job
handler stays so any already-queued legacy marker runs harmlessly.

Tested: spec covers the NX lock (run when acquired / skip when held) plus
the existing lifo + Phase-1/Phase-2 gating; full api suite 272 passed,
`pnpm build` + biome green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

root merged commit 6a54e63c0f into main

2026-06-12 12:14:25 +03:00

root referenced this issue from a commit

2026-06-12 12:14:26 +03:00

Merge pull request 'fix(backfill): in-process scan timer (cron marker buried behind 444k backlog)' (#136) from fix/backfill-inprocess-scan into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: root/sase.tr#136