fix(worker): graceful BullMQ worker shutdown (stop Missing-lock on deploy) #7

root · 2026-06-03T21:52:46+03:00

root commented

2026-06-03 21:52:46 +03:00

Workers were never closed on SIGTERM, so deploys killed them mid-flight and the next container saw stalled jobs -> Missing-lock churn + duplicate nightly-refresh. shutdown now closes all workers first (locks released, bounded 15s); nightly gets the same 5min lock as the other queues. tsc clean; smoke 30/30.

root added 1 commit 2026-06-03 21:52:46 +03:00

fix(worker): graceful BullMQ worker shutdown to stop Missing-lock churn on deploy 7a56a72037

On SIGTERM/SIGINT the worker only disconnected prisma/redis and exited — the
BullMQ Workers were never closed, so in-flight job locks were never released. On
every deploy the orchestrator killed the worker mid-flight, the next container
saw the half-finished jobs as "stalled", re-ran them, and the dead worker's
pending moveToFinished surfaced as `Missing lock for job
repeat:nightly-refresh:… moveToFinished`, with duplicate nightly-refresh firings
during the container overlap.

- startScheduledJobs / startInsightPipeline / startContentPipeline now return
  their Worker so index.ts can close them.
- shutdown() closes all workers FIRST (releases locks, drains in-flight),
  bounded by a 15s race so an in-flight job can't block past the orchestrator's
  stop grace period, then disconnects prisma/redis.
- nightly worker now uses lockDuration 5min + stalledInterval 60s (matching the
  pipeline/content workers); the 30s default could expire during panel-backup
  (pg_dump) and trip the same stalled → re-run → Missing-lock cycle.

The scheduler config itself was already correct (nightly@03:00, audit@03:30,
panel-backup@04:00 — verified against the Redis job-scheduler ZSET; next fires
were exactly 03:00/03:30/04:00). tsc --noEmit clean; dedup + tagger smoke 30/30.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

root merged commit f8d38426e6 into main

2026-06-03 21:52:48 +03:00

root referenced this issue from a commit

2026-06-03 21:52:48 +03:00

Merge pull request 'fix(worker): graceful BullMQ worker shutdown (stop Missing-lock on deploy)' (#7) from fix/worker-graceful-shutdown into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: root/sp#7