On SIGTERM/SIGINT the worker only disconnected prisma/redis and exited — the
BullMQ Workers were never closed, so in-flight job locks were never released. On
every deploy the orchestrator killed the worker mid-flight, the next container
saw the half-finished jobs as "stalled", re-ran them, and the dead worker's
pending moveToFinished surfaced as `Missing lock for job
repeat:nightly-refresh:… moveToFinished`, with duplicate nightly-refresh firings
during the container overlap.
- startScheduledJobs / startInsightPipeline / startContentPipeline now return
their Worker so index.ts can close them.
- shutdown() closes all workers FIRST (releases locks, drains in-flight),
bounded by a 15s race so an in-flight job can't block past the orchestrator's
stop grace period, then disconnects prisma/redis.
- nightly worker now uses lockDuration 5min + stalledInterval 60s (matching the
pipeline/content workers); the 30s default could expire during panel-backup
(pg_dump) and trip the same stalled → re-run → Missing-lock cycle.
The scheduler config itself was already correct (nightly@03:00, audit@03:30,
panel-backup@04:00 — verified against the Redis job-scheduler ZSET; next fires
were exactly 03:00/03:30/04:00). tsc --noEmit clean; dedup + tagger smoke 30/30.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
7a56a72037