feat(api): hourly catalog backfill for decoded-but-unfetched vehicles
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled

prefetch was reactive-only (on decode) and processInit read top-level
categories straight from DB, so a vehicle decoded but never viewed got
no catalog. Add a production-only sweep so no decoded vehicle is left
without catalog data.

- processInit self-seeds top categories via getCategoryTree when DB has
  none (fetches+inserts top groups from PL24/PSA/EMEX), closing the
  never-viewed gap for both reactive and backfill paths
- new backfill-scan job + hourly cron: phase 1 queues decoded vehicles
  with zero parts, phase 2 rolling createdAt-cursor rescan of all decoded
  vehicles (prefetch-init is idempotent → gap-fills partial ones)
- guardrails: skip wave if queue backlog > 1000, per-source cooldown,
  business-hours window (isWithinTimeWindow), batch <=20, in-flight guard
- PRODUCTION ONLY: gated on NODE_ENV both at cron registration and in
  processBackfillScan; dev has a separate DB and must not scrape

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-27 14:04:32 +03:00
parent f7d5f03572
commit d0ee4df57f
3 changed files with 199 additions and 11 deletions

View File

@@ -24,22 +24,34 @@ export async function checkCooldown(redis: RedisService, source: string): Promis
}
}
/**
* Check PL24 business hours (09:0018:00 Europe/Istanbul).
* Throws RateLimitError with delay until next 09:00 if outside window.
*/
export function checkTimeWindow(source: string): void {
if (source !== "pl24" && source !== "parts-catalogs") return;
/** Current hour (023) in Europe/Istanbul. */
function currentIstanbulHour(): number {
const hourStr = new Intl.DateTimeFormat("en-US", {
timeZone: "Europe/Istanbul",
hour: "numeric",
hour12: false,
}).format(new Date());
const h = Number.parseInt(hourStr, 10);
return Number.parseInt(hourStr, 10);
}
/**
* Whether `source` may be scraped right now.
* PL24 → 09:0018:00, parts-catalogs → 09:0019:00 (Europe/Istanbul).
* EMEX and everything else have no window (always true).
*/
export function isWithinTimeWindow(source: string): boolean {
if (source !== "pl24" && source !== "parts-catalogs") return true;
const h = currentIstanbulHour();
const endHour = source === "parts-catalogs" ? 19 : 18;
if (h < 9 || h >= endHour) {
return h >= 9 && h < endHour;
}
/**
* Check PL24 business hours (09:0018:00 Europe/Istanbul).
* Throws RateLimitError with delay until next 09:00 if outside window.
*/
export function checkTimeWindow(source: string): void {
if (!isWithinTimeWindow(source)) {
throw new RateLimitError(msUntilNext9AM());
}
}