feat(api): hourly catalog backfill for decoded-but-unfetched vehicles
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
prefetch was reactive-only (on decode) and processInit read top-level categories straight from DB, so a vehicle decoded but never viewed got no catalog. Add a production-only sweep so no decoded vehicle is left without catalog data. - processInit self-seeds top categories via getCategoryTree when DB has none (fetches+inserts top groups from PL24/PSA/EMEX), closing the never-viewed gap for both reactive and backfill paths - new backfill-scan job + hourly cron: phase 1 queues decoded vehicles with zero parts, phase 2 rolling createdAt-cursor rescan of all decoded vehicles (prefetch-init is idempotent → gap-fills partial ones) - guardrails: skip wave if queue backlog > 1000, per-source cooldown, business-hours window (isWithinTimeWindow), batch <=20, in-flight guard - PRODUCTION ONLY: gated on NODE_ENV both at cron registration and in processBackfillScan; dev has a separate DB and must not scrape Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -24,22 +24,34 @@ export async function checkCooldown(redis: RedisService, source: string): Promis
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check PL24 business hours (09:00–18:00 Europe/Istanbul).
|
||||
* Throws RateLimitError with delay until next 09:00 if outside window.
|
||||
*/
|
||||
export function checkTimeWindow(source: string): void {
|
||||
if (source !== "pl24" && source !== "parts-catalogs") return;
|
||||
|
||||
/** Current hour (0–23) in Europe/Istanbul. */
|
||||
function currentIstanbulHour(): number {
|
||||
const hourStr = new Intl.DateTimeFormat("en-US", {
|
||||
timeZone: "Europe/Istanbul",
|
||||
hour: "numeric",
|
||||
hour12: false,
|
||||
}).format(new Date());
|
||||
const h = Number.parseInt(hourStr, 10);
|
||||
return Number.parseInt(hourStr, 10);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether `source` may be scraped right now.
|
||||
* PL24 → 09:00–18:00, parts-catalogs → 09:00–19:00 (Europe/Istanbul).
|
||||
* EMEX and everything else have no window (always true).
|
||||
*/
|
||||
export function isWithinTimeWindow(source: string): boolean {
|
||||
if (source !== "pl24" && source !== "parts-catalogs") return true;
|
||||
const h = currentIstanbulHour();
|
||||
const endHour = source === "parts-catalogs" ? 19 : 18;
|
||||
if (h < 9 || h >= endHour) {
|
||||
return h >= 9 && h < endHour;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check PL24 business hours (09:00–18:00 Europe/Istanbul).
|
||||
* Throws RateLimitError with delay until next 09:00 if outside window.
|
||||
*/
|
||||
export function checkTimeWindow(source: string): void {
|
||||
if (!isWithinTimeWindow(source)) {
|
||||
throw new RateLimitError(msUntilNext9AM());
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user