root/sase.tr
1
0
Fork 0
Code Issues 5 Pull Requests 3 Actions 7 Packages Projects Releases Wiki Activity

feat(backfill): widen scrape window for pcat + double batch size
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
Details

Browse Source 
Backfill was sweeping only 20 vehicles/hour and skipping pcat outside
09:00–19:00 Istanbul, leaving the catalog backlog (115 pl24 + 63 pcat +
27 emex zero-parts vehicles) crawling forward at ~528 parts/day and the
biggest user vehicle (Toyota Corolla 2026) unchanged across 24h.

Two related changes — both unblocked by the Redis-persisted warm JWT pool
(aa4d055 + dcf7e06) which keeps pcat captures alive 24/7 instead of
needing the 09:00–19:00 office-hours assumption:

* isWithinTimeWindow: parts-catalogs no longer gated — the warm pool +
  Redis hydration cover the cold-start case the old office-hours rule
  was working around. PL24 keeps its 09:00–18:00 window because the
  upstream rate-limit is still tighter outside it.
* BACKFILL_BATCH_SIZE 20 → 40 — twice as many vehicles per wave, still
  protected by MAX_BACKLOG=1000 self-throttle and per-source cooldown
  (prefetch:activity:<source>) so live-user traffic still gets priority.

Combined effect: pcat goes from ~10h/day to 24h/day, batch doubles —
roughly 3× backfill throughput. Worst-case proxy spend tracked by the
DataImpulse daily cap; if a wave saturates upstream the rate-limit
handler (c7e59b9) defers without burning attempts.
This commit is contained in:
Semih Yesilyurt
2026-06-02 02:00:16 +03:00
parent 0f647b3931
commit 316a5e014c
2 changed files with 16 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
apps/api/src/jobs/prefetch-utils.ts
View File

@@ -65,14 +65,18 @@ function currentIstanbulHour(): number {
/**
* Whether `source` may be scraped right now.
* PL24 → 09:00–18:00, parts-catalogs → 09:00–19:00 (Europe/Istanbul).
* EMEX and everything else have no window (always true).
* PL24 → 09:00–18:00 (Europe/Istanbul); upstream rate-limit is still
* tighter outside that window so user-facing requests benefit from the
* guard. EMEX, parts-catalogs, and everything else have no window
* (always true). parts-catalogs used to be 09:00–19:00 too, but the
* Redis-persisted warm JWT pool (commits aa4d055 + dcf7e06) now keeps
* captures alive 24/7 so backfill can sweep through the night when the
* upstream is most idle.
*/
export function isWithinTimeWindow(source: string): boolean {
if (source !== "pl24" && source !== "parts-catalogs") return true;
if (source !== "pl24") return true;
const h = currentIstanbulHour();
const endHour = source === "parts-catalogs" ? 19 : 18;
return h >= 9 && h < endHour;
return h >= 9 && h < 18;
}
/**

9
apps/api/src/jobs/prefetch-worker.service.ts
View File

@@ -27,8 +27,13 @@ import { CATALOG_PREFETCH_QUEUE } from "./queues/catalog-prefetch.queue";
const MAX_DEPTH = 5;
// ── Backfill scan (hourly cron) tuning ──
/** Vehicles queued per scan wave. */
const BACKFILL_BATCH_SIZE = 20;
/**
* Vehicles queued per scan wave. Was 20 while pcat needed business-hours
* gating; doubled to 40 once the warm JWT pool + Redis hydration made the
* scrape stack 24/7. The MAX_BACKLOG guard below still gates run-away
* pileups, and per-source cooldown still yields to live users.
*/
const BACKFILL_BATCH_SIZE = 40;
/** Skip the wave entirely if the queue already has more than this many jobs pending. */
const BACKFILL_MAX_BACKLOG = 1000;
/** In-flight guard TTL (seconds) — safety net if a run dies without clearing. */