Semih Yesilyurt 3514272936 perf(emex): replace networkidle + 2s sleeps with targeted selector waits ...

EMEX leaf parts fetch was averaging 12.5s cold per category (Honda Civic
prod traces). Almost all of that lived in the scraper's two waterfall page
loads, each of which (a) waited for `networkidle` — which only fires after
analytics/ads stop chattering — then (b) slept an unconditional 2 seconds.
The actual extraction work runs in <100ms once the DOM is parsed.

Three focused changes in scripts/emex-vin-scraper.js:

1) getParts() resolves the Unit.aspx URL via a plain `fetch()` of
   QuickDetails.aspx instead of opening it in the browser. Probe showed
   the page is fully server-rendered and the anchor is present in the
   initial HTML, so the browser stage is dead weight (~3s saved). Falls
   back to a browser load if the fetch fails or the anchor isn't there,
   so catalogs that gate the link behind JS still work.

2) Both remaining `page.goto()` calls in getParts/getCategories/
   getCategoryTree drop `waitUntil:'networkidle'` for `'domcontentloaded'`
   plus a targeted `waitForSelector('img.dragger' | 'a[href*=…]', {timeout:5000-8000})`.
   `.catch(()=>null)` makes the wait advisory — the evaluate() below has
   its own null-safe fallbacks — but in practice the selector is present
   well before networkidle would have fired.

3) Removes the two unconditional `setTimeout(r,2000)` sleeps in getParts.
   They predate the selector-wait pattern and were belt-and-braces.

Expected: ~12.5s → ~3-5s cold leaf fetch. Warm path (DB-cached) is
unchanged at ~45ms. Dev-only push for verification before main merge.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-18 09:24:38 +03:00

46 KiB

Raw Blame History

View Raw