One row per proxied upstream attempt (pcat call/capture/validate, emex http)
written fire-and-forget by the new ProxyTelemetryService (buffered, capped,
errors swallowed — telemetry can never hurt the request path).
- banned = upstream 403/429 (IP-block signal), distinct from auth/data errors
- sticky legs (pcat capture, emex floxy) carry a session_key; pcat capture
also resolves the actual residential exit IP via a parallel ipify probe
through the same sticky session → concrete banned-IP tracking
- rotating legs log provider + outcome (ban *rate* instead of per-IP)
- 30-day retention piggybacked on the query-cleanup job
Feeds the Süper Panel /analytics/proxy page (provider grading + banned IPs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The hardcoded Floxy account (d739255e819b) ran out of balance → 402 Payment
Required on every CONNECT → pcat decode dead on prod since ~07:16 UTC 2026-06-11
(pcat has no proxy fallback; EMEX limped on its DataImpulse fallback). New funded
account cac4b0d96a80 verified working (call leg 401-reachable, capture leg 200).
Updates the hardcoded defaults in all 3 Floxy consumers (pcat auth, emex http,
emex browser); env overrides PCAT_FLOXY_USER/PASS + EMEX_FLOXY_USER/PASS unchanged.
Follow-up: wire FLOXY_USER/PASS into docker-compose env so future rotations are
Coolify-env-only (no code change / rebuild).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EMEX egress was DataImpulse-first: fetchEmexHtml tried 3 dead-prone DataImpulse
ports before falling to the Floxy fallback, and the browser scraper launched on a
random DataImpulse port. With ~50% of DataImpulse ports dead, this burned the 25s
decode budget before reaching Floxy (observed: a Ford VIN where pcat correctly
returned null fast, then EMEX's DataImpulse retries ate the budget).
ssd tokens are replayed from emex's own HTML and are NOT strictly IP-bound on the
.aspx endpoints (the prior random-port-per-call primary proved that), so a sticky
Floxy IP across the flow is strictly safer.
- EMEX_PROXY_PROVIDER (default "floxy" | "dataimpulse" | "none"). floxy →
fetchEmexHtml schedule = [floxy, floxy, dataimpulse] (Floxy sticky primary,
rolls IP on transport failure, single DataImpulse last-ditch). dataimpulse →
legacy [di,di,di,floxy,floxy]. Default HTTP agent (image-dims) follows provider.
- EmexBrowserService launches chromium on a sticky Floxy session (one residential
exit IP for the browser lifetime; fresh id per relaunch) under the same flag.
- Reuses the existing EMEX_FLOXY_* config (same Floxy account pcat now uses).
Rollback: EMEX_PROXY_PROVIDER=dataimpulse. typecheck+biome+11 emex tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer entered VXKUPHNKSP4032040 (2023 Corsa F) and got a bare
'Marka desteklenmiyor' — VXK was missing from all three WMI maps, so
identifyBasic could not even name the brand. PL24's opel_parts catalog
ends at CORSA-E (2015-2019, verified via the model-picker page), so the
catalog itself stays a gap, but the user now gets the honest
'Opel 2023 olarak tanındı, katalog henüz mevcut değil' message and the
miss is logged for the coverage backlog.
- WMI_BRAND_MAP + PL24_WMI_SERVICE_MAP + EMEX CATALOG_MAP: VXK -> Opel
- DECODE_CHAIN_VERSION 2 -> 3 (brand-mapping change; invalidates the
stale negative cache for the customer's VIN)
- vin-cache-keys.spec: lock-key assertion checked the bare version char,
which now appears inside the test VIN — assert the ✌️ segment instead
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add server-side PostHog feature-flag evaluation to the API and wire three
high-leverage uses. The flags live in PostHog (project 127747), dormant and
fail-open, so this commit is a no-op until POSTHOG_PERSONAL_API_KEY is set and
a switch is flipped.
Phase 1 - upstream kill switches:
PostHogService gains local flag evaluation (onlyEvaluateLocally +
sendFeatureFlagEvents:false -> zero decode latency, no VIN leak) and
isSourceLive(). Guards at each source's decode entry (parts-catalogs, emex,
pl24 whole-source + per-brand via LEGACY_ARCH_SOURCE_TAG) let a flailing
upstream be disabled from the PostHog UI in ~5s instead of a
code-fix -> dev -> prod redeploy. Fail-open: any unresolved flag keeps the
source live, so a PostHog outage can never black out decoding.
Phase 2 - guarded rollout primitive:
isEnabled()/variant() with VIN bucketing, ready to ramp a new decode/parser
path 0->100% behind a decode-*-v2 flag (recipe in feature-flags-strategy.md).
Phase 4 - remote-config ops tuning:
prefetch-worker reads cfg-backfill-tuning to retune backfill
batchSize/maxBacklog/businessHoursOnly live; malformed/missing -> the
compiled-in constants.
POSTHOG_PERSONAL_API_KEY wired into the api + worker compose blocks (empty ->
flags inert, no added latency). Tests updated for the new constructor params.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
emex's ssd tokens appear IP-bound: a rotating proxy sends each hop of a
decode→tree→drill→parts flow from a different exit IP, so emex rejects the
ssd and returns empty (the rotating-Floxy verify still seeded 0). Pin the
Floxy exit IP with a sticky session (password suffix
`_session-<id>_lifetime-300`); a rolling session id is reused across the
flow and rotates near expiry or after a transport failure (dead IP → fresh
one). lifetime via EMEX_FLOXY_LIFETIME (0 = rotating).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DataImpulse (74.81.81.81, random port 10001-10099) intermittently throws
connect-timeouts / resets, dropping real emex hits. Add a Floxy residential
fallback (residential.floxy.io:12321): fetchEmexHtml now runs a provider
schedule — 3 DataImpulse attempts (rotating port), then 2 Floxy attempts —
and only falls back on transport errors (a definitive HTTP answer like 404
still stops the schedule). On by default; endpoint/creds overridable via
EMEX_FLOXY_* env. Direct proxy-less last resort still gated by
EMEX_DIRECT_FALLBACK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
emex catalogs were seeded from the flat QuickGroups taxonomy (~351
unsearchable leaves on one page). Switch the seeder to the Vehicle.aspx
OEM-microfiche tree (top groups -> sub groups -> Unit.aspx leaves), which
also reaches units that have no QuickGroups gid (e.g. the torque
converter, OEM 5205547 on the Fiesta).
- emex-tree.parser.ts: parseVehicleTree / parseUnitLeaves — pure,
GET-based (per-href ssd, no browser/postback). Unit-tested vs fixtures.
- emex.service: fetchVehicleTree + drillVehicleNode; all 3 decode methods
now populate categoryTree; Unit.aspx leaves fetch parts directly.
- categories.service: revive the dormant tree-insert branch with group
markers (linkWid="emex-group" / hasSubgroups), add an emex getChildren
lazy-drill branch + emex-group parent guard, tighten leaf detection,
disambiguate colliding sub-group names.
- prefetch: emex Vehicle.aspx = parent to drill; Unit/QuickDetails = leaf.
No DDL and no frontend changes — the generic tree UI renders it like
pl24/pcat. Existing flat emex categories re-seed lazily once deleted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PSA (Peugeot/Citroën/DS) VIN decode was systemically broken: the catalog
vin-group page returns all families unfiltered, so decode fell back to the first
family/salesType (a manual base variant) — yielding "{Brand} {VIN}" model names,
empty transmission, wrong model year, and manual-only parts trees (automatic
gearbox parts missing). Reported for a 1999 Peugeot 106 automatic shown as a 2029
manual with no automatic parts.
- New self-contained PL24PsaService: consumes PL24's FI flow (vin.action →
hintstoken → FI page → json-vin-main-groups → json-vin-illustrations →
vin-image-board). Reads model/year/transmission from the FI identification
table; builds the VIN-indexed parts tree (correct per actual VIN). Does not
touch Ford/Volvo/Nissan/Opel/Hyundai-Kia/Fiat.
- Orchestrator + categories.service route PSA VIN decode/drill to the new service.
- Cycle-correct extractModelYear in @sase/shared (X→1999, not 2029): resolve the
30-yr VIN year code to the most-recent plausible year (≤ now+1); dedupe 6 copies.
Validated live against 13 already-decoded PSA VINs: 12/13 full trees with real
model/year/transmission; automatics correctly detected (106 BVA, 206 AL4, 3008 BVA8).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ConfigService.get<number>("EMEX_PROXY_PORT_START") returns the raw env
STRING; the port-pick arithmetic then string-concatenated it
(45 + "10001" = "4510001"), producing an out-of-range port that made
undici's `new URL` throw "Invalid URL" at EmexService construction —
crashing the entire API on boot.
A single-port range (823) happened to concat to a still-parseable "0823",
which masked the bug for months. It surfaced the moment the prod
EMEX_PROXY_PORT range was widened (823 -> 10001-10099) to let Q1's
per-request port rotation work: prod crash-looped until the env was
reverted. Coerce to a validated integer port (1-65535) with default
fallback so a real range is safe.
Regression test: constructing EmexService with string port env over a
real range must not throw.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root-cause analysis live re-decoded all 124 historically-undecoded prod
VINs; 38 already decode now. These 4 fixes target confirmed code bugs
that drop or mask real decodes (see undecoded-vin-rca.md):
Q2 — PL24 circuit breaker now only counts transient transport faults. A
definitive upstream negative (NotFound/BadRequest) no longer trips the
global 30s breaker that was starving PL24 for every subsequent VIN
(the sibling-VIN inconsistency in the report). Live-proven on VR7.
Q3 — previewVin / multi-candidate path no longer returns an empty
success: the pcat/emex candidate branches fill brandName (from catalogId
/ WMI), fixing the 6 "HTTP 200 with null brand+model" cases.
Q1 — EMEX fetch retries transient proxy failures with a FRESH ProxyAgent
per attempt (rotates the DataImpulse port; ~42% blip rate observed),
plus an opt-in direct fallback (EMEX_DIRECT_FALLBACK). HTTP answers are
never retried.
Q4 — VIN resolve cache keys namespaced by DECODE_CHAIN_VERSION and the
negative TTL drops 6h -> 30m, so a decode-chain fix self-heals stale
negatives on deploy instead of masking phantom-undecoded VINs for hours.
The admin cache-buster uses the same key builder.
Tests: 179 passed (+ new Q2/Q3/Q4 specs). typecheck + biome clean.
Deploy note: prod EMEX_PROXY_PORT_START/END are both 823 (single port);
widen to a real range (e.g. 10001-10099) in Coolify so Q1's port
rotation takes full effect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The worker's own upstream fetches called touchActivity(), setting the
prefetch:activity:<source> cooldown key (TTL 300s) that checkCooldown then
honoured — so after each fetch the worker paused itself for up to ~5 minutes
(re-checking every 60s, ~5 empty cycles per key). At ~1 fetch / 5 min the
3387-job backlog needed ~6 days to drain.
- Wrap each worker job in an AsyncLocalStorage backfill context; touchActivity
skips the cooldown key when invoked from the worker, so the cooldown reflects
only real user requests (worker yields to users, never to itself).
- Cooldown TTL 300s -> 90s (a request 5 min ago isn't "active").
- checkCooldown pauses for the key's actual remaining TTL (one wait) instead of
a fixed 60s re-check loop.
No extra upstream load — only removes the worker's self-imposed idle time.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prod decode failures clustered on French/Stellantis VINs that no source even
tried: the post-merger WMIs VR3/VR7 (Peugeot), VR1 (Citroën/DS) and W0V (Opel)
were missing from every map, so getServiceName/isSupported returned null and the
VIN was rejected as "Marka desteklenmiyor" before any scrape ran.
Add them to all three maps, matching the existing VF3/VF7/W0L routing:
- Corgi WMI_DATABASE (display brand)
- PL24_WMI_SERVICE_MAP (peugeot_parts / citroen_parts / opel_parts)
- EMEX CATALOG_MAP (PSA -> PEUGEOT00, Opel -> GM_OP201809)
Lets EMEX scrape and PL24 fall back for these mass-market TR brands instead of
hard-failing at the routing step. Phase 4 of 4 on decode reliability.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PCAT was missing parity with EMEX in two places:
- Part `description` (notice) was stored raw English alongside translated `name`.
Now batched into translateMany so users see Turkish notices.
- Vehicle body/engine/transmission attrs from VIN decode were raw upstream
values. Wire up the existing emex.mapper dictionaries on the PCAT
single-car, PCAT resolveById, and EMEX single-vehicle result paths.
- emex.mapper translateToTurkish now falls back to the original term on
dictionary miss instead of null — upstream values are heterogeneous
(engine codes, multi-word descriptors); losing them was worse than
leaving them untranslated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Probe (scripts/dev) confirmed that everything PartsPanel reads from an
emexdwc.ae leaf — parts table, hotspot coordinates, and the schema image
URL — is fully server-rendered in Unit.aspx HTML. Parts arrive as
`<tr name>` rows with `td[name=c_oem|c_pnc|c_name]`; hotspots are
`<div class="dragger g_highlight" name=N style="margin-top:Ypx; margin-left:Xpx; ...">`
with the coords already in image-natural pixel space; the image URL is
in `<img class="dragger" src=...laximo...>` and its native dims can be
read from the first 24 bytes of the GIF/PNG via a Range GET.
Behaviour:
1. fetchCategoryParts now tries fetchCategoryPartsViaHttp first — two
sequential GETs (QuickDetails → Unit) + a Range GET for image dims.
2. If the HTML yields ≥1 part, we return it.
3. If the HTML returns no Unit.aspx anchor, or 0 parts, we fall back to
the existing Playwright scraper (same code path as before).
The plain-HTTP path skips the ~1-2s browser launch, sidesteps the
3-page semaphore in EmexBrowserService (concurrency cap was throttling
prefetch fan-out), and uses no chromium memory. Measured on dev with 5
fresh-ssd Renault Espace IV leaves: 4.9-5.5s wall per leaf (vs 6-7s on
the Tier 1 browser path, vs 12-14s pre-Tier-1). The 6th sample
(stale-ssd Fren Kaliyeri) failed both paths identically — confirms the
plain-HTTP path doesn't introduce new failure modes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EMEX-decoded vehicles surfaced empty category pages on prod because two
runtime prerequisites were absent from the production image:
1. `scripts/emex-vin-scraper.js` was never copied — the build stage's
`COPY . .` brings it in but the production stage only cherry-picks
`apps/api/dist`, `drizzle`, and `start.sh`. Every EMEX leaf hit
therefore failed with "Scraper file not found at: /app/scripts/…"
and the category page rendered "Bu kategori icin parca bulunamadi."
2. `PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH=/usr/bin/chromium-browser` was
exported in the Dockerfile but never reached `chromium.launch()` —
that env var is consumed by `playwright install`, not at runtime.
Playwright fell back to its bundled headless-shell cache path
(`/root/.cache/ms-playwright/chromium_headless_shell-*/…`) which
does not exist on the alpine image, so even with the scraper file
present the browser pool init would have kept failing.
Fix:
- Dockerfile: `COPY --from=build /app/scripts ./scripts`.
- `emex.browser.ts` + `parts-catalogs-auth.service.ts`: read
`process.env.PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH` and pass it as
`executablePath` to `chromium.launch()` when set.
Verified on prod container: `ls /app/scripts` → missing pre-fix; the
binary at `/usr/bin/chromium-browser` exists, so the env-var hand-off
will resolve cleanly once the new image lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- CATALOG_MAP'e ~40 yeni WMI eklendi (Mitsubishi, Nissan, Volvo, vb.)
- brandFromCatalogCode listesi 55 DB kataloğuna genişletildi
- brand="UNKNOWN" olsa bile catalogCode varsa EMEX sonucu kabul edilir
- emex-catalog: linkPath'li kategorilerde parçalar DB'den yüklenir (scraper bug fix)
- VIN cache: emex_vehicle_vins tablosu ile tekrar VIN'lerde emexdwc.ae atlanır
- getCategoryTree Step 0: VIN cache lookup (~10ms)
- Match sonrası otomatik VIN cache doldurma
- Proaktif cache: ilk decode sonrası matchByName + saveVinCache
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When EMEX DB has no vehicle match (Ford EU, Mercedes etc.), instead of
falling back to full on-demand VIN decode (~8s), use the VIN-decode SSD
directly with QuickGroups.aspx?c={catalog}&vid=0&ssd={vinSsd} (~2-3s).
Tested: Ford Focus CB4 VIN SSD returns 376 categories via QuickGroups.
New step 3.5 in the cascade:
1. pathData → matchByName (DB-only, ~50ms)
2. emexVehicleName → matchByName (DB-only, ~50ms)
3. matchBySsd wizard walk (DB, ~2-3s)
3.5. VIN SSD → QuickGroups shortcut (~2-3s) ← NEW
4. Fallback → full on-demand scrape (~8s)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ConfigService returns empty string (not the default) when env var
exists but is empty. Add || fallbacks in code and set proper defaults
in docker-compose to prevent invalid proxy URL like http://user:pass@:10000.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Node's global fetch doesn't support the dispatcher option. Use
undici's exported fetch when proxy is configured, which properly
handles ProxyAgent as a dispatcher.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
EMEX server blocks direct server IP requests (geo/bot protection),
causing all HTTP-based VIN decodes to return empty results. Add proxy
support to fetchEmexHtml() using undici ProxyAgent, reusing the same
EMEX_USE_PROXY / EMEX_PROXY_* env vars as the browser service.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace Playwright browser with pure HTTP for EMEX VIN decode:
- Vehicles.aspx, QuickGroups.aspx, QuickDetails.aspx all work
without authentication — no session/browser needed
- VIN decode drops from ~10-20s (Playwright) to ~1-5s (HTTP)
- Playwright browser kept as fallback if HTTP fails
Add multi-candidate selection for EMEX (mirrors PartsCatalogs flow):
- EmexCandidate interface matches PcatCar shape so VehicleSelectModal
renders without changes
- decodeVinOrCandidates(): single HTTP fetch returns vehicle OR
candidates (avoids double Vehicles.aspx request)
- decodeVinByIndex(): resolves user-selected candidate by index
- vehicles.service: emexCandidates in VinResolveResult, emexCarIndex
param in decodeVin/resolveVin, resolveEmexCarByIndex() method
- vehicles.controller: @Body("emexCarIndex") accepted
- search.tsx: candidateSource state, handleCandidateSelect sends
emexCarIndex or pcatCarId based on source
Test VIN NM417800006410193 (Fiat SIENA) now shows 2-option modal.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Integrate external parts catalogs API with auth service, add BullMQ-based
catalog prefetch worker for background data caching, expand vehicles service
with shared vehicle support, and add vehicle select modal to frontend.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add PL24 Ford legacy service for fordt_parts architecture
- Refactor EMEX to use persistent browser pool instead of per-call instances
- Make vehicle decode resilient: fallback to PL24 when Corgi doesn't recognize VIN
- Add collapsible sidebar with persistent user preference
- Improve brand access guard and categories service
- Add debug/test scripts for VIN e2e testing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace browser-based Puppeteer scraper with standalone scraper wrapper.
Add EMEX as fallback source for categories when PL24 is unavailable.
Update vehicle decoding to use new synchronous EMEX API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
