fix(pcat): widen JWT capture for slow residential IPs (domcontentloaded + 12s poll)
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
After the Floxy account rotation, the new residential exit IPs are slower:
measured DOM-ready ~7s and the widget's /v3/api/proxy token call firing ~11.8s —
past the 10s PAGE_TIMEOUT + networkidle, so every capture timed out ("No token
after 12s") and the warm pool stayed empty (pcat dead despite valid creds).
- waitUntil networkidle → domcontentloaded (reliable ~7s; networkidle often never
settles on JS catalog sites through a slow proxy → goto times out pre-token).
- PAGE_TIMEOUT 10s → 25s (env PCAT_PAGE_TIMEOUT_MS); token fired at ~11.8s.
- CAPTURE_POLL_AFTER_OK 5s → 12s (env PCAT_CAPTURE_POLL_OK) to cover the ~5s
gap between DOM-ready and the token call. Capture is background (warm pool,
cold-pool fast-fail never blocks users), so the longer cap is free insurance.
Verified via standalone Playwright probe through the new Floxy: token captured at
11842ms (DOM 7023ms). typecheck clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -36,16 +36,19 @@ const REDIS_SAFETY_BUFFER_S = 60; // don't serve from Redis if <60s of life left
|
||||
const TOKEN_TTL = 600; // seconds — TWS- has no built-in expiry, refresh aggressively
|
||||
const REFRESH_BUFFER = 90; // Refresh 90s before expiry
|
||||
const CAPTURE_POLL_INTERVAL = 500; // ms
|
||||
// Two-armed post-goto poll: a healthy goto means the widget JS already loaded
|
||||
// and the API call typically fires within ~1-2s — short cap is enough. A
|
||||
// failed goto means we lost the page, but the widget request may still have
|
||||
// gone through before the navigation timeout — give a brief grace window
|
||||
// and bail. Old single-armed 20s cap was the dominant cost on failed-site
|
||||
// attempts (e.g. 10s page timeout + 20s blind poll = 30s wasted).
|
||||
const CAPTURE_POLL_AFTER_OK = 10; // 10 × 500ms = 5s after a healthy goto
|
||||
const CAPTURE_POLL_AFTER_FAIL = 4; // 4 × 500ms = 2s grace after goto failure
|
||||
const PAGE_TIMEOUT = 10_000; // 10s — healthy partner sites load in <5s through
|
||||
// the proxy; a longer wait only prolongs dead-port connects in capture retries.
|
||||
// Post-goto poll. We now navigate with `domcontentloaded` (reliable, doesn't
|
||||
// depend on networkidle settling on JS-heavy catalog sites), then poll for the
|
||||
// widget's token call. Measured through Floxy residential 2026-06-11: DOM ready
|
||||
// ~7s, the /v3/api/proxy token call fires ~11.8s (≈5s after DOM). The poll must
|
||||
// comfortably cover that gap, so 12s — too short and every capture misses the
|
||||
// token (seen at the old 5s cap: "No token after 12s", whole pool starved).
|
||||
const CAPTURE_POLL_AFTER_OK = Number(process.env.PCAT_CAPTURE_POLL_OK) || 24; // 24 × 500ms = 12s
|
||||
const CAPTURE_POLL_AFTER_FAIL = 8; // 8 × 500ms = 4s grace after goto failure
|
||||
// 25s — residential exit IPs are slow; the token call fired at ~11.8s in the
|
||||
// 2026-06-11 measurement, so 10s timed out before it. Capture is background
|
||||
// (warm pool, never blocks a user with cold-pool fast-fail), so a longer cap is
|
||||
// cheap insurance against residential variance. Env-tunable for future rotation.
|
||||
const PAGE_TIMEOUT = Number(process.env.PCAT_PAGE_TIMEOUT_MS) || 25_000;
|
||||
const CONTEXT_CLOSE_TIMEOUT = 5_000;
|
||||
const SITE_COOLDOWN = 10 * 60 * 1000; // 10 min per site
|
||||
const MAX_POOL_SIZE = 5;
|
||||
@@ -763,12 +766,16 @@ export class PartsCatalogsAuthService implements OnModuleInit, OnModuleDestroy {
|
||||
return route.continue();
|
||||
});
|
||||
|
||||
// Navigate — networkidle waits for widget JS to load + make API calls
|
||||
// Navigate with domcontentloaded (not networkidle): on JS-heavy catalog
|
||||
// sites through a slow residential proxy networkidle often never settles
|
||||
// and the goto times out before the widget fires its token call. DOM-ready
|
||||
// is reliable (~7s measured); the request listener + post-goto poll then
|
||||
// catch the /v3/api/proxy token call (~11.8s measured).
|
||||
let gotoOk = true;
|
||||
try {
|
||||
await page.goto(siteUrl, {
|
||||
timeout: PAGE_TIMEOUT,
|
||||
waitUntil: "networkidle",
|
||||
waitUntil: "domcontentloaded",
|
||||
});
|
||||
} catch (navErr) {
|
||||
// Navigation may timeout but JWT could still be captured if the
|
||||
|
||||
Reference in New Issue
Block a user