perf(backfill): stop the prefetch worker from throttling itself (Tier 1)
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled

The worker's own upstream fetches called touchActivity(), setting the
prefetch:activity:<source> cooldown key (TTL 300s) that checkCooldown then
honoured — so after each fetch the worker paused itself for up to ~5 minutes
(re-checking every 60s, ~5 empty cycles per key). At ~1 fetch / 5 min the
3387-job backlog needed ~6 days to drain.

- Wrap each worker job in an AsyncLocalStorage backfill context; touchActivity
  skips the cooldown key when invoked from the worker, so the cooldown reflects
  only real user requests (worker yields to users, never to itself).
- Cooldown TTL 300s -> 90s (a request 5 min ago isn't "active").
- checkCooldown pauses for the key's actual remaining TTL (one wait) instead of
  a fixed 60s re-check loop.

No extra upstream load — only removes the worker's self-imposed idle time.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-03 18:23:54 +03:00
parent d694bc8c35
commit 1e3953417e
6 changed files with 50 additions and 10 deletions

View File

@@ -14,6 +14,7 @@ import {
ServiceUnavailableException,
} from "@nestjs/common";
import { ConfigService } from "@nestjs/config";
import { isBackfillContext } from "../../jobs/prefetch-context";
import { RedisService } from "../../redis/redis.service";
import { StorageService } from "../../storage/storage.service";
import { PL24AuthService } from "./pl24-auth.service";
@@ -827,8 +828,11 @@ export class PL24Service {
/** Mark PL24 as actively used (5min TTL) to defer prefetch worker */
private async touchActivity(): Promise<void> {
// Worker-originated fetches must NOT register as user activity, or the
// backfill worker throttles itself via checkCooldown.
if (isBackfillContext()) return;
try {
await this.redis.set("prefetch:activity:pl24", String(Date.now()), 300);
await this.redis.set("prefetch:activity:pl24", String(Date.now()), 90);
} catch {
// Non-critical — don't break the request
}