Live verification surfaced two issues in archive-recordings:
- blob_v2 rejects wide blob-key ranges (a 23-key span 400s; ~5 is fine).
Fetch in contiguous chunks of BLOB_CHUNK (default 10) and concatenate.
- PostHog's snapshot API is aggressively rate-limited; back-to-back
sessions tripped 429. Pace requests (per-session + per-chunk delays)
and stop the run on sustained 429 — unarchived rows keep
rrwebArchivedAt=null and retry next cycle (oldest-first ordering
protects soon-to-be-deleted recordings first). Result now carries
rateLimited.
Verified against prod: 87- and 23-blob recordings reconstruct fully
(no 400), failed=0, graceful 429 stop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase A of the external observability archive. PostHog Cloud free tier
deletes data on a rolling window (events ~1yr, session recordings ~30d);
this mirrors both into our own infra permanently (hot Postgres + cold
MinIO gzip JSONL).
PostHog raw event archive:
- New `PosthogEvent` model (uuid PK → dedup, full properties JSONB,
hot columns promoted + indexed).
- `posthog-event-archive` job: watermark-paged HogQL pull (cursor in the
existing IngestionWatermark via a "sase:events" stream key), createMany
+ skipDuplicates for idempotency, 365d backfill. Daily closed-day cold
dump to MinIO `posthog-archive/{project}/YYYY/MM/DD.jsonl.gz`.
- `hogqlQuery` helper added to posthog.ts. Timestamp cursor uses
parseDateTimeBestEffort() — ClickHouse 500s on a raw ISO8601 literal
(verified live against the project).
Raw rrweb recording preservation (separate from compress, which only
covers scored sessions and caps blobs):
- `SessionMeta.rrwebArchivedAt` / `rrwebArchiveKey`.
- `archive-recordings` job: every recording within 25d (margin before
30d deletion), full blob range (no 12-cap), gzip → MinIO
`rrweb-archive/{project}/YYYY/MM/DD/{sessionId}.jsonl.gz`.
- `putBuffer` gzip helper in minio.ts.
- Both jobs wired into pipeline.ts (event-archive@*/15min, recordings@*/6h).
- retention.ts untouched → new table + buckets persist forever (the goal).
Schema applies via the existing `prisma db push` on web start (additive:
new table + nullable columns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
