Phase A of the external observability archive. PostHog Cloud free tier
deletes data on a rolling window (events ~1yr, session recordings ~30d);
this mirrors both into our own infra permanently (hot Postgres + cold
MinIO gzip JSONL).
PostHog raw event archive:
- New `PosthogEvent` model (uuid PK → dedup, full properties JSONB,
hot columns promoted + indexed).
- `posthog-event-archive` job: watermark-paged HogQL pull (cursor in the
existing IngestionWatermark via a "sase:events" stream key), createMany
+ skipDuplicates for idempotency, 365d backfill. Daily closed-day cold
dump to MinIO `posthog-archive/{project}/YYYY/MM/DD.jsonl.gz`.
- `hogqlQuery` helper added to posthog.ts. Timestamp cursor uses
parseDateTimeBestEffort() — ClickHouse 500s on a raw ISO8601 literal
(verified live against the project).
Raw rrweb recording preservation (separate from compress, which only
covers scored sessions and caps blobs):
- `SessionMeta.rrwebArchivedAt` / `rrwebArchiveKey`.
- `archive-recordings` job: every recording within 25d (margin before
30d deletion), full blob range (no 12-cap), gzip → MinIO
`rrweb-archive/{project}/YYYY/MM/DD/{sessionId}.jsonl.gz`.
- `putBuffer` gzip helper in minio.ts.
- Both jobs wired into pipeline.ts (event-archive@*/15min, recordings@*/6h).
- retention.ts untouched → new table + buckets persist forever (the goal).
Schema applies via the existing `prisma db push` on web start (additive:
new table + nullable columns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
f663c0aa09