feat(insights): permanent archive of PostHog events + raw recordings

Phase A of the external observability archive. PostHog Cloud free tier
deletes data on a rolling window (events ~1yr, session recordings ~30d);
this mirrors both into our own infra permanently (hot Postgres + cold
MinIO gzip JSONL).

PostHog raw event archive:
- New `PosthogEvent` model (uuid PK → dedup, full properties JSONB,
  hot columns promoted + indexed).
- `posthog-event-archive` job: watermark-paged HogQL pull (cursor in the
  existing IngestionWatermark via a "sase:events" stream key), createMany
  + skipDuplicates for idempotency, 365d backfill. Daily closed-day cold
  dump to MinIO `posthog-archive/{project}/YYYY/MM/DD.jsonl.gz`.
- `hogqlQuery` helper added to posthog.ts. Timestamp cursor uses
  parseDateTimeBestEffort() — ClickHouse 500s on a raw ISO8601 literal
  (verified live against the project).

Raw rrweb recording preservation (separate from compress, which only
covers scored sessions and caps blobs):
- `SessionMeta.rrwebArchivedAt` / `rrwebArchiveKey`.
- `archive-recordings` job: every recording within 25d (margin before
  30d deletion), full blob range (no 12-cap), gzip → MinIO
  `rrweb-archive/{project}/YYYY/MM/DD/{sessionId}.jsonl.gz`.

- `putBuffer` gzip helper in minio.ts.
- Both jobs wired into pipeline.ts (event-archive@*/15min, recordings@*/6h).
- retention.ts untouched → new table + buckets persist forever (the goal).

Schema applies via the existing `prisma db push` on web start (additive:
new table + nullable columns).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Semih
2026-05-27 20:15:29 +03:00
parent fb33692452
commit f663c0aa09
6 changed files with 380 additions and 1 deletions

View File

@@ -142,6 +142,10 @@ model SessionMeta {
processedAt DateTime?
createdAt DateTime @default(now())
// Raw rrweb recording preservation (archived to MinIO before PostHog's ~30-day deletion).
rrwebArchivedAt DateTime?
rrwebArchiveKey String?
compressed CompressedSession?
@@index([status, createdAt])
@@ -445,3 +449,27 @@ model SaseUserNote {
@@index([saseUserId, pinned, createdAt])
@@map("sase_user_notes")
}
// ---------- Phase 9: External Observability Archive ----------
// PostHog raw event stream mirrored to our DB permanently (free tier deletes
// events on a rolling window). Lossless: full properties kept as JSONB, with
// hot fields promoted to indexed columns. Also cold-dumped to MinIO as gzip JSONL.
model PosthogEvent {
uuid String @id // PostHog event uuid → natural dedup key
projectKey String // "sase"
event String
distinctId String
personId String?
sessionId String? // $session_id — joins to SessionMeta.id
timestamp DateTime // PostHog event time
properties Json // full properties — lossless
ingestedAt DateTime @default(now())
dumpedAt DateTime? // set once written to MinIO cold partition
@@index([projectKey, timestamp])
@@index([event, timestamp])
@@index([distinctId, timestamp])
@@index([dumpedAt])
@@map("posthog_events")
}