feat(insights): permanent archive of PostHog events + raw recordings
Phase A of the external observability archive. PostHog Cloud free tier
deletes data on a rolling window (events ~1yr, session recordings ~30d);
this mirrors both into our own infra permanently (hot Postgres + cold
MinIO gzip JSONL).
PostHog raw event archive:
- New `PosthogEvent` model (uuid PK → dedup, full properties JSONB,
hot columns promoted + indexed).
- `posthog-event-archive` job: watermark-paged HogQL pull (cursor in the
existing IngestionWatermark via a "sase:events" stream key), createMany
+ skipDuplicates for idempotency, 365d backfill. Daily closed-day cold
dump to MinIO `posthog-archive/{project}/YYYY/MM/DD.jsonl.gz`.
- `hogqlQuery` helper added to posthog.ts. Timestamp cursor uses
parseDateTimeBestEffort() — ClickHouse 500s on a raw ISO8601 literal
(verified live against the project).
Raw rrweb recording preservation (separate from compress, which only
covers scored sessions and caps blobs):
- `SessionMeta.rrwebArchivedAt` / `rrwebArchiveKey`.
- `archive-recordings` job: every recording within 25d (margin before
30d deletion), full blob range (no 12-cap), gzip → MinIO
`rrweb-archive/{project}/YYYY/MM/DD/{sessionId}.jsonl.gz`.
- `putBuffer` gzip helper in minio.ts.
- Both jobs wired into pipeline.ts (event-archive@*/15min, recordings@*/6h).
- retention.ts untouched → new table + buckets persist forever (the goal).
Schema applies via the existing `prisma db push` on web start (additive:
new table + nullable columns).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -142,6 +142,10 @@ model SessionMeta {
|
||||
processedAt DateTime?
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
// Raw rrweb recording preservation (archived to MinIO before PostHog's ~30-day deletion).
|
||||
rrwebArchivedAt DateTime?
|
||||
rrwebArchiveKey String?
|
||||
|
||||
compressed CompressedSession?
|
||||
|
||||
@@index([status, createdAt])
|
||||
@@ -445,3 +449,27 @@ model SaseUserNote {
|
||||
@@index([saseUserId, pinned, createdAt])
|
||||
@@map("sase_user_notes")
|
||||
}
|
||||
|
||||
// ---------- Phase 9: External Observability Archive ----------
|
||||
|
||||
// PostHog raw event stream mirrored to our DB permanently (free tier deletes
|
||||
// events on a rolling window). Lossless: full properties kept as JSONB, with
|
||||
// hot fields promoted to indexed columns. Also cold-dumped to MinIO as gzip JSONL.
|
||||
model PosthogEvent {
|
||||
uuid String @id // PostHog event uuid → natural dedup key
|
||||
projectKey String // "sase"
|
||||
event String
|
||||
distinctId String
|
||||
personId String?
|
||||
sessionId String? // $session_id — joins to SessionMeta.id
|
||||
timestamp DateTime // PostHog event time
|
||||
properties Json // full properties — lossless
|
||||
ingestedAt DateTime @default(now())
|
||||
dumpedAt DateTime? // set once written to MinIO cold partition
|
||||
|
||||
@@index([projectKey, timestamp])
|
||||
@@index([event, timestamp])
|
||||
@@index([distinctId, timestamp])
|
||||
@@index([dumpedAt])
|
||||
@@map("posthog_events")
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user