feat(FN-399): cross-session funnel attribution audit + regression check #18
@@ -733,6 +733,58 @@ After filling in Section 4, evaluate each row against its threshold:
|
||||
|
||||
---
|
||||
|
||||
## Appendix D: Cross-Session Funnel Attribution Config Audit (FN-399)
|
||||
|
||||
**Date:** 2026-05-15
|
||||
**Auditor:** Fusion agent (FN-399)
|
||||
**Status:** ✅ PASS — config + identify/reset + all four funnel events present
|
||||
**Why this matters:** the cross-session funnel `vin_decode_success → checkout_started → payment_initiated → payment_success` spans multiple sessions per user (VIN decode often happens pre-login; checkout and payment happen post-login on a later session). PostHog can only stitch those events into one funnel per user when `person_profiles: "identified_only"` is set and `identify()` / `reset()` are wired to auth. Without this, all P1 CRO attribution at the user level is unreliable.
|
||||
|
||||
### D.1 Findings (audit)
|
||||
|
||||
| Check | File | Result |
|
||||
|-------|------|--------|
|
||||
| `person_profiles: "identified_only"` on init | `apps/web/src/lib/posthog.ts` | ✅ Set |
|
||||
| `identifyUser(user)` → `ph.identify(user.id, …)` | `apps/web/src/lib/posthog.ts` | ✅ Implemented |
|
||||
| `resetUser()` → `ph.reset()` (logout safety) | `apps/web/src/lib/posthog.ts` | ✅ Implemented |
|
||||
| `identifyUser(…)` invoked from auth bootstrap | `apps/web/src/routes/__root.tsx` | ✅ Wired |
|
||||
| `resetUser()` invoked from auth bootstrap (sign-out) | `apps/web/src/routes/__root.tsx` | ✅ Wired |
|
||||
| `vin_decode_success` captured | `apps/web/src/routes/dashboard/search.tsx` | ✅ Frontend `capture(…)` |
|
||||
| `checkout_started` captured | `apps/web/src/routes/dashboard/subscription/index.tsx` | ✅ Frontend `capture(…)` |
|
||||
| `payment_initiated` captured | `apps/web/src/components/payment/stripe-checkout-button.tsx` | ✅ Frontend `capture(…)` |
|
||||
| `payment_success` captured | `apps/api/src/payments/stripe/stripe.service.ts` | ✅ Server `captureForUser(userId, …)` |
|
||||
|
||||
**Cross-session join sanity check:** `payment_success` is emitted server-side from the Stripe webhook handler via `captureForUser(userId, …)`, which passes the user id as PostHog `distinctId`. That is the same `distinctId` the frontend `identifyUser()` call established on login. Therefore the server-side `payment_success` event joins the same person profile as the earlier client-side `vin_decode_success` (which was anonymous, then back-filled by `identify()` thanks to `person_profiles: "identified_only"`). The four-step funnel can be reconstructed per user across sessions.
|
||||
|
||||
### D.2 Regression Check Script
|
||||
|
||||
A regression check lives at `scripts/check-posthog-identified-only.mjs` and exits non-zero if any of the conditions in D.1 drift:
|
||||
|
||||
```
|
||||
node scripts/check-posthog-identified-only.mjs
|
||||
```
|
||||
|
||||
The script verifies:
|
||||
1. `person_profiles: "identified_only"` is the literal string in `apps/web/src/lib/posthog.ts`.
|
||||
2. `identifyUser` exists and calls `ph.identify(user.id, …)`.
|
||||
3. `resetUser` exists and calls `ph.reset()`.
|
||||
4. `apps/web/src/routes/__root.tsx` actually invokes both, so the config isn't dead code.
|
||||
5. All four funnel events (`vin_decode_success`, `checkout_started`, `payment_initiated`, `payment_success`) are still captured under `apps/web/src` or `apps/api/src` via `capture(…)` or `captureForUser(userId, …)`.
|
||||
|
||||
It walks both source trees in pure Node (no shell quoting), skips `node_modules` / `__tests__` / `dist`, and excludes `*.test.ts`/`*.spec.ts` so a mock string in a fixture can't false-pass. Suggested wiring: run it from CI on PRs that touch `apps/web/src/lib/posthog.ts`, `apps/web/src/routes/__root.tsx`, the checkout/subscription/payment paths, or the Stripe webhook handler.
|
||||
|
||||
### D.3 Drift Symptoms (what failure looks like in PostHog)
|
||||
|
||||
If this regresses without the script catching it, the symptoms in PostHog are subtle and easy to misread as a CRO problem rather than an attribution problem:
|
||||
|
||||
- `vin_decode_success` events still flow, but they belong to anonymous `distinct_id`s that never reconcile to a user — funnels built on persons drop these out at step 1.
|
||||
- `payment_success` (server-side, with a real user id) appears in person funnels, but the user has no preceding `vin_decode_success` / `checkout_started` against the same profile, so the funnel shows step-1 → step-4 conversion as `0%`.
|
||||
- Per-user revenue attribution under-counts because anonymous and identified events split into two profiles per real user.
|
||||
|
||||
If you see step-1 conversion drop to ~0% in person-mode funnels while event-mode funnels still look healthy, suspect this config first, then run the script.
|
||||
|
||||
---
|
||||
|
||||
**Document status:** ⏳ AWAITING MANUAL DATA — See Section 8 for step-by-step data collection instructions.
|
||||
**Data collected by:** FN-336 (Fusion executor) — attempted 2026-05-13, PostHog API key unavailable, dashboard login required.
|
||||
**Next step:** Human operator: follow Section 8 guide, fill Section 9 form, transcribe into Sections 3/4, run stop-loss evaluation.
|
||||
|
||||
220
scripts/check-posthog-identified-only.mjs
Normal file
220
scripts/check-posthog-identified-only.mjs
Normal file
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env node
|
||||
// FN-399 — Cross-session funnel attribution regression check.
|
||||
//
|
||||
// Why this exists: the P0/P1 CRO funnel
|
||||
// `vin_decode_success → checkout_started → payment_initiated → payment_success`
|
||||
// spans multiple sessions per user (VIN decode often happens anonymously on the
|
||||
// landing page; checkout and payment happen later, post-login). PostHog can
|
||||
// only join those events into one funnel per user when:
|
||||
//
|
||||
// 1. `person_profiles: "identified_only"` is set on init, so anonymous
|
||||
// pre-login events still emit but get reconciled to the user once
|
||||
// `posthog.identify()` is called.
|
||||
// 2. `identifyUser()` exists in `lib/posthog.ts` and calls
|
||||
// `ph.identify(user.id, ...)`.
|
||||
// 3. `resetUser()` exists and calls `ph.reset()` so logout doesn't leak the
|
||||
// prior user's distinct_id onto the next visitor on the same browser.
|
||||
// 4. The auth bootstrap (`routes/__root.tsx`) actually invokes both, so the
|
||||
// config above isn't dead code.
|
||||
// 5. All four cross-session funnel events still emit with their canonical
|
||||
// `lowercase_snake` names — frontend (`apps/web/src`) via `capture("evt", …)`
|
||||
// or backend (`apps/api/src`) via `captureForUser(userId, "evt", …)` /
|
||||
// `capture("evt", …)`. Server-side `captureForUser` passes the user id
|
||||
// as `distinctId`, so the event joins the same person profile that the
|
||||
// frontend `identify()` created — the cross-session join still holds.
|
||||
//
|
||||
// If any of those drift, the funnel silently breaks: events still flow but
|
||||
// PostHog can no longer stitch sessions to a user, and every CRO attribution
|
||||
// number at the user level becomes unreliable. This script fails when that
|
||||
// drift is introduced so the regression is caught at PR / CI time.
|
||||
//
|
||||
// Run: `node scripts/check-posthog-identified-only.mjs`
|
||||
// Exit: 0 on PASS, 1 on FAIL.
|
||||
|
||||
import { readFileSync, readdirSync, statSync } from "node:fs";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const repoRoot = resolve(__dirname, "..");
|
||||
|
||||
const POSTHOG_LIB = "apps/web/src/lib/posthog.ts";
|
||||
const AUTH_BOOTSTRAP = "apps/web/src/routes/__root.tsx";
|
||||
const EVENT_SOURCES = ["apps/web/src", "apps/api/src"];
|
||||
|
||||
const REQUIRED_EVENTS = [
|
||||
"vin_decode_success",
|
||||
"checkout_started",
|
||||
"payment_initiated",
|
||||
"payment_success",
|
||||
];
|
||||
|
||||
const failures = [];
|
||||
const notes = [];
|
||||
|
||||
function read(relPath) {
|
||||
try {
|
||||
return readFileSync(resolve(repoRoot, relPath), "utf8");
|
||||
} catch (err) {
|
||||
failures.push(`Cannot read ${relPath}: ${err.message}`);
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
// 1. person_profiles must be "identified_only" in the posthog-js init call.
|
||||
const lib = read(POSTHOG_LIB);
|
||||
if (lib) {
|
||||
const personProfilesMatch = lib.match(/person_profiles\s*:\s*["']([^"']+)["']/);
|
||||
if (!personProfilesMatch) {
|
||||
failures.push(
|
||||
`${POSTHOG_LIB}: \`person_profiles\` option is missing from the posthog-js init. ` +
|
||||
"Without this, PostHog defaults to creating a person profile for every anonymous " +
|
||||
"visitor, which breaks user-level cross-session funnel joins.",
|
||||
);
|
||||
} else if (personProfilesMatch[1] !== "identified_only") {
|
||||
failures.push(
|
||||
`${POSTHOG_LIB}: \`person_profiles\` is "${personProfilesMatch[1]}", expected ` +
|
||||
'"identified_only". Drift here means anonymous pre-login VIN decode events ' +
|
||||
"won't reconcile to the user that later signs in and checks out.",
|
||||
);
|
||||
} else {
|
||||
notes.push(`${POSTHOG_LIB}: person_profiles = "identified_only" ok`);
|
||||
}
|
||||
|
||||
// 2. identifyUser must exist and call ph.identify(user.id, ...).
|
||||
if (!/export function identifyUser\b/.test(lib)) {
|
||||
failures.push(`${POSTHOG_LIB}: \`identifyUser\` export is missing.`);
|
||||
} else if (!/ph\.identify\(\s*user\.id/.test(lib)) {
|
||||
failures.push(
|
||||
`${POSTHOG_LIB}: \`identifyUser\` no longer calls \`ph.identify(user.id, ...)\`. ` +
|
||||
"Without identify() the anonymous events never stitch to a user profile.",
|
||||
);
|
||||
} else {
|
||||
notes.push(`${POSTHOG_LIB}: identifyUser -> ph.identify(user.id, …) ok`);
|
||||
}
|
||||
|
||||
// 3. resetUser must exist and call ph.reset() — required on logout so the
|
||||
// next anonymous visitor on the same browser doesn't inherit the prior
|
||||
// user's distinct_id.
|
||||
if (!/export function resetUser\b/.test(lib) || !/ph\.reset\(\)/.test(lib)) {
|
||||
failures.push(
|
||||
`${POSTHOG_LIB}: \`resetUser\` / \`ph.reset()\` is missing. Logout must reset ` +
|
||||
"PostHog identity, otherwise the next anonymous visitor on the same browser " +
|
||||
"inherits the prior user's distinct_id.",
|
||||
);
|
||||
} else {
|
||||
notes.push(`${POSTHOG_LIB}: resetUser -> ph.reset() ok`);
|
||||
}
|
||||
}
|
||||
|
||||
// 4. The auth bootstrap must wire identifyUser to the authenticated user and
|
||||
// resetUser to the signed-out branch.
|
||||
const root = read(AUTH_BOOTSTRAP);
|
||||
if (root) {
|
||||
if (!/identifyUser\s*\(/.test(root)) {
|
||||
failures.push(
|
||||
`${AUTH_BOOTSTRAP}: \`identifyUser(...)\` is not invoked. Config is correct ` +
|
||||
"but never fires, so PostHog stays anonymous for every user.",
|
||||
);
|
||||
} else {
|
||||
notes.push(`${AUTH_BOOTSTRAP}: identifyUser(...) invoked ok`);
|
||||
}
|
||||
if (!/resetUser\s*\(/.test(root)) {
|
||||
failures.push(
|
||||
`${AUTH_BOOTSTRAP}: \`resetUser()\` is not invoked on sign-out. Logout will ` +
|
||||
"leak the previous user's identity to the next visitor on the same browser.",
|
||||
);
|
||||
} else {
|
||||
notes.push(`${AUTH_BOOTSTRAP}: resetUser() invoked ok`);
|
||||
}
|
||||
}
|
||||
|
||||
// 5. All four funnel events must still be captured — frontend
|
||||
// (`apps/web/src`) via `capture("evt", …)` from `lib/posthog.ts`, or
|
||||
// backend (`apps/api/src`) via `captureForUser(userId, "evt", …)` /
|
||||
// `capture("evt", …)` from `posthog/posthog.service.ts`. We walk both
|
||||
// trees in node (no shell-quoting issues) and only count matches in
|
||||
// production .ts/.tsx — tests, node_modules, and build artifacts are
|
||||
// excluded so we don't false-pass on a mock string.
|
||||
function* walk(dir) {
|
||||
let entries;
|
||||
try {
|
||||
entries = readdirSync(dir);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
for (const name of entries) {
|
||||
if (name === "node_modules" || name === "__tests__" || name === "dist") continue;
|
||||
const full = join(dir, name);
|
||||
let st;
|
||||
try {
|
||||
st = statSync(full);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
if (st.isDirectory()) {
|
||||
yield* walk(full);
|
||||
} else if (
|
||||
st.isFile() &&
|
||||
/\.(ts|tsx)$/.test(name) &&
|
||||
!/\.(test|spec)\.(ts|tsx)$/.test(name)
|
||||
) {
|
||||
yield full;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const sourceFiles = [];
|
||||
for (const root of EVENT_SOURCES) {
|
||||
for (const f of walk(resolve(repoRoot, root))) sourceFiles.push(f);
|
||||
}
|
||||
|
||||
function findEmit(eventName) {
|
||||
// Frontend: `capture("evt", …)` from lib/posthog.ts.
|
||||
// Backend: `captureForUser(userId, "evt", …)` or `capture("evt", …)` from
|
||||
// PostHogService. `captureForUser` passes userId as distinctId, so the
|
||||
// event joins the same person profile the frontend `identify()` created
|
||||
// — the cross-session join still holds.
|
||||
const fe = new RegExp(`\\bcapture\\(\\s*["']${eventName}["']`);
|
||||
const beUser = new RegExp(`\\bcaptureForUser\\(\\s*[^,]+,\\s*["']${eventName}["']`);
|
||||
for (const f of sourceFiles) {
|
||||
let src;
|
||||
try {
|
||||
src = readFileSync(f, "utf8");
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
const rel = f.slice(repoRoot.length + 1);
|
||||
if (beUser.test(src)) return { kind: "captureForUser (server)", path: rel };
|
||||
if (fe.test(src)) {
|
||||
const kind = rel.startsWith("apps/api/") ? "capture (server)" : "capture (client)";
|
||||
return { kind, path: rel };
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
for (const event of REQUIRED_EVENTS) {
|
||||
const hit = findEmit(event);
|
||||
if (hit) {
|
||||
notes.push(`event: ${event} → ${hit.kind} @ ${hit.path} ok`);
|
||||
} else {
|
||||
failures.push(
|
||||
`Funnel event \`${event}\` is not captured anywhere under ${EVENT_SOURCES.join(" or ")}. ` +
|
||||
"Cross-session funnel `vin_decode_success → checkout_started → payment_initiated " +
|
||||
"→ payment_success` cannot be reconstructed without all four steps emitting.",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Report ──
|
||||
const headline =
|
||||
failures.length === 0
|
||||
? "PASS — PostHog identified_only config + cross-session funnel events intact."
|
||||
: `FAIL — ${failures.length} issue(s) detected.`;
|
||||
|
||||
console.log(`[check-posthog-identified-only] ${headline}`);
|
||||
for (const n of notes) console.log(` ok ${n}`);
|
||||
for (const f of failures) console.log(` FAIL ${f}`);
|
||||
|
||||
process.exit(failures.length === 0 ? 0 : 1);
|
||||
Reference in New Issue
Block a user