feat(FN-399): cross-session funnel attribution audit + regression check

Commits merged:
- feat(FN-399): cross-session funnel attribution audit + regression check

Files changed:
docs/product/funnel-audit-p0-cro-2026-05.md |  52 +++++++
 scripts/check-posthog-identified-only.mjs   | 220 ++++++++++++++++++++++++++++
 2 files changed, 272 insertions(+)

Fusion-Task-Id: FN-399
This commit is contained in:
Fusion
2026-05-15 22:03:31 +00:00
committed by Semih Yesilyurt
parent 8685196ab8
commit 584cda2b6c
2 changed files with 272 additions and 0 deletions

View File

@@ -733,6 +733,58 @@ After filling in Section 4, evaluate each row against its threshold:
---
## Appendix D: Cross-Session Funnel Attribution Config Audit (FN-399)
**Date:** 2026-05-15
**Auditor:** Fusion agent (FN-399)
**Status:** ✅ PASS — config + identify/reset + all four funnel events present
**Why this matters:** the cross-session funnel `vin_decode_success → checkout_started → payment_initiated → payment_success` spans multiple sessions per user (VIN decode often happens pre-login; checkout and payment happen post-login on a later session). PostHog can only stitch those events into one funnel per user when `person_profiles: "identified_only"` is set and `identify()` / `reset()` are wired to auth. Without this, all P1 CRO attribution at the user level is unreliable.
### D.1 Findings (audit)
| Check | File | Result |
|-------|------|--------|
| `person_profiles: "identified_only"` on init | `apps/web/src/lib/posthog.ts` | ✅ Set |
| `identifyUser(user)``ph.identify(user.id, …)` | `apps/web/src/lib/posthog.ts` | ✅ Implemented |
| `resetUser()``ph.reset()` (logout safety) | `apps/web/src/lib/posthog.ts` | ✅ Implemented |
| `identifyUser(…)` invoked from auth bootstrap | `apps/web/src/routes/__root.tsx` | ✅ Wired |
| `resetUser()` invoked from auth bootstrap (sign-out) | `apps/web/src/routes/__root.tsx` | ✅ Wired |
| `vin_decode_success` captured | `apps/web/src/routes/dashboard/search.tsx` | ✅ Frontend `capture(…)` |
| `checkout_started` captured | `apps/web/src/routes/dashboard/subscription/index.tsx` | ✅ Frontend `capture(…)` |
| `payment_initiated` captured | `apps/web/src/components/payment/stripe-checkout-button.tsx` | ✅ Frontend `capture(…)` |
| `payment_success` captured | `apps/api/src/payments/stripe/stripe.service.ts` | ✅ Server `captureForUser(userId, …)` |
**Cross-session join sanity check:** `payment_success` is emitted server-side from the Stripe webhook handler via `captureForUser(userId, …)`, which passes the user id as PostHog `distinctId`. That is the same `distinctId` the frontend `identifyUser()` call established on login. Therefore the server-side `payment_success` event joins the same person profile as the earlier client-side `vin_decode_success` (which was anonymous, then back-filled by `identify()` thanks to `person_profiles: "identified_only"`). The four-step funnel can be reconstructed per user across sessions.
### D.2 Regression Check Script
A regression check lives at `scripts/check-posthog-identified-only.mjs` and exits non-zero if any of the conditions in D.1 drift:
```
node scripts/check-posthog-identified-only.mjs
```
The script verifies:
1. `person_profiles: "identified_only"` is the literal string in `apps/web/src/lib/posthog.ts`.
2. `identifyUser` exists and calls `ph.identify(user.id, …)`.
3. `resetUser` exists and calls `ph.reset()`.
4. `apps/web/src/routes/__root.tsx` actually invokes both, so the config isn't dead code.
5. All four funnel events (`vin_decode_success`, `checkout_started`, `payment_initiated`, `payment_success`) are still captured under `apps/web/src` or `apps/api/src` via `capture(…)` or `captureForUser(userId, …)`.
It walks both source trees in pure Node (no shell quoting), skips `node_modules` / `__tests__` / `dist`, and excludes `*.test.ts`/`*.spec.ts` so a mock string in a fixture can't false-pass. Suggested wiring: run it from CI on PRs that touch `apps/web/src/lib/posthog.ts`, `apps/web/src/routes/__root.tsx`, the checkout/subscription/payment paths, or the Stripe webhook handler.
### D.3 Drift Symptoms (what failure looks like in PostHog)
If this regresses without the script catching it, the symptoms in PostHog are subtle and easy to misread as a CRO problem rather than an attribution problem:
- `vin_decode_success` events still flow, but they belong to anonymous `distinct_id`s that never reconcile to a user — funnels built on persons drop these out at step 1.
- `payment_success` (server-side, with a real user id) appears in person funnels, but the user has no preceding `vin_decode_success` / `checkout_started` against the same profile, so the funnel shows step-1 → step-4 conversion as `0%`.
- Per-user revenue attribution under-counts because anonymous and identified events split into two profiles per real user.
If you see step-1 conversion drop to ~0% in person-mode funnels while event-mode funnels still look healthy, suspect this config first, then run the script.
---
**Document status:** ⏳ AWAITING MANUAL DATA — See Section 8 for step-by-step data collection instructions.
**Data collected by:** FN-336 (Fusion executor) — attempted 2026-05-13, PostHog API key unavailable, dashboard login required.
**Next step:** Human operator: follow Section 8 guide, fill Section 9 form, transcribe into Sections 3/4, run stop-loss evaluation.