feat(FN-275): deliver PostHog funnel audit — P1 CRO sprint instrumentation gap report

Commits merged: - docs(FN-275): deliver PostHog funnel audit — P1 CRO sprint instrumentation gap report Files changed: docs/product/funnel-audit-p1-cro-2026-05.md | 231 ++++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) Fusion-Task-Id: FN-275
2026-05-13 02:02:45 +00:00
parent 6ae26e7a62
commit d985e771d1
1 changed files with 231 additions and 0 deletions
--- a/docs/product/funnel-audit-p1-cro-2026-05.md
+++ b/docs/product/funnel-audit-p1-cro-2026-05.md
@@ -0,0 +1,231 @@
+# PostHog Funnel Audit — P1 CRO Sprint (FN-207/208/209/210)
+
+**Date:** 2026-05-13  
+**Auditor:** Fusion agent (FN-275)  
+**Status:** ⚠️ BLOCKED — Missing instrumentation prevents full analysis
+
+---
+
+## Executive Summary
+
+A code audit of the P1 CRO sprint reveals that **none of the four fixes shipped with PostHog instrumentation**. The events specified in the funnel audit brief (`cancel_flow_viewed`, `cancel_save_clicked`, `trial_banner_viewed`, `trial_banner_dismissed`, `trial_banner_converted`, social proof impression/engagement) do not exist in the codebase. The requested funnel analysis cannot be performed on the fix-specific events.
+
+The baseline funnel — `plan_selected → checkout_started → payment_initiated` — **is instrumented** and could be queried in PostHog for a pre/post comparison. However, `payment_success` is also absent (gap documented in FN-101), so the full 4-step funnel is unavailable.
+
+---
+
+## 1. Event Inventory Audit
+
+### 1.1 Required vs Available Events
+
+| Fix | Required Event | Status | Notes |
+|-----|---------------|--------|-------|
+| FN-207 | `cancel_flow_viewed` | ❌ Missing | Not instrumented anywhere |
+| FN-207 | `cancel_save_clicked` | ❌ Missing | Not instrumented anywhere |
+| FN-208 | (none specified) | N/A | Sticky bar CTA reuses `checkout_started` |
+| FN-209 | `trial_banner_viewed` | ❌ Missing | Banner renders but sends no event |
+| FN-209 | `trial_banner_dismissed` | ❌ Missing | Dismiss tracked only via localStorage |
+| FN-209 | `trial_banner_converted` | ❌ Missing | CTA link navigates, no conversion event |
+| FN-210 | `social_proof_impression` | ❌ Missing | Testimonial is static HTML, no visibility tracking |
+| FN-210 | `social_proof_engagement` | ❌ Missing | No interaction tracking on social proof elements |
+| Baseline | `payment_success` | ❌ Missing | Gap documented in FN-101; never instrumented |
+
+### 1.2 Existing Funnel Events (Available)
+
+All existing events on the subscription/checkout page:
+
+| Event | Properties | Location |
+|-------|-----------|----------|
+| `plan_selected` | `plan` (key) | `subscription/index.tsx:316` |
+| `checkout_started` | `plan`, `period` | `subscription/index.tsx:334` |
+| `payment_initiated` | `method`, `plan`, `period`, `amount` | `payment-content.tsx:176,182` |
+| `trial_started` | (none) | `subscription/index.tsx:798` |
+| `subscription_cancelled` | (none) | `subscription/index.tsx:761` |
+| `subscription_resumed` | (none) | `subscription/index.tsx:612` |
+| `downgrade_offer_shown` | `from_plan`, `to_plan` | `subscription/index.tsx:354` |
+| `downgrade_offer_accepted` | `from_plan`, `to_plan` | `subscription/index.tsx:258` |
+| `downgrade_offer_declined` | `from_plan` | `subscription/index.tsx:702` |
+| `receipt_uploaded` | `payment_id` | `payment-content.tsx:189` |
+
+### 1.3 What the "Cancel-Save Flow" Actually Is
+
+FN-207's cancel retention mechanism is the **downgrade offer** flow — not a separate save flow. When a user clicks cancel:
+- If a lower plan exists → `downgrade_offer_shown` fires, dialog offers a downgrade
+- If user accepts → `downgrade_offer_accepted` fires, subscription is downgraded
+- If user declines → `downgrade_offer_declined` fires, then cancel confirmation opens
+- On confirm cancel → `subscription_cancelled` fires
+
+The `cancel_flow_viewed` / `cancel_save_clicked` event pair was never implemented. The existing downgrade events serve as partial proxies.
+
+### 1.4 Trial Urgency Banner — No Instrumentation
+
+The `TrialUrgencyBanner` component (`components/trial-urgency-banner.tsx`):
+- Renders when trial has ≤3 days remaining
+- Uses `localStorage` for dismiss state (keyed by endDate)
+- CTA links to `/dashboard/subscription` with a `<Link>` component
+- **Sends zero PostHog events** — no impression, no dismiss tracking, no CTA click event
+
+---
+
+## 2. Funnel Analysis (Baseline Events Only)
+
+### 2.1 Query Structure for PostHog
+
+The following PostHog funnel query structure is provided for manual execution in the PostHog dashboard. API credentials are not available from the dev worktree.
+
+```
+Funnel Name: P1 CRO — Checkout Funnel
+Steps:
+  1. checkout_started
+  2. payment_initiated
+
+Date Range: Compare two periods
+  Period A (Pre):  2026-05-06 to 2026-05-12 (7 days pre-FN-207 deploy)
+  Period B (Post): 2026-05-13 to 2026-05-19 (7 days post-FN-210 deploy)
+
+Breakdown: plan property (plan tier)
+```
+
+### 2.2 Data Collection
+
+> **⚠️ This section requires manual PostHog access.** From the PostHog dashboard:
+>
+> 1. Navigate to **Product analytics → Funnels**
+> 2. Create a new funnel: **checkout_started → payment_initiated**
+> 3. Set date range to 2026-05-06 through 2026-05-19
+> 4. Compare 7 days pre vs 7 days post
+> 5. Export the funnel table and fill in the tables below
+
+### 2.3 Funnel Conversion Table (to be filled)
+
+| Period | checkout_started | payment_initiated | Conversion Rate | Δ vs Pre |
+|--------|-----------------|-------------------|----------------|----------|
+| Pre (May 6-12) | TBD | TBD | TBD% | — |
+| Post (May 13-19) | TBD | TBD | TBD% | TBDpp |
+
+### 2.4 Plan Tier Breakdown (to be filled)
+
+| Plan | Pre Rate | Post Rate | Δ |
+|------|----------|-----------|---|
+| brand1 | TBD% | TBD% | TBDpp |
+| brand2 | TBD% | TBD% | TBDpp |
+| brand3 | TBD% | TBD% | TBDpp |
+| full | TBD% | TBD% | TBDpp |
+
+### 2.5 Stop-Loss Threshold Check
+
+| Metric | Pre | Post | Δ | Threshold | Status |
+|--------|-----|------|---|-----------|--------|
+| payment_initiated / checkout_started | TBD% | TBD% | TBDpp | ≤ -5pp | TBD |
+
+**Stop-loss rule:** If the ratio drops more than 5 percentage points from pre-deploy baseline, flag as P1 and recommend immediate revert investigation.
+
+---
+
+## 3. Verdict Per Fix
+
+### FN-207: Cancel-Save Flow (Retention)
+
+**Verdict: ⚠️ Cannot evaluate — no instrumentation**
+
+The downgrade offer flow exists in the UI and fires `downgrade_offer_shown/accepted/declined`. These events can be monitored for retention behavior (how many users downgrade vs cancel outright), but the requested `cancel_flow_viewed` / `cancel_save_clicked` funnel doesn't exist.
+
+**Partial proxy metric:** `downgrade_offer_shown → downgrade_offer_accepted` ratio measures save effectiveness.
+
+### FN-208: Sticky Checkout Bar (UX)
+
+**Verdict: ⚠️ Cannot isolate — shared event path**
+
+The sticky bar reuses the same CTA handler and fires the same `checkout_started` event as the main CTA. There's no way to distinguish sticky-bar-initiated checkouts from main-CTA checkouts in the current event taxonomy. Any improvement from the sticky bar would appear as an aggregate lift in `checkout_started` volume, but cannot be attributed specifically.
+
+**Proxy metric:** Total `checkout_started` volume pre vs post — if volume increased with no other changes, the sticky bar may be contributing.
+
+### FN-209: Trial Urgency Banner (Conversion Urgency)
+
+**Verdict: ❌ Cannot evaluate — no instrumentation**
+
+The banner has **zero PostHog events**. Dismiss state is tracked only in localStorage (no `trial_banner_dismissed` event). The CTA is a plain `<Link>` that navigates to `/dashboard/subscription` without firing any click event. There is no way to:
+- Measure banner impression rate
+- Track dismiss vs engagement behavior
+- Correlate banner exposure with conversion
+- Measure the conversion of banner CTA clicks
+
+### FN-210: Social Proof Strip (Trust)
+
+**Verdict: ❌ Cannot evaluate — no instrumentation**
+
+The testimonial on the auth page (`_auth.tsx`) and any social proof elements on the subscription page are static HTML with no impression tracking, no scroll-depth tracking, and no engagement events.
+
+---
+
+## 4. Critical Observations
+
+### 4.1 Instrumentation Was Not Part of the CRO Fix Scope
+
+All four P1 CRO tasks (FN-207, FN-208, FN-209, FN-210) were implemented as UI-only changes. None of their PROMPT.md specifications included PostHog event instrumentation steps. This is a process gap: **CRO fixes must ship with measurement capabilities** to validate their effectiveness.
+
+### 4.2 The `payment_success` Gap
+
+FN-101 identified the `payment_success` gap but it was never resolved. The funnel currently has a visible drop-off after `payment_initiated` (when users redirect to Iyzico or upload an EFT receipt) with no completion event. This means:
+- The full `checkout_started → payment_success` conversion rate cannot be calculated
+- Revenue attribution per plan tier is not possible in PostHog
+- ABI testing for payment-related CRO changes lacks an endpoint metric
+
+### 4.3 Events Ship in a Disconnected Architecture
+
+The `vin_decode_success` event fires on the landing page and dashboard search page, but there is no user-level connection between a VIN decode session and a subsequent subscription checkout. The funnel step `vin_decode_success → checkout_started` spans two separate user sessions (typically) and requires user-level event correlation in PostHog — which may or may not be set up depending on `person_profiles: "identified_only"` configuration.
+
+---
+
+## 5. Recommended Next Actions
+
+### Immediate (this sprint)
+
+| Priority | Action | Owner |
+|----------|--------|-------|
+| **P0** | Add `payment_success` event to Iyzico callback and EFT approval flows | Backend Eng |
+| **P0** | Add `trial_banner_viewed`, `trial_banner_dismissed`, `trial_banner_converted` events to `TrialUrgencyBanner` | Frontend Eng |
+| **P1** | Add `cancel_flow_viewed` and `cancel_save_clicked` events to the cancel/downgrade dialog flow | Frontend Eng |
+| **P1** | Add social proof impression tracking (IntersectionObserver-based) to testimonial components | Frontend Eng |
+| **P2** | Run the baseline PostHog funnel comparison (checkout_started → payment_initiated) manually in the dashboard | Product/Data |
+
+### Process improvement
+
+- **CRO tasks must include instrumentation steps in their specifications.** The task template for CRO work should require at minimum: (a) an interaction event, (b) a conversion event, and (c) a success/failure event.
+- **Pre-merge review must verify analytics events are implemented.** The QA gate bypass documented for FN-199 is not isolated — FN-207/208/209/210 also shipped without review.
+
+### Follow-up Tasks Created
+
+See the Fusion board for:
+- Instrumentation tasks for the missing events
+- A follow-up audit after instrumentation is in place (allow 7 days of data collection post-instrumentation)
+
+---
+
+## Appendix A: Event Locations in Codebase
+
+| Event | File | Line |
+|-------|------|------|
+| `plan_selected` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 316 |
+| `checkout_started` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 334 |
+| `payment_initiated` (iyzico) | `apps/web/src/components/payment/payment-content.tsx` | 176 |
+| `payment_initiated` (eft) | `apps/web/src/components/payment/payment-content.tsx` | 182 |
+| `trial_started` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 798 |
+| `subscription_cancelled` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 761 |
+| `subscription_resumed` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 612 |
+| `downgrade_offer_shown` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 354 |
+| `downgrade_offer_accepted` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 258 |
+| `downgrade_offer_declined` | `apps/web/src/routes/dashboard/subscription/index.tsx` | 702 |
+| `receipt_uploaded` | `apps/web/src/components/payment/payment-content.tsx` | 189 |
+| Trial banner (no events) | `apps/web/src/components/trial-urgency-banner.tsx` | 23 |
+| Social proof (no events) | `apps/web/src/routes/_auth.tsx` | 103 |
+
+## Appendix B: Files Requiring Instrumentation
+
+| File | Events to Add |
+|------|--------------|
+| `apps/web/src/components/trial-urgency-banner.tsx` | `trial_banner_viewed`, `trial_banner_dismissed`, `trial_banner_converted` |
+| `apps/web/src/routes/dashboard/subscription/index.tsx` | `cancel_flow_viewed`, `cancel_save_clicked` (cancel dialog section) |
+| `apps/web/src/routes/_auth.tsx` | `social_proof_impression` (IntersectionObserver on testimonial) |
+| `apps/api/src/payments/payments.controller.ts` | `payment_success`, `payment_failed` (Iyzico callback + EFT approval) |
+| `apps/web/src/components/payment/payment-content.tsx` | `payment_success` (on Iyzico redirect return or EFT confirmation) |