fix(parts): dedupe ingest path + clean ~108k duplicate rows
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled

Backfill + reactive prefetch can re-drill the same category multiple
times, and the parts insert path had no dedupe guard. Result: 8.2%
duplicate rows on pl24, 14.5% on parts-catalogs, and 33.7% on emex —
~108k extra rows across 7,571 categories on 224 vehicles. Every drilled
catalog page rendered each part twice (the Tampon example: 32 rows for
19 distinct OEMs).

* Migration 0010 — phase 1 deletes existing dupes preserving the
  oldest row per (vehicle_id, category_id, oem_code, name, position)
  group; phase 2 adds a UNIQUE INDEX over the same tuple with NULLS NOT
  DISTINCT (PG 15+) so null position/vehicle_id collapse like equal
  values rather than each counting as its own "distinct" row. Idempotent
  CREATE UNIQUE INDEX IF NOT EXISTS so the runner is safe to re-apply.
* All five insert(parts).values(...).returning() call sites
  (parts.service, categories.service ×3, catalog.service) get
  .onConflictDoNothing() so future re-drills no-op instead of erroring
  on the new constraint. `.returning()` continues to surface only the
  newly-inserted rows; existing logs read `Stored N parts` as actual
  net insertions, which is what we want.

Dry-run on dev DB: 524,540 → 425,186 parts (99,354 dupes deleted), index
created cleanly. Same delta expected on prod (~108k drop).

drizzle-orm 0.41 doesn't expose .nullsNotDistinct() on the index builder
so the constraint is owned by raw SQL — see the inline comment in the
parts schema and the migration file. Future schema generators should NOT
try to drop or rewrite this index.
This commit is contained in:
2026-06-02 02:08:57 +03:00
parent d792bf0efb
commit 10dcf2d6ae
6 changed files with 5603 additions and 2 deletions

View File

@@ -0,0 +1,21 @@
-- Phase 1: remove existing duplicate rows.
-- "Duplicate" = same (vehicle_id, category_id, oem_code, name, position).
-- Backfill + reactive prefetch can re-drill the same category multiple times;
-- without a unique constraint that re-insert kept all rows, leaving 834%
-- duplicate per source (~108k extra rows of ~588k). Keep the oldest row per
-- group (preserves original created_at) and drop the rest.
DELETE FROM "parts" WHERE id IN (
SELECT id FROM (
SELECT id, row_number() OVER (
PARTITION BY vehicle_id, category_id, oem_code, name, position
ORDER BY created_at ASC, id ASC
) AS rn FROM "parts"
) t WHERE rn > 1
);
--> statement-breakpoint
-- Phase 2: prevent future dupes. NULLS NOT DISTINCT (Postgres 15+) so NULL
-- position / NULL vehicle_id collapse like equal values rather than each
-- counting as a separate "distinct" row.
CREATE UNIQUE INDEX IF NOT EXISTS "parts_dedup_idx" ON "parts"
(vehicle_id, category_id, oem_code, name, position) NULLS NOT DISTINCT;

File diff suppressed because it is too large Load Diff

View File

@@ -71,6 +71,13 @@
"when": 1779703007129,
"tag": "0009_referral_rewards_rework",
"breakpoints": true
},
{
"idx": 10,
"version": "7",
"when": 1780281600000,
"tag": "0010_dedupe_parts",
"breakpoints": true
}
]
}