fix(catalog-source): gate emex parts behind allowlist (safety)
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
Some checks failed
QA Gate (P0/P1) / Test affected app (pull_request) Has been cancelled
The catalog-wide bridge in EmexSourceDbService.fetchCategoryParts was
measured against vehicle_parts on 2026-06-01 and found to return 7-114x
more parts than belong to the requesting vehicle, with 49-98 wrong OEM
codes per 100 served. That directly violates the project rule that the
user must never see a wrong OEM.
Per-catalog noiseRatio sample (catalog-wide / per-vehicle):
RENAULT201910 51x | FFIAT84 45x | VOLVO201410 24x | MB201810 14x
AU1587 8x | BMW202501 70x (+ gid namespace mismatch ETK vs numeric)
GM_C201809 114x | MINI202501 12x | LRE201412 7x | MAZDA2020 54x
GM_OP201809 dump has only 1 wildcard vehicle (unique_key="_") so the
single Crossland X "owns" all 47k Opel parts — same firehose served
to any Opel sub-model in sase prod.
All alternative bridges were proven dead:
SSD eşleştirme - session-bound, 0/91 sase SSDs match dump
scrape_queue_v2.vehicle_ssd - same session SSD format
api_cache replay - table empty (0 rows)
wizard_parameters - table empty (0 rows)
VIN direct - no VIN column in dump
The only viable per-vehicle bridge is vehicles.unique_key reconstruction
from raw_data.parsedOptions, but sase currently stores the required 4
wizard fields on just 5/103 emex vehicles (all Renault). That work is
follow-up; this patch only stops the bleeding.
Change:
- Add EMEX_SOURCE_DB_ALLOWED_CATALOGS env (comma-separated, default "")
- EmexSourceDbService.fetchCategoryParts returns null unless catalogCode
is in the allowlist. Empty allowlist = service is effectively off for
parts, full fallthrough to live emex.
- Connection pool stays alive so the follow-up per-vehicle bridge /
schema-only path can use it without flipping env.
- Boot logs warn loudly when connected with an empty allowlist.
Prod was never affected — CATALOG_SOURCE_DB_ENABLED was unset there. This
fixes dev branch behaviour (default-on since commit 3a3a7d3) and keeps
prod safe by default once main is promoted.
Files:
- packages/config/src/index.ts env schema + audit notes
- apps/api/src/config/configuration.ts parse allowlist into string[]
- apps/api/src/integrations/catalog-source-db/emex-source-db.service.ts
allowlist field, init logging, fetchCategoryParts gate, class doc
- docker-compose.coolify.yml env injection for api + worker
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -77,6 +77,13 @@ export default () => ({
|
||||
// doesn't cover sase's TR-market vehicles; see service comment for details.
|
||||
emexEnabled: (process.env.EMEX_SOURCE_DB_ENABLED ?? "true") === "true",
|
||||
pcatEnabled: process.env.PCAT_SOURCE_DB_ENABLED === "true",
|
||||
// Per-catalog allowlist for emex parts lookup. Empty → service returns
|
||||
// null for every catalog (safe default after the 2026-06-01 noise audit).
|
||||
// Populate ONLY after a catalog's per-vehicle bridge is wired & verified.
|
||||
emexAllowedCatalogs: (process.env.EMEX_SOURCE_DB_ALLOWED_CATALOGS ?? "")
|
||||
.split(",")
|
||||
.map((s) => s.trim())
|
||||
.filter(Boolean),
|
||||
},
|
||||
otel: {
|
||||
enabled: process.env.OTEL_ENABLED === "true",
|
||||
|
||||
@@ -11,21 +11,29 @@ import type { EmexHotspot, EmexHotspotArea, EmexPart, EmexPartsResult } from "..
|
||||
* Bridge: catalogs.code ↔ sase's vehicles.rawData.catalogCode,
|
||||
* part_groups.group_id (varchar) ↔ sase's categories.externalId (gid).
|
||||
*
|
||||
* Why catalog-wide (no per-vehicle filter): the dump's only stable per-vehicle
|
||||
* key is the EMEX session-state-descriptor (`ssd`), but SSD is regenerated
|
||||
* every decode session, so the SSD sase stored for its decoded vehicle never
|
||||
* matches the SSD the dump scraper recorded (0 / 10 sampled — verified). The
|
||||
* vehicle_id we'd need to join `vehicle_parts` is unreachable. So we return
|
||||
* all parts in (catalog, group) across every vehicle in the dump for that
|
||||
* catalog — over-returns variants that may not apply to the user's specific
|
||||
* vehicle, but parts overlap heavily and the cost is a slightly noisier list.
|
||||
* Hits: ~71% of sase's 8287 (catalogCode, gid) pairs (verified 2026-06-01).
|
||||
* SAFETY (2026-06-01 audit): the catalog-wide bridge below (no per-vehicle
|
||||
* filter) measured a 7-114x noiseRatio across every sampled catalog and
|
||||
* 49-98 wrong-OEM per 100 parts served, because vehicle_parts is shared
|
||||
* densely across all variants in a (catalog, group) pair (e.g. Renault
|
||||
* Mégane vs Duster vs Clio share the same gids). That directly violates the
|
||||
* "always correct OEM, no exceptions" rule. To prevent leakage, fetchCategoryParts
|
||||
* now returns null unless the catalogCode is in the EMEX_SOURCE_DB_ALLOWED_CATALOGS
|
||||
* allowlist — empty by default. A catalog should ONLY be added once a
|
||||
* per-vehicle bridge (vehicles.unique_key reconstruction from raw_data.parsedOptions)
|
||||
* lands and has been verified OEM-by-OEM vs live. All alternative bridges (SSD,
|
||||
* api_cache, wizard_parameters, scrape_queue_v2.vehicle_ssd) were verified
|
||||
* dead — see memory `sase-emex-source-db-safety.md` for the full inventory.
|
||||
*
|
||||
* The connection pool stays alive (master + emex switches default-on) so
|
||||
* follow-up code can use it for per-vehicle queries / schema-only diagrams
|
||||
* without having to flip env again.
|
||||
*/
|
||||
@Injectable()
|
||||
export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
|
||||
private readonly logger = new Logger(EmexSourceDbService.name);
|
||||
private pool: Pool | null = null;
|
||||
private enabled = false;
|
||||
private allowedCatalogs: Set<string> = new Set();
|
||||
|
||||
constructor(private readonly config: ConfigService) {}
|
||||
|
||||
@@ -33,6 +41,8 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
|
||||
const masterEnabled = this.config.get<boolean>("catalogSource.enabled");
|
||||
const emexEnabled = this.config.get<boolean>("catalogSource.emexEnabled");
|
||||
const url = this.config.get<string>("catalogSource.emexUrl");
|
||||
const allowed = this.config.get<string[]>("catalogSource.emexAllowedCatalogs") ?? [];
|
||||
this.allowedCatalogs = new Set(allowed);
|
||||
if (!masterEnabled || !emexEnabled || !url) {
|
||||
this.logger.log(
|
||||
`[emex-src] disabled (master=${masterEnabled}, emex=${emexEnabled}, urlSet=${Boolean(
|
||||
@@ -48,7 +58,15 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
|
||||
waitForConnections: true,
|
||||
});
|
||||
this.enabled = true;
|
||||
this.logger.log("[emex-src] connected, lookup-first enabled");
|
||||
if (this.allowedCatalogs.size === 0) {
|
||||
this.logger.warn(
|
||||
"[emex-src] connected but ALLOWLIST EMPTY — all fetchCategoryParts calls return null until EMEX_SOURCE_DB_ALLOWED_CATALOGS is populated (safety: catalog-wide bridge has 7-114x noise; see sase-emex-source-db-safety memory)",
|
||||
);
|
||||
} else {
|
||||
this.logger.log(
|
||||
`[emex-src] connected, allowlist: [${[...this.allowedCatalogs].sort().join(",")}]`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async onModuleDestroy() {
|
||||
@@ -70,6 +88,10 @@ export class EmexSourceDbService implements OnModuleInit, OnModuleDestroy {
|
||||
): Promise<EmexPartsResult | null> {
|
||||
if (!this.enabled || !this.pool) return null;
|
||||
if (!catalogCode || !gid) return null;
|
||||
// Safety gate: the only path currently implemented (catalog-wide DISTINCT
|
||||
// on vehicle_parts.group_id, ignoring vehicle_id) is unsafe — see class
|
||||
// doc. Until a per-vehicle bridge lands, allow no catalogs.
|
||||
if (!this.allowedCatalogs.has(catalogCode)) return null;
|
||||
|
||||
try {
|
||||
// Resolve (catalog.id, part_group.id) in one round trip.
|
||||
|
||||
@@ -82,6 +82,11 @@ services:
|
||||
# Per-source kill switches. pcat default off (dump doesn't cover TR vehicles).
|
||||
- EMEX_SOURCE_DB_ENABLED=${EMEX_SOURCE_DB_ENABLED:-true}
|
||||
- PCAT_SOURCE_DB_ENABLED=${PCAT_SOURCE_DB_ENABLED:-false}
|
||||
# Per-catalog parts-lookup allowlist (comma-separated). Empty → all
|
||||
# fetchCategoryParts calls return null. See sase-emex-source-db-safety
|
||||
# memory: catalog-wide bridge had 7-114x noise; only add a catalog once
|
||||
# its per-vehicle bridge is wired & OEM-verified.
|
||||
- EMEX_SOURCE_DB_ALLOWED_CATALOGS=${EMEX_SOURCE_DB_ALLOWED_CATALOGS:-}
|
||||
depends_on:
|
||||
sase-redis:
|
||||
condition: service_healthy
|
||||
@@ -151,6 +156,11 @@ services:
|
||||
# Per-source kill switches. pcat default off (dump doesn't cover TR vehicles).
|
||||
- EMEX_SOURCE_DB_ENABLED=${EMEX_SOURCE_DB_ENABLED:-true}
|
||||
- PCAT_SOURCE_DB_ENABLED=${PCAT_SOURCE_DB_ENABLED:-false}
|
||||
# Per-catalog parts-lookup allowlist (comma-separated). Empty → all
|
||||
# fetchCategoryParts calls return null. See sase-emex-source-db-safety
|
||||
# memory: catalog-wide bridge had 7-114x noise; only add a catalog once
|
||||
# its per-vehicle bridge is wired & OEM-verified.
|
||||
- EMEX_SOURCE_DB_ALLOWED_CATALOGS=${EMEX_SOURCE_DB_ALLOWED_CATALOGS:-}
|
||||
depends_on:
|
||||
sase-redis:
|
||||
condition: service_healthy
|
||||
|
||||
@@ -121,12 +121,19 @@ export const envSchema = z.object({
|
||||
PCAT_SOURCE_DB_URL: z.string().url().optional(),
|
||||
EMEX_SOURCE_DB_URL: z.string().optional(), // mysql://... — not a strict URL per WHATWG
|
||||
// Per-source kill switches under the master CATALOG_SOURCE_DB_ENABLED.
|
||||
// EMEX defaults true (catalog-allowlist tightens which catalogs hit the dump).
|
||||
// PCAT defaults FALSE — verified 2026-06-01 that the dump's deep-scrape covers
|
||||
// a US/JDM market subset (Toyota/Nissan/Audi/Chevy/Hyundai) that doesn't
|
||||
// intersect sase's TR-market vehicle pool (0 / 103 dev carIds had real parts
|
||||
// data through either bridge). Container stays running for future use cases
|
||||
// (OEM cross-ref, alt-part search).
|
||||
// EMEX_SOURCE_DB_ENABLED keeps the connection pool alive but, per the
|
||||
// 2026-06-01 safety audit, fetchCategoryParts ALWAYS returns null unless the
|
||||
// requested catalogCode is also in EMEX_SOURCE_DB_ALLOWED_CATALOGS. The
|
||||
// catalog-wide bridge measured 7-114x noiseRatio across every catalog and
|
||||
// 49-98 wrong-OEM per 100 served — direct violation of the "always correct
|
||||
// OEM" rule. Default allowlist is EMPTY → behaviour is safe by default; the
|
||||
// master/emex switches stay default-on so the service is ready for the
|
||||
// per-vehicle unique_key bridge (follow-up work).
|
||||
// PCAT_SOURCE_DB_ENABLED defaults FALSE — verified 2026-06-01 that the dump's
|
||||
// deep-scrape covers a US/JDM market subset (Toyota/Nissan/Audi/Chevy/Hyundai)
|
||||
// that doesn't intersect sase's TR-market vehicle pool (0 / 103 dev carIds
|
||||
// had real parts data through either bridge). Container stays running for
|
||||
// future use cases (OEM cross-ref, alt-part search).
|
||||
EMEX_SOURCE_DB_ENABLED: z
|
||||
.string()
|
||||
.transform((v) => v === "true")
|
||||
@@ -135,6 +142,13 @@ export const envSchema = z.object({
|
||||
.string()
|
||||
.transform((v) => v === "true")
|
||||
.default("false"),
|
||||
// Per-catalog parts-lookup allowlist. Comma-separated catalog codes
|
||||
// (e.g. "RENAULT201910,FFIAT84"). Empty (default) → fetchCategoryParts
|
||||
// always returns null → live emex handles every request. A catalog SHOULD
|
||||
// only be added here AFTER its per-vehicle bridge (unique_key) is wired and
|
||||
// verified against live OEM-by-OEM on at least 5 sampled vehicles. See
|
||||
// memory `sase-emex-source-db-safety.md` for the bridge inventory & audit.
|
||||
EMEX_SOURCE_DB_ALLOWED_CATALOGS: z.string().default(""),
|
||||
});
|
||||
|
||||
export type Env = z.infer<typeof envSchema>;
|
||||
|
||||
Reference in New Issue
Block a user