Recall — upstream pre-mortem

Remediation

• Repoint the feature device_risk_score in fraud_features (urn:li:mlFeature:(fraud_features,device_risk_score)) off urn:li:dataset:(urn:li:dataPlatform:snowflake,retail.ml.device_signals,PROD) and onto urn:li:dataset:(urn:li:dataPlatform:snowflake,retail.ml.device_signals_v2,PROD), mapping the source column from device_risk to risk_score. The v1 deprecation note ("Replaced by retail.ml.device_signals_v2 during the vendor migration") makes v2 the intended successor, and Carol Whitfield (urn:li:corpuser:carol) is technical owner of both tables plus fraud_detector_v3 — so this is a single-owner change, no cross-team handoff needed.

• Before that repoint lands, get a decision from Carol on the score scale. v1 device_risk is documented as "vendor-supplied risk score in [0,1]"; v2 risk_score is documented as "rescaled vendor risk score" and v2 adds a model_version column that v1 did not have. If the scales differ, swapping the source silently shifts the input distribution under a live gradient-boosted classifier with two deployments. The decision Carol owns: rescale at the feature-materialization step to reproduce the v1 [0,1] range, or accept the v2 scale and retrain/recalibrate fraud_detector_v3. Confirm the actual value ranges in both tables rather than trusting the field descriptions.

• Find and stop whatever is still writing retail.ml.device_signals. It is soft-deleted and out of the catalog, so any job still populating it is running outside governance and is invisible to lineage and to this detector on the next scan. Trace the writer from the Snowflake job/warehouse history for that table — DataHub will not have it, since the entity was removed.

• Treat device_signals_v2 as not-yet-clean before cutover: it currently reports 4 active incidents in DataHub. Have Carol confirm those incidents are unrelated to risk_score freshness or correctness, otherwise the fix moves fraud_detector_v3 from a stale source onto a degraded one.

Remediation

I'll check what's actually known about the two device_signals tables and the feature before writing this.

Version skew is the crux here — let me confirm the column-level difference.

• Repoint device_risk_score in the Feast fraud_features table off device_signals_v2.risk_score and back onto retail.ml.device_signals.device_risk — the exact column the training run consumed — or freeze the two live deployments until step 3 lands. Treat this as a stopgap measured in days, not a fix: device_signals is already marked deprecated ("Replaced by retail.ml.device_signals_v2 during the vendor migration"), carries a decommission timestamp within roughly a day of today, and is currently failing 1 of its 2 assertions (urn:li:assertion:recall-7322dbce62882e8b). Confirm that exact decommission date before committing to the rollback path.

• Quantify the damage, because this is not a column rename. Training fitted on device_signals.device_risk, documented as "Vendor-supplied risk score in [0,1]"; serving reads device_signals_v2.risk_score, documented as "Rescaled vendor risk score." The two are on different scales from different vendor scoring models. Join the tables on device_id over an overlap window and compare distributions, and read the distinct values of device_signals_v2.model_version to see how many vendor scoring versions have been served since cutover. That result decides whether every authorisation scored since the cutover needs re-review.

• Retrain and re-register fraud_detector_v3 against device_signals_v2.risk_score so the fitted and served inputs match, and record retail.ml.device_signals_v2 on the new training run's lineage. Rolling forward is the only durable option once device_signals is decommissioned. Note that retail.ml.fraud_labels appearing as a training-only input is expected for a label source and needs no serving counterpart — it is not part of this defect.

• Carol Whitfield (urn:li:corpuser:carol, ML Platform Engineer) is the sole listed owner of the model and both device tables, so she owns the rollback-versus-freeze call in step 1 and the retrain in step 3. The re-review decision from step 2 — whether fraud decisions made on rescaled scores get reopened — is a fraud-operations call, and DataHub shows no owner for that; identify the fraud-ops decision owner rather than defaulting it to Carol. The vendor's rescaling definition is also not recorded anywhere in the catalog and will need to come from whoever ran the vendor migration.

Remediation

• Freeze the serving path for txn_amount_zscore (urn:li:mlFeature:(fraud_features,txn_amount_zscore)) — it is the only live feature downstream of the retyped column, and its 30-day rolling z-score per customer will shift as soon as unrounded FLOAT values enter the window. Carol Whitfield (urn:li:corpuser:carol, ML Platform Engineer, model technical owner) should have the feature transform cast amount back to 2-decimal fixed precision before z-scoring, so both live deployments keep scoring on the distribution v3 was fit against. This is the hotfix, not the fix.

• Establish which shape the current model actually saw: check the training run urn:li:dataProcessInstance:fraud_detector_v3_run_2026_07_14 against the retype timestamp (1785026121694). If the run predates the retype, v3 is serving on a type it never trained on and retraining is mandatory; if it postdates it, the drift is serving-side only and step 1 is sufficient until the next scheduled retrain. Carol owns this call and it gates everything below.

• Get Alice Nakamura (urn:li:corpuser:alice, Staff Data Engineer, technical owner of retail.raw.transactions) to confirm whether the NUMBER(10,2) → FLOAT retype was intentional and whether historical rows were backfilled or only new rows are affected. The table currently has 3 active incidents and a failing assertion (urn:li:assertion:recall-6774d68d4aa365b5) — establish whether those are the same event before treating them as separate work. A backfill means past training data was rewritten and the v3 training set is no longer reproducible.

• Decide explicitly to exclude device_fingerprint (VARCHAR(128), nullable) from the feature set for now, rather than leaving it undecided. It is a new column with no feature or lineage attached to it, and pulling a nullable high-cardinality string into a 40ms p99 inline scorer is a v4 modelling change with its own encoding and null-handling design — Carol should route it to the next model iteration, not this remediation. Confirm with Alice whether the field carries device-identity data that needs a PII classification before anyone builds on it.

Remediation

• Assign a named technical owner to retail.core.dim_customers today — it is the only table in retail.core with no owner (retail.core.fct_orders is owned by Bob Osei, Analytics Engineer). Bob is the obvious default: he owns the sibling fact table and churn_predictor. But dim_customers also feeds fraud_detector_v3 (Carol Whitfield, ML Platform Engineer; risk tier CRITICAL, inline authorisation scoring at a 40ms p99 budget), so Bob and Carol need to decide together whether this is Bob's individual ownership or a group owner spanning both teams. The table also carries the CustomerPII glossary term, so whoever takes it inherits a PII surface — name a data steward alongside the technical owner rather than leaving that implicit.

• Gate schema changes on the two columns that actually carry the blast radius: customer_id and tenure_days. Both feed customer_tenure_days, which exists twice downstream — once in churn_features and once in fraud_features — and customer_id is the primary key of both feature sets. Require review from both the churn and fraud owners on any PR touching those columns; a single rename breaks two feature sets and three deployments at once.

• Attach real freshness and null-rate assertions to dim_customers so a break pages the new owner. The only two assertions on this table right now are the Recall detector findings themselves, and both have failed on all 8 runs — there is currently no data-quality signal that fires independently of the structural finding.

• Have Carol confirm what the 3 live deployments actually are. fraud_detector_v2 still sits downstream of dim_customers in lineage but is marked deprecated with the note "Superseded by fraud_detector_v3 — retained for audit replay only, no live traffic." If v2 is being counted, the real live surface is smaller than reported; if it isn't, its stale lineage should still be retired so future blast-radius scoring stays accurate.

Remediation

• Assign a technical owner to urn:li:dataset:(urn:li:dataPlatform:snowflake,retail.core.dim_customers,PROD) in DataHub, with Alice Nakamura (urn:li:corpuser:alice) as the proposed name — she is the technical owner of the only upstream, retail.raw.customers_raw, so the pipeline that produces this table already sits in her scope. This is her decision to accept or decline; if she declines, the escalation is to whoever owns the retail.core schema, which is not recorded in DataHub and needs to be established from the dbt/warehouse repo rather than guessed.

• Until that ownership lands, name Carol Whitfield (urn:li:corpuser:carol) as the interim escalation contact for breakage. She owns fraud_detector_v3, which scores every authorisation inline against a 40ms p99 budget and carries the CRITICAL risk tier — she absorbs the incident either way, so she should be the one paged rather than discovering it from fraud metrics.

• Have the accepted owner declare a change-notification path to both Carol (fraud_detector_v3) and Bob Osei (urn:li:corpuser:bob, churn_predictor) covering the tenure_days and customer_id columns specifically. Those two fields are not incidental: tenure_days materialises as customer_tenure_days in both churn_features and fraud_features, and customer_id is the primary key of both feature sets, so a silent type or nullability change there hits two production models at once.

• Separately, decide who holds the business/steward ownership for the CustomerPII glossary term applied to this table — technical ownership does not cover PII accountability, and there is no data-governance owner recorded on the dataset. Confirm with your governance lead whether a steward is required on PII-tagged datasets before closing the finding.