Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson

Summary

The talk outlines critical lessons from deploying ML models in production, arguing that model accuracy alone is insufficient for real-world success. Success requires rigorous project scoping, continuous monitoring infrastructure, and ensuring the model's value proposition (Value Ad) significantly outweighs its maintenance cost and potential risks. Key failure points include minimal stakeholder consultation, ignoring data drift, and failing to plan for inevitable changes in the operational environment.

Technical details

• Model Drift Types

  Models can fail due to several types of drift: Concept Drift (change in relationship between input and target variables), Data Drift (change in underlying feature distributions, e.g., seasonal products), Label Drift (change in the target variable distribution, e.g., competitor launch increasing churn), and Reality Drift (rapid, drastic changes in the world).

• ML Pipeline Failure Points 3000s

  A production pipeline can fail at multiple stages: Data Ingestion (e.g., vendor schema change), Feature Generation (potential for feature drift), Model Training (overfitting, model file corruption), and Serving (latency issues, dependency failure). Robust monitoring must be implemented across all these points.

• Statistical Testing & Metrics

  To monitor data integrity, statistical tests like the Chi-square and Kolmogorov-Smirnov test can assess if underlying distributions have changed. Performance metrics vary by task: Mean Absolute Error/Mean Squared Error for value prediction; Precision, Recall, F1 scores for classification; Semantic Similarity for language modeling.

Mentioned resources

• Ben Wilson's book: Machine Learning Engineering and Action (Book)

Channel & topics

• NDC Conferences
• Machine Learning Operations (MLOps)
• Model Deployment
• Data Drift and Concept Drift
• Project Scoping
• Statistical Modeling
• Ben Wilson's book: Machine Learning Engineering and Action