Why Great Models Fail: Lessons From 9 Years of Deploying ML Models - Megan Robertson
The talk outlines critical lessons from deploying ML models in production, arguing that model accuracy alone is insufficient for real-world success. Success requires rigorous project scoping, continuous monitoring infrastructure, and ensuring the model's value proposition (Value Ad) significantly outweighs its maintenance cost and potential risks. Key failure points include minimal stakeholder consultation, ignoring data drift, and failing to plan for inevitable changes in the operational environment.
Key takeaways
-
Stakeholder Value is Paramount
13:20
The model's value must be quantifiable (KPI) and its contribution must outweigh the cost of maintenance. Stakeholders must guide the project scope, preventing engineers from building technically cool but commercially irrelevant solutions.
-
Scope Definition is a Multi-Step Process
23:20
Proper scoping requires defining who is served (stakeholders/end users), clearly articulating the problem, understanding constraints and risks, identifying possible solutions (MVP approach: crawl, walk, run), and planning maintenance.
-
ML Models Require Continuous Monitoring
58:20
Since models are trained on a single point in time, they must be monitored for performance degradation. Strategies include tracking data issues (e.g., distribution changes), feature drift, and model-specific metrics (e.g., Mean Absolute Error, F1 scores).