The Silent Problem -- Machine Learning Model Failure -- How to Diagnose and Fix Ailing Machine Learning Models

TL;DR

This paper discusses the challenges of diagnosing and fixing machine learning model failures caused by concept and data drift, emphasizing the need for resilience and proactive robustness in model development.

Contribution

It highlights the importance of detecting and diagnosing drift in ML models and advocates for designing models with resilience and robustness to handle inevitable data changes.

Findings

Drift causes model degradation in real-world scenarios.

Detecting drift is crucial for maintaining model reliability.

Developing resilient models can mitigate failure impacts.

Abstract

The COVID-19 pandemic has dramatically changed how healthcare is delivered to patients, how patients interact with healthcare providers, and how healthcare information is disseminated to both healthcare providers and patients. Analytical models that were trained and tested pre-pandemic may no longer be performing up to expectations, providing unreliable and irrelevant learning (ML) models given that ML depends on the basic principle that what happened in the past are likely to repeat in the future. ML faced to two important degradation principles, concept drift, when the underlying properties and characteristics of the variables change and data drift, when the data distributions, probabilities, co-variates, and other variable relationships change, both of which are prime culprits of model failure. Therefore, detecting and diagnosing drift in existing models is something that has become an imperative. And perhaps even more important is a shift in our mindset towards a conscious recognition that drift is inevitable, and model building must incorporate intentional resilience, the ability to offset and recover quickly from failure, and proactive robustness, avoiding failure by developing models that are less vulnerable to drift and disruption.