Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens

TL;DR

This paper explores how to systematically design monitoring strategies for deployed machine learning algorithms by applying causal inference, highlighting the importance of choosing appropriate metrics and data sources to account for performativity effects.

Contribution

It introduces a causal framework for selecting monitoring criteria and data sources, demonstrating their impact through simulation in a real-world risk prediction case study.

Findings

Not all monitoring systems perform equally in detecting issues.

Causal reasoning improves the design of effective monitoring strategies.

Different data sources and criteria significantly affect detection speed and interpretability.

Abstract

After a machine learning (ML)-based system is deployed, monitoring its performance is important to ensure the safety and effectiveness of the algorithm over time. When an ML algorithm interacts with its environment, the algorithm can affect the data-generating mechanism and be a major source of bias when evaluating its standalone performance, an issue known as performativity. Although prior work has shown how to validate models in the presence of performativity using causal inference techniques, there has been little work on how to monitor models in the presence of performativity. Unlike the setting of model validation, there is much less agreement on which performance metrics to monitor. Different monitoring criteria impact how interpretable the resulting test statistic is, what assumptions are needed for identifiability, and the speed of detection. When this choice is further coupled with the decision to use observational versus interventional data, ML deployment teams are faced with a multitude of monitoring options. The aim of this work is to highlight the relatively under-appreciated complexity of designing a monitoring strategy and how causal reasoning can provide a systematic framework for choosing between these options. As a motivating example, we consider an ML-based risk prediction algorithm for predicting unplanned readmissions. Bringing together tools from causal inference and statistical process control, we consider six monitoring procedures (three candidate monitoring criteria and two data sources) and investigate their operating characteristics in simulation studies. Results from this case study emphasize the seemingly simple (and obvious) fact that not all monitoring systems are created equal, which has real-world impacts on the design and documentation of ML monitoring systems.