A Human-Centric Take on Model Monitoring

TL;DR

This paper explores the human-centric challenges and needs in deploying and monitoring machine learning models across various high-stakes domains, emphasizing the importance of understandable, actionable, and customizable monitoring systems.

Contribution

It provides empirical insights from interviews with practitioners, highlighting key human-centric requirements and challenges in model monitoring post-deployment.

Findings

Need for monitoring systems to clarify impact on outcomes

Insights must be actionable and domain-specific

Monitoring should be cognitively considerate to prevent overload

Abstract

Predictive models are increasingly used to make various consequential decisions in high-stakes domains such as healthcare, finance, and policy. It becomes critical to ensure that these models make accurate predictions, are robust to shifts in the data, do not rely on spurious features, and do not unduly discriminate against minority groups. To this end, several approaches spanning various areas such as explainability, fairness, and robustness have been proposed in recent literature. Such approaches need to be human-centered as they cater to the understanding of the models to their users. However, there is a research gap in understanding the human-centric needs and challenges of monitoring machine learning (ML) models once they are deployed. To fill this gap, we conducted an interview study with 13 practitioners who have experience at the intersection of deploying ML models and engaging with customers spanning domains such as financial services, healthcare, hiring, online retail, computational advertising, and conversational assistants. We identified various human-centric challenges and requirements for model monitoring in real-world applications. Specifically, we found the need and the challenge for the model monitoring systems to clarify the impact of the monitoring observations on outcomes. Further, such insights must be actionable, robust, customizable for domain-specific use cases, and cognitively considerate to avoid information overload.