Monitoring and explainability of models in production

TL;DR

This paper discusses the importance of monitoring and explainability in deployed machine learning models, covering techniques for performance, data drift detection, and interpretability, with examples of open source solutions.

Contribution

It provides an overview of challenges and recent open source solutions for monitoring and explaining models in production environments.

Findings

Effective data drift detection techniques are essential for maintaining model performance.

Open source tools are increasingly capable of supporting production-level monitoring and explainability.

Monitoring practices help identify issues early, ensuring high-quality ML services.

Abstract

The machine learning lifecycle extends beyond the deployment stage. Monitoring deployed models is crucial for continued provision of high quality machine learning enabled services. Key areas include model performance and data monitoring, detecting outliers and data drift using statistical techniques, and providing explanations of historic predictions. We discuss the challenges to successful implementation of solutions in each of these areas with some recent examples of production ready solutions using open source tools.