MLDemon: Deployment Monitoring for Machine Learning Systems

TL;DR

MLDemon is a real-time deployment monitoring system for machine learning models that intelligently decides when to request labels to ensure reliability amid distribution shifts, outperforming existing methods.

Contribution

The paper introduces MLDemon, a novel approach combining unlabeled data and selective labeling for effective real-time model performance monitoring.

Findings

Outperforms existing monitoring approaches on diverse datasets.

Provides theoretical guarantees of minimax rate optimality.

Effectively manages label budget constraints in deployment scenarios.

Abstract

Post-deployment monitoring of ML systems is critical for ensuring reliability, especially as new user inputs can differ from the training distribution. Here we propose a novel approach, MLDemon, for ML DEployment MONitoring. MLDemon integrates both unlabeled data and a small amount of on-demand labels to produce a real-time estimate of the ML model's current performance on a given data stream. Subject to budget constraints, MLDemon decides when to acquire additional, potentially costly, expert supervised labels to verify the model. On temporal datasets with diverse distribution drifts and models, MLDemon outperforms existing approaches. Moreover, we provide theoretical analysis to show that MLDemon is minimax rate optimal for a broad class of distribution drifts.