Monitoring and Adapting ML Models on Mobile Devices

TL;DR

This paper presents a novel end-to-end system for monitoring and adapting machine learning models on mobile devices, addressing accuracy degradation due to data drift without user feedback.

Contribution

It introduces the first system capable of continuous monitoring, root cause analysis, and cause-specific adaptation of ML models on mobile devices.

Findings

System improves accuracy by 15% on driving car dataset.

Consistently outperforms existing approaches in accuracy.

Effectively identifies root causes of model degradation.

Abstract

ML models are increasingly being pushed to mobile devices, for low-latency inference and offline operation. However, once the models are deployed, it is hard for ML operators to track their accuracy, which can degrade unpredictably (e.g., due to data drift). We design the first end-to-end system for continuously monitoring and adapting models on mobile devices without requiring feedback from users. Our key observation is that often model degradation is due to a specific root cause, which may affect a large group of devices. Therefore, once the system detects a consistent degradation across a large number of devices, it employs a root cause analysis to determine the origin of the problem and applies a cause-specific adaptation. We evaluate the system on two computer vision datasets, and show it consistently boosts accuracy compared to existing approaches. On a dataset containing photos collected from driving cars, our system improves the accuracy on average by 15%.