Characterizing and Taming Model Instability Across Edge Devices

TL;DR

This paper systematically characterizes prediction divergence of machine learning models across various edge devices, introduces a new instability metric, and proposes fine-tuning methods to significantly reduce this instability.

Contribution

It is the first to analyze model prediction instability across real-world edge devices and introduces techniques to mitigate this divergence.

Findings

14-17% of images show divergent classifications across devices

Differences in compression and image processing significantly contribute to instability

Fine-tuning models reduces instability by 75%

Abstract

The same machine learning model running on different edge devices may produce highly-divergent outputs on a nearly-identical input. Possible reasons for the divergence include differences in the device sensors, the device's signal processing hardware and software, and its operating system and processors. This paper presents the first methodical characterization of the variations in model prediction across real-world mobile devices. We demonstrate that accuracy is not a useful metric to characterize prediction divergence, and introduce a new metric, instability, which captures this variation. We characterize different sources for instability, and show that differences in compression formats and image signal processing account for significant instability in object classification models. Notably, in our experiments, 14-17% of images produced divergent classifications across one or more phone models. We evaluate three different techniques for reducing instability. In particular, we adapt prior work on making models robust to noise in order to fine-tune models to be robust to variations across edge devices. We demonstrate our fine-tuning techniques reduce instability by 75%.