Window-Based Distribution Shift Detection for Deep Neural Networks

TL;DR

This paper introduces a novel, efficient distribution shift detection method for deep neural networks that monitors real-time data streams, outperforming existing methods in accuracy and computational efficiency, suitable for large-scale applications.

Contribution

The paper proposes a new distribution deviation detection technique based on a tight coverage generalization bound, reducing computational complexity and enabling real-world deployment at scale.

Findings

Performs on-par or better than state-of-the-art methods.

Consumes five orders of magnitude less computation time.

Eliminates linear dependence on source distribution size.

Abstract

To deploy and operate deep neural models in production, the quality of their predictions, which might be contaminated benignly or manipulated maliciously by input distributional deviations, must be monitored and assessed. Specifically, we study the case of monitoring the healthy operation of a deep neural network (DNN) receiving a stream of data, with the aim of detecting input distributional deviations over which the quality of the network's predictions is potentially damaged. Using selective prediction principles, we propose a distribution deviation detection method for DNNs. The proposed method is derived from a tight coverage generalization bound computed over a sample of instances drawn from the true underlying distribution. Based on this bound, our detector continuously monitors the operation of the network out-of-sample over a test window and fires off an alarm whenever a deviation is detected. Our novel detection method performs on-par or better than the state-of-the-art, while consuming substantially lower computation time (five orders of magnitude reduction) and space complexities. Unlike previous methods, which require at least linear dependence on the size of the source distribution for each detection, rendering them inapplicable to ``Google-Scale'' datasets, our approach eliminates this dependence, making it suitable for real-world applications.