Safety Performance of Neural Networks in the Presence of Covariate Shift

TL;DR

This paper proposes a method to re-evaluate neural network safety performance under covariate shift by reshaping the test set based on neuron activation patterns observed during operation, using bounds and MILP constraints.

Contribution

It introduces a novel approach to approximate operational data distributions through neuron activation analysis for safety assessment without collecting new data.

Findings

Bounded neuron activation values using static analysis

MILP-based method for test set adjustment

Initial prototype demonstrates potential benefits and limitations

Abstract

Covariate shift may impact the operational safety performance of neural networks. A re-evaluation of the safety performance, however, requires collecting new operational data and creating corresponding ground truth labels, which often is not possible during operation. We are therefore proposing to reshape the initial test set, as used for the safety performance evaluation prior to deployment, based on an approximation of the operational data. This approximation is obtained by observing and learning the distribution of activation patterns of neurons in the network during operation. The reshaped test set reflects the distribution of neuron activation values as observed during operation, and may therefore be used for re-evaluating safety performance in the presence of covariate shift. First, we derive conservative bounds on the values of neurons by applying finite binning and static dataflow analysis. Second, we formulate a mixed integer linear programming (MILP) constraint for constructing the minimum set of data points to be removed in the test set, such that the difference between the discretized test and operational distributions is bounded. We discuss potential benefits and limitations of this constraint-based approach based on our initial experience with an implemented research prototype.