HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks

TL;DR

HOMRS is a novel method that automatically selects high-order metamorphic relations to improve the validation of deep neural networks, especially in safety-critical applications, by generating effective transformations with less computation.

Contribution

HOMRS introduces a multi-objective search to automatically build a compact set of high-order metamorphic relations, enhancing DNN testing efficiency and effectiveness over existing methods.

Findings

HOMRS effectively generalizes to input data distributions.

It produces a small, optimized set of high-order transformations.

Compared to DeepXplore, HOMRS is more efficient and generates valid transformations.

Abstract

Deep Neural Networks (DNN) applications are increasingly becoming a part of our everyday life, from medical applications to autonomous cars. Traditional validation of DNN relies on accuracy measures, however, the existence of adversarial examples has highlighted the limitations of these accuracy measures, raising concerns especially when DNN are integrated into safety-critical systems. In this paper, we present HOMRS, an approach to boost metamorphic testing by automatically building a small optimized set of high order metamorphic relations from an initial set of elementary metamorphic relations. HOMRS' backbone is a multi-objective search; it exploits ideas drawn from traditional systems testing such as code coverage, test case, path diversity as well as input validation. We applied HOMRS to MNIST/LeNet and SVHN/VGG and we report evidence that it builds a small but effective set of high-order transformations that generalize well to the input data distribution. Moreover, comparing to similar generation technique such as DeepXplore, we show that our distribution-based approach is more effective, generating valid transformations from an uncertainty quantification point of view, while requiring less computation time by leveraging the generalization ability of the approach.