Requirements for Developing Robust Neural Networks

TL;DR

This paper emphasizes that developing robust neural networks requires considering multiple performance metrics beyond validation accuracy, including explainability, adversarial resistance, and handling out-of-distribution data, to prevent unexpected failures.

Contribution

It highlights the importance of integrating robustness and interpretability metrics into neural network development, addressing overlooked aspects in traditional validation-focused approaches.

Findings

Validation accuracy alone is insufficient for model quality.

Models can be vulnerable or unreliable despite high validation accuracy.

Additional metrics like explainability and adversarial resistance are crucial.

Abstract

Validation accuracy is a necessary, but not sufficient, measure of a neural network classifier's quality. High validation accuracy during development does not guarantee that a model is free of serious flaws, such as vulnerability to adversarial attacks or a tendency to misclassify (with high confidence) data it was not trained on. The model may also be incomprehensible to a human or base its decisions on unreasonable criteria. These problems, which are not unique to classifiers, have been the focus of a substantial amount of recent research. However, they are not prioritized during model development, which almost always optimizes on validation accuracy to the exclusion of everything else. The product of this approach is likely to fail in unexpected ways outside of the training environment. We believe that, in addition to validation accuracy, the model development process must give added weight to other performance metrics such as explainability, resistance to adversarial attacks, and overconfidence on out-of-distribution data.