A Comprehensive Assessment Benchmark for Rigorously Evaluating Deep Learning Image Classifiers

TL;DR

This paper introduces a comprehensive benchmark for evaluating deep learning image classifiers across diverse data types, revealing current models' vulnerabilities and emphasizing the need for more robust evaluation methods.

Contribution

It proposes a unified evaluation benchmark using multiple data types and a single metric, highlighting the limitations of current evaluation protocols.

Findings

Current deep neural networks are vulnerable to certain data types.

State-of-the-art models are not reliably robust in diverse scenarios.

Models can be easily fooled into incorrect predictions.

Abstract

Reliable and robust evaluation methods are a necessary first step towards developing machine learning models that are themselves robust and reliable. Unfortunately, current evaluation protocols typically used to assess classifiers fail to comprehensively evaluate performance as they tend to rely on limited types of test data, and ignore others. For example, using the standard test data fails to evaluate the predictions made by the classifier to samples from classes it was not trained on. On the other hand, testing with data containing samples from unknown classes fails to evaluate how well the classifier can predict the labels for known classes. This article advocates benchmarking performance using a wide range of different types of data and using a single metric that can be applied to all such data types to produce a consistent evaluation of performance. Using the proposed benchmark it is found that current deep neural networks, including those trained with methods that are believed to produce state-of-the-art robustness, are vulnerable to making mistakes on certain types of data. This means that such models will be unreliable in real-world scenarios where they may encounter data from many different domains, and that they are insecure as they can be easily fooled into making the wrong decisions. It is hoped that these results will motivate the wider adoption of more comprehensive testing methods that will, in turn, lead to the development of more robust machine learning methods in the future. Code is available at: https://codeberg.org/mwspratling/RobustnessEvaluation