Applying the Decisiveness and Robustness Metrics to Convolutional Neural Networks

TL;DR

This paper reviews and applies three classifier quality metrics—geometric accuracy, decisiveness, and robustness—to convolutional neural networks like AlexNet and DenseNet on large datasets such as ImageNet.

Contribution

The paper evaluates the suitability of three recent classifier quality metrics for large-scale CNNs and provides standardized definitions and practical calculations on real datasets.

Findings

Metrics can effectively assess CNN performance on large datasets

Decisiveness and robustness metrics offer new insights into classifier confidence

Application to AlexNet and DenseNet demonstrates practical utility

Abstract

We review three recently-proposed classifier quality metrics and consider their suitability for large-scale classification challenges such as applying convolutional neural networks to the 1000-class ImageNet dataset. These metrics, referred to as the "geometric accuracy," "decisiveness," and "robustness," are based on the generalized mean ($\rho$ equals 0, 1, and -2/3, respectively) of the classifier's self-reported and measured probabilities of correct classification. We also propose some minor clarifications to standardize the metric definitions. With these updates, we show some examples of calculating the metrics using deep convolutional neural networks (AlexNet and DenseNet) acting on large datasets (the German Traffic Sign Recognition Benchmark and ImageNet).