Utilizing Class Separation Distance for the Evaluation of Corruption Robustness of Machine Learning Classifiers

TL;DR

This paper introduces MSCR, a dataset-specific metric based on class separation distance, to evaluate and compare the corruption robustness of classifiers, revealing that robustness can be improved without accuracy tradeoffs.

Contribution

The paper proposes a novel robustness metric, MSCR, derived from class separation distance, enabling interpretable and comparable robustness evaluation across classifiers.

Findings

MSCR effectively reflects different robustness levels on 2D and image data.

Training with simple data augmentation can slightly enhance classifier accuracy.

Robustness and accuracy tradeoff is not necessarily inherent, challenging common assumptions.

Abstract

Robustness is a fundamental pillar of Machine Learning (ML) classifiers, substantially determining their reliability. Methods for assessing classifier robustness are therefore essential. In this work, we address the challenge of evaluating corruption robustness in a way that allows comparability and interpretability on a given dataset. We propose a test data augmentation method that uses a robustness distance $\epsilon$ derived from the datasets minimal class separation distance. The resulting MSCR (minimal separation corruption robustness) metric allows a dataset-specific comparison of different classifiers with respect to their corruption robustness. The MSCR value is interpretable, as it represents the classifiers avoidable loss of accuracy due to statistical corruptions. On 2D and image data, we show that the metric reflects different levels of classifier robustness. Furthermore, we observe unexpected optima in classifiers robust accuracy through training and testing classifiers with different levels of noise. While researchers have frequently reported on a significant tradeoff on accuracy when training robust models, we strengthen the view that a tradeoff between accuracy and corruption robustness is not inherent. Our results indicate that robustness training through simple data augmentation can already slightly improve accuracy.