Using the Overlapping Score to Improve Corruption Benchmarks

TL;DR

This paper introduces the corruption overlapping score, a metric to evaluate and improve the diversity and representativeness of corruption benchmarks for neural network robustness testing.

Contribution

The paper proposes a novel metric called corruption overlapping score to identify flaws and enhance the quality of corruption benchmarks.

Findings

Correlations between corruptions reveal overlaps in robustness.

Using the score can improve benchmark diversity.

The metric helps identify weaknesses in existing benchmarks.

Abstract

Neural Networks are sensitive to various corruptions that usually occur in real-world applications such as blurs, noises, low-lighting conditions, etc. To estimate the robustness of neural networks to these common corruptions, we generally use a group of modeled corruptions gathered into a benchmark. Unfortunately, no objective criterion exists to determine whether a benchmark is representative of a large diversity of independent corruptions. In this paper, we propose a metric called corruption overlapping score, which can be used to reveal flaws in corruption benchmarks. Two corruptions overlap when the robustnesses of neural networks to these corruptions are correlated. We argue that taking into account overlappings between corruptions can help to improve existing benchmarks or build better ones.