On the Extreme Variance of Certified Local Robustness Across Model Seeds

TL;DR

This paper reveals that the certified robustness of neural networks varies extremely across different training seeds and datasets, raising concerns about the reliability of current robustness verification methods.

Contribution

It demonstrates the significant variance in certified robustness due to random seed choices and dataset differences, highlighting the need for more reliable verification practices.

Findings

Certified robustness variance exceeds recent robustness improvements.

Robustness generalization to unseen data is highly inconsistent.

Results challenge the dependability of current robustness verification methods.

Abstract

Robustness verification of neural networks, referring to formally proving that neural networks satisfy robustness properties, is of crucial importance in safety-critical applications, where model failures can result in loss of human life or million-dollar damages. However, the dependability of verification results may be questioned due to sources of randomness in machine learning, and although this has been widely investigated for accuracy, its impact on robustness verification remains unknown. In this paper, we demonstrate a concerning result: Models that differ only in random seeds during training exhibit extreme variance in their certified robustness, with a standard deviation that is statistically larger than the marginal robustness improvements reported in recent machine learning papers. In addition, we also show that certified robustness generalization to unseen data varies significantly across datasets, falling short of the dependability expectations for safety-critical tasks. Our findings are major concerns because: (i) machine learning results in certified robustness are likely unconvincing due to extreme variance in certified robustness, and (ii) a ``lucky'' model seed in a test set cannot be guaranteed to maintain its higher certified robustness under a different test set. In light of these results, we urge researchers to increase the reporting of confidence intervals for certified robustness, and we urge those verifying neural networks to be more comprehensive in verification by using large-scale, diverse, and unseen data.