Gradient Obfuscation Checklist Test Gives a False Sense of Security

TL;DR

The paper critiques the effectiveness of the Gradient Obfuscation Checklist Test, demonstrating it can falsely suggest robustness when models are actually vulnerable, by providing a counterexample that invalidates its conclusiveness.

Contribution

The authors show that the Gradient Obfuscation Checklist Test is not sufficient to determine the true source of robustness, highlighting the need for more reliable evaluation methods.

Findings

The checklist test can produce false positives for gradient obfuscation.

A counterexample demonstrates the test's limitations.

Gradient obfuscation can be mistaken for genuine robustness.

Abstract

One popular group of defense techniques against adversarial attacks is based on injecting stochastic noise into the network. The main source of robustness of such stochastic defenses however is often due to the obfuscation of the gradients, offering a false sense of security. Since most of the popular adversarial attacks are optimization-based, obfuscated gradients reduce their attacking ability, while the model is still susceptible to stronger or specifically tailored adversarial attacks. Recently, five characteristics have been identified, which are commonly observed when the improvement in robustness is mainly caused by gradient obfuscation. It has since become a trend to use these five characteristics as a sufficient test, to determine whether or not gradient obfuscation is the main source of robustness. However, these characteristics do not perfectly characterize all existing cases of gradient obfuscation, and therefore can not serve as a basis for a conclusive test. In this work, we present a counterexample, showing this test is not sufficient for concluding that gradient obfuscation is not the main cause of improvements in robustness.