Noise as a Double-Edged Sword: Reinforcement Learning Exploits Randomized Defenses in Neural Networks

TL;DR

This paper reveals that noise-based defenses in neural networks can sometimes be exploited by reinforcement learning attackers, leading to increased evasion success, and emphasizes the need for more nuanced defense strategies.

Contribution

It demonstrates that noise-based defenses can backfire against adaptive RL attackers, challenging the assumption that randomness always improves robustness.

Findings

Noise can be exploited by RL attackers to increase evasion success.

In some cases, noise defenses outperform other strategies by up to 20%.

The effectiveness of noise defenses varies across different classifiers.

Abstract

This study investigates a counterintuitive phenomenon in adversarial machine learning: the potential for noise-based defenses to inadvertently aid evasion attacks in certain scenarios. While randomness is often employed as a defensive strategy against adversarial examples, our research reveals that this approach can sometimes backfire, particularly when facing adaptive attackers using reinforcement learning (RL). Our findings show that in specific cases, especially with visually noisy classes, the introduction of noise in the classifier's confidence values can be exploited by the RL attacker, leading to a significant increase in evasion success rates. In some instances, the noise-based defense scenario outperformed other strategies by up to 20\% on a subset of classes. However, this effect was not consistent across all classifiers tested, highlighting the complexity of the interaction between noise-based defenses and different models. These results suggest that in some cases, noise-based defenses can inadvertently create an adversarial training loop beneficial to the RL attacker. Our study emphasizes the need for a more nuanced approach to defensive strategies in adversarial machine learning, particularly in safety-critical applications. It challenges the assumption that randomness universally enhances defense against evasion attacks and highlights the importance of considering adaptive, RL-based attackers when designing robust defense mechanisms.