Is ReLU Adversarially Robust?

TL;DR

This paper investigates the vulnerability of ReLU activation functions to adversarial examples, proposing a modified ReLU to improve robustness and demonstrating its effectiveness through experiments.

Contribution

It introduces a novel ReLU modification that enhances adversarial robustness and shows that adversarial training further improves model security.

Findings

ReLU functions are not inherently robust against adversarial attacks.

The proposed ReLU modification improves robustness in experiments.

Adversarial training on the modified ReLU model further enhances security.

Abstract

The efficacy of deep learning models has been called into question by the presence of adversarial examples. Addressing the vulnerability of deep learning models to adversarial examples is crucial for ensuring their continued development and deployment. In this work, we focus on the role of rectified linear unit (ReLU) activation functions in the generation of adversarial examples. ReLU functions are commonly used in deep learning models because they facilitate the training process. However, our empirical analysis demonstrates that ReLU functions are not robust against adversarial examples. We propose a modified version of the ReLU function, which improves robustness against adversarial examples. Our results are supported by an experiment, which confirms the effectiveness of our proposed modification. Additionally, we demonstrate that applying adversarial training to our customized model further enhances its robustness compared to a general model.