Suppressing the Unusual: towards Robust CNNs using Symmetric Activation Functions

TL;DR

This paper introduces Symmetric Activation Functions (SAF) into CNNs to suppress unusual signals, significantly enhancing robustness against adversarial and nonsensical inputs while maintaining performance on clean data.

Contribution

The paper proposes the integration of SAFs into CNNs, providing a simple method to improve adversarial robustness without sacrificing accuracy on normal samples.

Findings

SAF-enhanced CNNs are more resistant to adversarial attacks.

SAF networks perform comparably to standard CNNs on clean data.

The approach is easy to implement and train with existing strategies.

Abstract

Many deep Convolutional Neural Networks (CNN) make incorrect predictions on adversarial samples obtained by imperceptible perturbations of clean samples. We hypothesize that this is caused by a failure to suppress unusual signals within network layers. As remedy we propose the use of Symmetric Activation Functions (SAF) in non-linear signal transducer units. These units suppress signals of exceptional magnitude. We prove that SAF networks can perform classification tasks to arbitrary precision in a simplified situation. In practice, rather than use SAFs alone, we add them into CNNs to improve their robustness. The modified CNNs can be easily trained using popular strategies with the moderate training load. Our experiments on MNIST and CIFAR-10 show that the modified CNNs perform similarly to plain ones on clean samples, and are remarkably more robust against adversarial and nonsense samples.