Revisiting the Trade-off between Accuracy and Robustness via Weight Distribution of Filters

TL;DR

This paper investigates the intrinsic trade-off between accuracy and robustness in neural networks, analyzes filter weight distributions, and proposes a dynamic architecture called AW-Net to improve both aspects simultaneously.

Contribution

It introduces a theoretical analysis of filter weight distributions related to the accuracy-robustness trade-off and proposes a novel dynamic network architecture, AW-Net, that adaptively handles clean and adversarial inputs.

Findings

AW-Net outperforms state-of-the-art models in trade-off performance.

Dynamic weight adjustment improves robustness without sacrificing accuracy.

Theoretical analysis links filter weight distribution to the accuracy-robustness trade-off.

Abstract

Adversarial attacks have been proven to be potential threats to Deep Neural Networks (DNNs), and many methods are proposed to defend against adversarial attacks. However, while enhancing the robustness, the clean accuracy will decline to a certain extent, implying a trade-off existed between the accuracy and robustness. In this paper, to meet the trade-off problem, we theoretically explore the underlying reason for the difference of the filters' weight distribution between standard-trained and robust-trained models and then argue that this is an intrinsic property for static neural networks, thus they are difficult to fundamentally improve the accuracy and adversarial robustness at the same time. Based on this analysis, we propose a sample-wise dynamic network architecture named Adversarial Weight-Varied Network (AW-Net), which focuses on dealing with clean and adversarial examples with a "divide and rule" weight strategy. The AW-Net adaptively adjusts the network's weights based on regulation signals generated by an adversarial router, which is directly influenced by the input sample. Benefiting from the dynamic network architecture, clean and adversarial examples can be processed with different network weights, which provides the potential to enhance both accuracy and adversarial robustness. A series of experiments demonstrate that our AW-Net is architecture-friendly to handle both clean and adversarial examples and can achieve better trade-off performance than state-of-the-art robust models.