Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks

TL;DR

This paper investigates how different architectural choices in deep neural networks affect adversarial robustness, revealing that certain configurations can enhance robustness without increasing parameters.

Contribution

It provides a comprehensive analysis of the impact of network width and depth on adversarial robustness, offering new architectural guidelines and theoretical explanations.

Findings

More parameters do not necessarily improve robustness.

Reducing capacity at the last stage can enhance robustness.

An optimal architecture exists under the same parameter budget.

Abstract

Deep neural networks (DNNs) are known to be vulnerable to adversarial attacks. A range of defense methods have been proposed to train adversarially robust DNNs, among which adversarial training has demonstrated promising results. However, despite preliminary understandings developed for adversarial training, it is still not clear, from the architectural perspective, what configurations can lead to more robust DNNs. In this paper, we address this gap via a comprehensive investigation on the impact of network width and depth on the robustness of adversarially trained DNNs. Specifically, we make the following key observations: 1) more parameters (higher model capacity) does not necessarily help adversarial robustness; 2) reducing capacity at the last stage (the last group of blocks) of the network can actually improve adversarial robustness; and 3) under the same parameter budget, there exists an optimal architectural configuration for adversarial robustness. We also provide a theoretical analysis explaning why such network configuration can help robustness. These architectural insights can help design adversarially robust DNNs. Code is available at \url{https://github.com/HanxunH/RobustWRN}.