Towards Both Accurate and Robust Neural Networks without Extra Data

TL;DR

This paper introduces an adversarial feature stacking (AFS) model that combines multiple feature extractors to improve both accuracy and robustness of neural networks without requiring extra data, addressing a key trade-off.

Contribution

The paper proposes a novel AFS model that effectively balances accuracy and robustness, supported by theoretical analysis and strong empirical results.

Findings

Achieves ~6% accuracy improvement on CIFAR-10

Achieves ~10% accuracy improvement on CIFAR-100

Maintains or exceeds robustness of state-of-the-art methods

Abstract

Deep neural networks have achieved remarkable performance in various applications but are extremely vulnerable to adversarial perturbation. The most representative and promising methods that can enhance model robustness, such as adversarial training and its variants, substantially degrade model accuracy on benign samples, limiting practical utility. Although incorporating extra training data can alleviate the trade-off to a certain extent, it remains unsolved to achieve both robustness and accuracy under limited training data. Here, we demonstrate the feasibility of overcoming the trade-off, by developing an adversarial feature stacking (AFS) model, which combines multiple independent feature extractors with varied levels of robustness and accuracy. Theoretical analysis is further conducted, and general principles for the selection of basic feature extractors are provided. We evaluate the AFS model on CIFAR-10 and CIFAR-100 datasets with strong adaptive attack methods, significantly advancing the state-of-the-art in terms of the trade-off. The AFS model achieves a benign accuracy improvement of ~6% on CIFAR-10 and ~10% on CIFAR-100 with comparable or even stronger robustness than the state-of-the-art adversarial training methods.