Sparse Mixture Once-for-all Adversarial Training for Efficient In-Situ Trade-Off Between Accuracy and Robustness of DNNs

TL;DR

The paper introduces SMART, a sparse mixture approach enabling a single DNN to efficiently balance accuracy and robustness against adversarial attacks with reduced compute and parameter costs.

Contribution

SMART is a novel method that trains a model once with sparse expert paths for clean and adversarial images, enabling in-situ trade-offs with lower overhead.

Findings

Achieves up to 2.72x fewer non-zero parameters.

Maintains state-of-the-art accuracy-robustness trade-off.

Reduces compute overhead while preserving performance.

Abstract

Existing deep neural networks (DNNs) that achieve state-of-the-art (SOTA) performance on both clean and adversarially-perturbed images rely on either activation or weight conditioned convolution operations. However, such conditional learning costs additional multiply-accumulate (MAC) or addition operations, increasing inference memory and compute costs. To that end, we present a sparse mixture once for all adversarial training (SMART), that allows a model to train once and then in-situ trade-off between accuracy and robustness, that too at a reduced compute and parameter overhead. In particular, SMART develops two expert paths, for clean and adversarial images, respectively, that are then conditionally trained via respective dedicated sets of binary sparsity masks. Extensive evaluations on multiple image classification datasets across different models show SMART to have up to 2.72x fewer non-zero parameters costing proportional reduction in compute overhead, while yielding SOTA accuracy-robustness trade-off. Additionally, we present insightful observations in designing sparse masks to successfully condition on both clean and perturbed images.