CSTAR: Towards Compact and STructured Deep Neural Networks with Adversarial Robustness

TL;DR

CSTAR is a novel method that simultaneously achieves high model compression, structuredness, and adversarial robustness in deep neural networks, overcoming the performance trade-offs of previous approaches.

Contribution

The paper introduces CSTAR, a unified framework that jointly optimizes low-rankness, structuredness, and robustness, leading to better compressed and robust DNNs compared to prior methods.

Findings

CSTAR outperforms state-of-the-art methods on CIFAR-10 and ImageNet datasets.

CSTAR improves benign accuracy by up to 20.07% and robust accuracy by up to 11.91%.

CSTAR maintains high compression ratios with minimal accuracy loss.

Abstract

Model compression and model defense for deep neural networks (DNNs) have been extensively and individually studied. Considering the co-importance of model compactness and robustness in practical applications, several prior works have explored to improve the adversarial robustness of the sparse neural networks. However, the structured sparse models obtained by the exiting works suffer severe performance degradation for both benign and robust accuracy, thereby causing a challenging dilemma between robustness and structuredness of the compact DNNs. To address this problem, in this paper, we propose CSTAR, an efficient solution that can simultaneously impose the low-rankness-based Compactness, high STructuredness and high Adversarial Robustness on the target DNN models. By formulating the low-rankness and robustness requirement within the same framework and globally determining the ranks, the compressed DNNs can simultaneously achieve high compression performance and strong adversarial robustness. Evaluations for various DNN models on different datasets demonstrate the effectiveness of CSTAR. Compared with the state-of-the-art robust structured pruning methods, CSTAR shows consistently better performance. For instance, when compressing ResNet-18 on CIFAR-10, CSTAR can achieve up to 20.07% and 11.91% improvement for benign accuracy and robust accuracy, respectively. For compressing ResNet-18 with 16x compression ratio on Imagenet, CSTAR can obtain 8.58% benign accuracy gain and 4.27% robust accuracy gain compared to the existing robust structured pruning method.