Robust NAS under adversarial training: benchmark, theory, and beyond
Yongtao Wu, Fanghui Liu, Carl-Johann Simon-Gabriel, Grigorios G, Chrysos, Volkan Cevher
TL;DR
This paper introduces a benchmark dataset and a theoretical framework for neural architecture search (NAS) focused on robustness against adversarial attacks, addressing current gaps in evaluation and guarantees.
Contribution
It provides a comprehensive dataset for evaluating robust NAS and develops a generalization theory using neural tangent kernels for multi-objective adversarial training.
Findings
Benchmark dataset includes clean and robust accuracy for NAS-Bench-201 networks.
Theoretical analysis offers guarantees for architecture search under adversarial training.
Facilitates reliable evaluation and theoretical understanding in robust NAS.
Abstract
Recent developments in neural architecture search (NAS) emphasize the significance of considering robust architectures against malicious data. However, there is a notable absence of benchmark evaluations and theoretical guarantees for searching these robust architectures, especially when adversarial training is considered. In this work, we aim to address these two challenges, making twofold contributions. First, we release a comprehensive data set that encompasses both clean accuracy and robust accuracy for a vast array of adversarially trained networks from the NAS-Bench-201 search space on image datasets. Then, leveraging the neural tangent kernel (NTK) tool from deep learning theory, we establish a generalization theory for searching architecture in terms of clean accuracy and robust accuracy under multi-objective adversarial training. We firmly believe that our benchmark and…
Peer Reviews
Decision·ICLR 2024 poster
1. The paper's principal strength lies in its creation of an adversarially trained search space, which required 107k GPU hours to construct. This significant advancement is poised to benefit researchers focusing on robust architecture search in the future immensely. 2. The analyses conducted within this adversarial search space are considered to be novel. It is particularly noteworthy to observe cross-sectional evidence in the NAS search space, confirming the expectation that a 3x3 kernel size C
1. The paper lacks a detailed explanation and justification for the necessity of employing twice perturbation, as mentioned at the end of page 7. It remains unclear why robust NTK requires a double perturbation when conventional adversarial training typically examines generalization from a single perturbation. 2. While it is posited that robust accuracy is influenced by the adversarial term, the theoretical analysis provided appears to be a reiteration of what was presented by Zhu et al. and Cao
There is not much literature that presents a unified robust NAS benchmark, in that sense the work targets a unique space of research that needs exploration. The NTK score vs accuracy correlation sounds interesting.
1. It has been well known that adversarial robustness often may occur due to gradient obfuscation and is applicable for different sparse and dense model architectures [1,2]. Thus, a discussion on that would be necessary. In specific, is there a way to benchmark based on a subnet's susceptibility to be more prone towards obfuscation? 2. Di you use ImageNet or ImageNet-16-120? As the Fig. 2 and the contribution section has mention of each. 3. Please demonstrate the efficacy of the NASRobBench on
- As the authors note, adversarially training takes considerably more time than standard training with the cross-entropy loss. Given the enormous computational cost of adversarial training and evaluating a large number of architectures, I believe the authors really have done an impressive set of experiments. - I generally agree with the necessity to create a NAS benchmark that targets robustness and generalization beyond standard test data. - I do not feel confident enough to review and comment
- Currently, the robust accuracy is measured only under FGSM and PGD attacks. I think the performance under stronger attack methods (e.g., AutoAttack) is necessary to make the benchmark more reliable. - The authors observe that the standard and robust accuracies exhibit a meaningful level of correlation. If they are, do we really need a separate robustness benchmark that includes robust accuracies? Wouldn’t searching for the optimal architecture in a conventional sense work as a good proxy and n
Videos
Taxonomy
TopicsAdversarial Robustness in Machine Learning
MethodsSparse Evolutionary Training