ARBiBench: Benchmarking Adversarial Robustness of Binarized Neural Networks

TL;DR

This paper introduces ARBiBench, a comprehensive benchmark for evaluating the adversarial robustness of binarized neural networks on CIFAR-10 and ImageNet, revealing dataset-dependent robustness patterns and potential for future improvements.

Contribution

The paper presents ARBiBench, the first extensive benchmark for assessing BNN robustness against adversarial attacks, providing new insights into their vulnerabilities and robustness characteristics.

Findings

BNNs show opposite robustness performance on CIFAR-10 and ImageNet under white-box attacks.

BNNs are more robust under black-box attacks across datasets.

Different BNNs display similar robustness patterns.

Abstract

Network binarization exhibits great potential for deployment on resource-constrained devices due to its low computational cost. Despite the critical importance, the security of binarized neural networks (BNNs) is rarely investigated. In this paper, we present ARBiBench, a comprehensive benchmark to evaluate the robustness of BNNs against adversarial perturbations on CIFAR-10 and ImageNet. We first evaluate the robustness of seven influential BNNs on various white-box and black-box attacks. The results reveal that 1) The adversarial robustness of BNNs exhibits a completely opposite performance on the two datasets under white-box attacks. 2) BNNs consistently exhibit better adversarial robustness under black-box attacks. 3) Different BNNs exhibit certain similarities in their robustness performance. Then, we conduct experiments to analyze the adversarial robustness of BNNs based on these insights. Our research contributes to inspiring future research on enhancing the robustness of BNNs and advancing their application in real-world scenarios.