Defensive Quantization: When Efficiency Meets Robustness

TL;DR

This paper introduces Defensive Quantization, a novel method that enhances the robustness of quantized neural networks against adversarial attacks while maintaining efficiency, by controlling the network's Lipschitz constant during quantization.

Contribution

The paper proposes a new quantization technique that jointly optimizes for efficiency and robustness, addressing the vulnerability of traditional quantization to adversarial noise.

Findings

DQ improves robustness against adversarial attacks compared to vanilla quantization.

DQ can outperform full-precision models in robustness while maintaining efficiency.

DQ also enhances accuracy of quantized models in non-adversarial settings.

Abstract

Neural network quantization is becoming an industry standard to efficiently deploy deep learning models on hardware platforms, such as CPU, GPU, TPU, and FPGAs. However, we observe that the conventional quantization approaches are vulnerable to adversarial attacks. This paper aims to raise people's awareness about the security of the quantized models, and we designed a novel quantization methodology to jointly optimize the efficiency and robustness of deep learning models. We first conduct an empirical study to show that vanilla quantization suffers more from adversarial attacks. We observe that the inferior robustness comes from the error amplification effect, where the quantization operation further enlarges the distance caused by amplified noise. Then we propose a novel Defensive Quantization (DQ) method by controlling the Lipschitz constant of the network during quantization, such that the magnitude of the adversarial noise remains non-expansive during inference. Extensive experiments on CIFAR-10 and SVHN datasets demonstrate that our new quantization method can defend neural networks against adversarial examples, and even achieves superior robustness than their full-precision counterparts while maintaining the same hardware efficiency as vanilla quantization approaches. As a by-product, DQ can also improve the accuracy of quantized models without adversarial attack.