An Integrated Approach to Produce Robust Models with High Efficiency

TL;DR

This paper introduces a combined approach using a relaxation quantization algorithm and a novel loss function to produce deep neural networks that are both robust against adversarial attacks and efficient for practical deployment.

Contribution

The work proposes a convergent relaxation quantization method combined with a new loss function to enhance robustness and efficiency of DNNs, addressing accuracy and sparsity issues under adversarial training.

Findings

Quantization produces sparse DNNs, especially at high precision.

The trade-off loss preserves accuracy and enhances channel sparsity.

Robustness is maintained with minimal loss under strong attacks.

Abstract

Deep Neural Networks (DNNs) needs to be both efficient and robust for practical uses. Quantization and structure simplification are promising ways to adapt DNNs to mobile devices, and adversarial training is the most popular method to make DNNs robust. In this work, we try to obtain both features by applying a convergent relaxation quantization algorithm, Binary-Relax (BR), to a robust adversarial-trained model, ResNets Ensemble via Feynman-Kac Formalism (EnResNet). We also discover that high precision, such as ternary (tnn) and 4-bit, quantization will produce sparse DNNs. However, this sparsity is unstructured under advarsarial training. To solve the problems that adversarial training jeopardizes DNNs' accuracy on clean images and the struture of sparsity, we design a trade-off loss function that helps DNNs preserve their natural accuracy and improve the channel sparsity. With our trade-off loss function, we achieve both goals with no reduction of resistance under weak attacks and very minor reduction of resistance under strong attcks. Together with quantized EnResNet with trade-off loss function, we provide robust models that have high efficiency.