Fast Approximate Spectral Normalization for Robust Deep Neural Networks

TL;DR

This paper proposes a fast approximate spectral normalization method for deep neural networks that enhances robustness against adversarial attacks while reducing computational costs.

Contribution

It introduces an efficient approximation algorithm based on Fourier transform and layer separation, improving spectral normalization scalability.

Findings

Significantly reduces spectral normalization computation time by up to 60%.

Improves model robustness against adversarial attacks by 61% on average.

Demonstrates effectiveness on large-scale neural networks.

Abstract

Deep neural networks (DNNs) play an important role in machine learning due to its outstanding performance compared to other alternatives. However, DNNs are not suitable for safety-critical applications since DNNs can be easily fooled by well-crafted adversarial examples. One promising strategy to counter adversarial attacks is to utilize spectral normalization, which ensures that the trained model has low sensitivity towards the disturbance of input samples. Unfortunately, this strategy requires exact computation of spectral norm, which is computation intensive and impractical for large-scale networks. In this paper, we introduce an approximate algorithm for spectral normalization based on Fourier transform and layer separation. The primary contribution of our work is to effectively combine the sparsity of weight matrix and decomposability of convolution layers. Extensive experimental evaluation demonstrates that our framework is able to significantly improve both time efficiency (up to 60\%) and model robustness (61\% on average) compared with the state-of-the-art spectral normalization.