Improve the Robustness and Accuracy of Deep Neural Network with $L_{2,\infty}$ Normalization

TL;DR

This paper introduces $L_{2, abla}$ normalization for deep neural networks, demonstrating its effectiveness in enhancing robustness and accuracy by smoothing the network functions and reducing overfitting through theoretical analysis and empirical validation.

Contribution

It proposes a novel $L_{2, abla}$ normalization method for DNNs, providing theoretical bounds and an algorithm, with experimental results confirming improved robustness and accuracy.

Findings

Enhanced robustness measured by larger robust spheres.

Improved accuracy demonstrated on benchmark datasets.

Theoretical bounds support empirical results.

Abstract

In this paper, the robustness and accuracy of the deep neural network (DNN) was enhanced by introducing the $L_{2,\infty}$ normalization of the weight matrices of the DNN with Relu as the activation function. It is proved that the $L_{2,\infty}$ normalization leads to large dihedral angles between two adjacent faces of the polyhedron graph of the DNN function and hence smoother DNN functions, which reduces over-fitting. A measure is proposed for the robustness of a classification DNN, which is the average radius of the maximal robust spheres with the sample data as centers. A lower bound for the robustness measure is given in terms of the $L_{2,\infty}$ norm. Finally, an upper bound for the Rademacher complexity of DNN with $L_{2,\infty}$ normalization is given. An algorithm is given to train a DNN with the $L_{2,\infty}$ normalization and experimental results are used to show that the $L_{2,\infty}$ normalization is effective to improve the robustness and accuracy.