Adaptive Normalized Risk-Averting Training For Deep Neural Networks

TL;DR

This paper introduces Adaptive Normalized Risk-Averting Training (ANRAT), a novel approach for training deep neural networks that improves optimization by adaptively learning a convexity parameter, leading to better performance on visual recognition tasks.

Contribution

The paper presents a new error criterion and training method, ANRAT, which adaptively learns a convexity index to enhance deep neural network training and optimization.

Findings

Achieves comparable or superior results to existing methods on MNIST and CIFAR-10.

Improves training of deep neural networks without pretraining or tricks.

Compatible with various network architectures and regularization techniques.

Abstract

This paper proposes a set of new error criteria and learning approaches, Adaptive Normalized Risk-Averting Training (ANRAT), to attack the non-convex optimization problem in training deep neural networks (DNNs). Theoretically, we demonstrate its effectiveness on global and local convexity lower-bounded by the standard $L_p$-norm error. By analyzing the gradient on the convexity index $\lambda$, we explain the reason why to learn $\lambda$ adaptively using gradient descent works. In practice, we show how this method improves training of deep neural networks to solve visual recognition tasks on the MNIST and CIFAR-10 datasets. Without using pretraining or other tricks, we obtain results comparable or superior to those reported in recent literature on the same tasks using standard ConvNets + MSE/cross entropy. Performance on deep/shallow multilayer perceptrons and Denoised Auto-encoders is also explored. ANRAT can be combined with other quasi-Newton training methods, innovative network variants, regularization techniques and other specific tricks in DNNs. Other than unsupervised pretraining, it provides a new perspective to address the non-convex optimization problem in DNNs.