Langevin algorithms for very deep Neural Networks with application to image classification

TL;DR

This paper compares Langevin algorithms with traditional methods for training very deep neural networks, demonstrating that Langevin methods improve performance in deep architectures by escaping local traps, especially with the new Layer Langevin variant.

Contribution

The paper introduces Layer Langevin, a novel Langevin algorithm that adds noise only to the deepest layers, and proves its effectiveness for training deep residual networks in image classification.

Findings

Langevin algorithms outperform non-Langevin methods in very deep networks.

Adding noise helps escape local minima in deep neural network training.

Layer Langevin improves training efficiency for deep residual architectures.

Abstract

Training a very deep neural network is a challenging task, as the deeper a neural network is, the more non-linear it is. We compare the performances of various preconditioned Langevin algorithms with their non-Langevin counterparts for the training of neural networks of increasing depth. For shallow neural networks, Langevin algorithms do not lead to any improvement, however the deeper the network is and the greater are the gains provided by Langevin algorithms. Adding noise to the gradient descent allows to escape from local traps, which are more frequent for very deep neural networks. Following this heuristic we introduce a new Langevin algorithm called Layer Langevin, which consists in adding Langevin noise only to the weights associated to the deepest layers. We then prove the benefits of Langevin and Layer Langevin algorithms for the training of popular deep residual architectures for image classification.