Volume-preserving Neural Networks

TL;DR

This paper introduces a volume-preserving neural network architecture that addresses the vanishing and exploding gradient problems by ensuring all layers maintain volume, leading to more stable training.

Contribution

The authors develop a novel neural network architecture with volume-preserving layers, combining rotation, permutation, diagonal, and a new coupled activation function.

Findings

Successfully applied to two standard datasets

Maintains stable gradients during training

Reduces vanishing and exploding gradient issues

Abstract

We propose a novel approach to addressing the vanishing (or exploding) gradient problem in deep neural networks. We construct a new architecture for deep neural networks where all layers (except the output layer) of the network are a combination of rotation, permutation, diagonal, and activation sublayers which are all volume preserving. Our approach replaces the standard weight matrix of a neural network with a combination of diagonal, rotational and permutation matrices, all of which are volume-preserving. We introduce a coupled activation function allowing us to preserve volume even in the activation function portion of a neural network layer. This control on the volume forces the gradient (on average) to maintain equilibrium and not explode or vanish. To demonstrate our architecture we apply our volume-preserving neural network model to two standard datasets.