Pushing Stochastic Gradient towards Second-Order Methods -- Backpropagation Learning with Transformations in Nonlinearities

TL;DR

This paper introduces transformations in neural network hidden layers that make stochastic gradient descent behave more like second-order methods, accelerating learning but with potential risks of worse local optima.

Contribution

It extends previous work by adding a third normalization transformation and analyzing its connection to second-order optimization, supported by theoretical and experimental evidence.

Findings

Transformations make stochastic gradient behave closer to second-order methods.

The third transformation speeds up learning but may lead to worse local optima.

Experiments confirm theoretical analysis and reveal trade-offs in transformations.

Abstract

Recently, we proposed to transform the outputs of each hidden neuron in a multi-layer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scale of the outputs of each hidden neuron, and secondly by analyzing the connections to second order optimization methods. We show that the transformations make a simple stochastic gradient behave closer to second-order optimization methods and thus speed up learning. This is shown both in theory and with experiments. The experiments on the third transformation show that while it further increases the speed of learning, it can also hurt performance by converging to a worse local optimum, where both the inputs and outputs of many hidden neurons are close to zero.