Diagonal Rescaling For Neural Networks

TL;DR

This paper introduces a second-order stochastic gradient algorithm for neural networks that normalizes activations and offers new insights into stepsize scaling and curvature management, enhancing training robustness.

Contribution

It proposes a novel second-order algorithm with a block-diagonal structure that normalizes activations and provides new understanding of stepsize scaling and curvature adaptation.

Findings

Clarifies the role of stepsize scaling in popular algorithms

Highlights the importance of handling rapid curvature changes

Connects old tricks with modern normalization techniques

Abstract

We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations. Investigating why this algorithm lacks in robustness then reveals two interesting insights. The first insight suggests a new way to scale the stepsizes, clarifying popular algorithms such as RMSProp as well as old neural network tricks such as fanin stepsize scaling. The second insight stresses the practical importance of dealing with fast changes of the curvature of the cost.