Path-SGD: Path-Normalized Optimization in Deep Neural Networks

TL;DR

This paper introduces Path-SGD, a new optimization method for deep neural networks based on a rescaling-invariant geometry, demonstrating empirical improvements over traditional optimizers like SGD and AdaGrad.

Contribution

The paper proposes Path-SGD, an approximate steepest descent method using a path-wise regularizer, offering a novel geometry for training neural networks.

Findings

Path-SGD is easy and efficient to implement.

Path-SGD outperforms SGD and AdaGrad in experiments.

The method is based on a rescaling-invariant geometry.

Abstract

We revisit the choice of SGD for training deep neural networks by reconsidering the appropriate geometry in which to optimize the weights. We argue for a geometry invariant to rescaling of weights that does not affect the output of the network, and suggest Path-SGD, which is an approximate steepest descent method with respect to a path-wise regularizer related to max-norm regularization. Path-SGD is easy and efficient to implement and leads to empirical gains over SGD and AdaGrad.