Depth Without the Magic: Inductive Bias of Natural Gradient Descent

TL;DR

This paper investigates how natural gradient descent's invariance to reparameterization affects the solutions it finds in deep linear networks and matrix factorization, revealing limitations in generalization compared to standard gradient descent.

Contribution

It characterizes the behavior of natural gradient flow in deep linear models and shows conditions where it fails to generalize, contrasting with standard gradient descent.

Findings

Natural gradient flow follows the same trajectory regardless of parameterization.

Natural gradient descent can fail to generalize in certain problems.

Standard gradient descent with appropriate architecture can outperform natural gradient in generalization.

Abstract

In gradient descent, changing how we parametrize the model can lead to drastically different optimization trajectories, giving rise to a surprising range of meaningful inductive biases: identifying sparse classifiers or reconstructing low-rank matrices without explicit regularization. This implicit regularization has been hypothesised to be a contributing factor to good generalization in deep learning. However, natural gradient descent is approximately invariant to reparameterization, it always follows the same trajectory and finds the same optimum. The question naturally arises: What happens if we eliminate the role of parameterization, which solution will be found, what new properties occur? We characterize the behaviour of natural gradient flow in deep linear networks for separable classification under logistic loss and deep matrix factorization. Some of our findings extend to nonlinear neural networks with sufficient but finite over-parametrization. We demonstrate that there exist learning problems where natural gradient descent fails to generalize, while gradient descent with the right architecture performs well.