The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networks

TL;DR

This paper explores the geometric structure of gradient flows in deep linear convolutional networks, showing they can be described as Riemannian gradient flows on function space under broad conditions.

Contribution

It generalizes previous results by establishing Riemannian gradient flow representations for convolutional networks regardless of initialization, including multi-dimensional cases.

Findings

Gradient flow for convolutional networks can be expressed as Riemannian gradient flow.

This holds for D-dimensional convolutions with D ≥ 2.

For D=1, it requires strides greater than one.

Abstract

We study geometric properties of the gradient flow for learning deep linear convolutional networks. For linear fully connected networks, it has been shown recently that the corresponding gradient flow on parameter space can be written as a Riemannian gradient flow on function space (i.e., on the product of weight matrices) if the initialization satisfies a so-called balancedness condition. We establish that the gradient flow on parameter space for learning linear convolutional networks can be written as a Riemannian gradient flow on function space regardless of the initialization. This result holds for $D$-dimensional convolutions with $D \geq 2$, and for $D =1$ it holds if all so-called strides of the convolutions are greater than one. The corresponding Riemannian metric depends on the initialization.