Geometry of Linear Convolutional Networks

TL;DR

This paper explores the geometric structure of functions represented by linear convolutional networks, revealing how their architecture influences the function space and optimization landscape, with implications for filter repetition and decomposition.

Contribution

It characterizes the semi-algebraic nature of LCN function spaces, analyzes critical points and invariants in optimization, and links network architecture to polynomial factorizations.

Findings

LCNs form semi-algebraic sets, unlike fully-connected networks.

Optimized LCN parameters often involve repeated or decomposable filters.

The architecture impacts the geometry and critical points of the function space.

Abstract

We study the family of functions that are represented by a linear convolutional neural network (LCN). These functions form a semi-algebraic subset of the set of linear maps from input space to output space. In contrast, the families of functions represented by fully-connected linear networks form algebraic sets. We observe that the functions represented by LCNs can be identified with polynomials that admit certain factorizations, and we use this perspective to describe the impact of the network's architecture on the geometry of the resulting function space. We further study the optimization of an objective function over an LCN, analyzing critical points in function space and in parameter space, and describing dynamical invariants for gradient descent. Overall, our theory predicts that the optimized parameters of an LCN will often correspond to repeated filters across layers, or filters that can be decomposed as repeated filters. We also conduct numerical and symbolic experiments that illustrate our results and present an in-depth analysis of the landscape for small architectures.