On the Geometry and Optimization of Polynomial Convolutional Networks

TL;DR

This paper investigates the geometric structure and optimization landscape of polynomial convolutional networks with monomial activations, revealing their expressivity and critical point characteristics using algebraic geometry tools.

Contribution

It provides a detailed geometric analysis of polynomial CNNs, including their neuromanifold properties, and derives explicit formulas for critical points in large dataset regression tasks.

Findings

Parameterization map is an isomorphism almost everywhere.

Computed the dimension and degree of the neuromanifold.

Derived an explicit formula for the number of critical points in optimization.

Abstract

We study convolutional neural networks with monomial activation functions. Specifically, we prove that their parameterization map is regular and is an isomorphism almost everywhere, up to rescaling the filters. By leveraging on tools from algebraic geometry, we explore the geometric properties of the image in function space of this map - typically referred to as neuromanifold. In particular, we compute the dimension and the degree of the neuromanifold, which measure the expressivity of the model, and describe its singularities. Moreover, for a generic large dataset, we derive an explicit formula that quantifies the number of critical points arising in the optimization of a regression loss.