On the Expressive Power of Deep Polynomial Neural Networks

TL;DR

This paper investigates the expressive capacity of deep neural networks with polynomial activations, providing theoretical formulas and bounds for their functional space, and exploring implications for network design and training.

Contribution

It introduces a novel algebraic geometric measure of expressive power, deriving exact formulas and bounds, and links network architecture to optimization and tensor decompositions.

Findings

Dimension of algebraic variety measures expressive power

Layer width configurations affect the functional space coverage

Connections between network architecture, optimization, and tensor decompositions

Abstract

We study deep neural networks with polynomial activations, particularly their expressive power. For a fixed architecture and activation degree, a polynomial neural network defines an algebraic map from weights to polynomials. The image of this map is the functional space associated to the network, and it is an irreducible algebraic variety upon taking closure. This paper proposes the dimension of this variety as a precise measure of the expressive power of polynomial neural networks. We obtain several theoretical results regarding this dimension as a function of architecture, including an exact formula for high activation degrees, as well as upper and lower bounds on layer widths in order for deep polynomials networks to fill the ambient functional space. We also present computational evidence that it is profitable in terms of expressiveness for layer widths to increase monotonically and then decrease monotonically. Finally, we link our study to favorable optimization properties when training weights, and we draw intriguing connections with tensor and polynomial decompositions.