Activation degree thresholds and expressiveness of polynomial neural networks

TL;DR

This paper investigates the expressive capacity of deep polynomial neural networks by analyzing their neurovariety geometry, introducing the activation degree threshold concept, and establishing bounds that demonstrate their maximum expressiveness.

Contribution

It introduces the activation degree threshold for polynomial neural networks, proves its existence and bounds, and confirms the high activation degree conjecture, highlighting architectures with maximal expressiveness.

Findings

Existence of activation degree threshold for all polynomial networks without width-one bottlenecks.

Universal quadratic upper bound on the activation degree threshold based on network width.

Equi-width polynomial neural networks are maximally expressive with activation degree threshold of one.

Abstract

We study the expressive power of deep polynomial neural networks through the geometry of their neurovariety. We introduce the notion of the activation degree threshold of a network architecture to express when the dimension of the neurovariety achieves its theoretical maximum. We prove the existence of the activation degree threshold for all polynomial neural networks without width-one bottlenecks and demonstrate a universal upper bound that is quadratic in the width of largest size. In doing so, we prove the high activation degree conjecture of Kileel, Trager, and Bruna. Certain structured architectures have exceptional activation degree thresholds, making them especially expressive in the sense of their neurovariety dimension. In this direction, we prove that polynomial neural networks with equi-width architectures are maximally expressive by showing their activation degree threshold is one.