Identifiability of Deep Polynomial Neural Networks

TL;DR

This paper analyzes the conditions under which deep Polynomial Neural Networks are identifiable, linking algebraic properties with network architecture and activation degrees to enhance interpretability.

Contribution

It provides a comprehensive, constructive analysis of deep PNN identifiability, including new bounds and resolution of an open conjecture.

Findings

Identifiability depends on activation degrees and layer widths.

Non-increasing layer widths lead to generic identifiability.

Encoder-decoder networks are identifiable under certain width constraints.

Abstract

Polynomial Neural Networks (PNNs) possess a rich algebraic and geometric structure. However, their identifiability -- a key property for ensuring interpretability -- remains poorly understood. In this work, we present a comprehensive analysis of the identifiability of deep PNNs, including architectures with and without bias terms. Our results reveal an intricate interplay between activation degrees and layer widths in achieving identifiability. As special cases, we show that architectures with non-increasing layer widths are generically identifiable under mild conditions, while encoder-decoder networks are identifiable when the decoder widths do not grow too rapidly compared to the activation degrees. Our proofs are constructive and center on a connection between deep PNNs and low-rank tensor decompositions, and Kruskal-type uniqueness theorems. We also settle an open conjecture on the dimension of PNN's neurovarieties, and provide new bounds on the activation degrees required for it to reach the expected dimension.