Digital implementations of deep feature extractors are intrinsically informative

TL;DR

This paper establishes an upper bound on energy propagation speed in deep feature extractors across various neural network models, highlighting how domain structure influences decay rates and demonstrating exponential decay in specific cases.

Contribution

It introduces a unified theoretical framework for analyzing energy propagation in deep neural networks over different domains, including Euclidean and non-Euclidean spaces.

Findings

Proves an upper bound for energy propagation speed in neural networks.

Shows exponential energy decay in feature extractors with discrete signals.

Demonstrates energy decay in CNNs via scattering on LCA groups.

Abstract

Rapid information (energy) propagation in deep feature extractors is crucial to balance computational complexity versus expressiveness as a representation of the input. We prove an upper bound for the speed of energy propagation in a unified framework that covers different neural network models, both over Euclidean and non-Euclidean domains. Additional structural information about the signal domain can be used to explicitly determine or improve the rate of decay. To illustrate this, we show global exponential energy decay for a range of 1) feature extractors with discrete-domain input signals, and 2) convolutional neural networks (CNNs) via scattering over locally compact abelian (LCA) groups.