Energy Propagation in Scattering Convolution Networks Can Be Arbitrarily Slow

TL;DR

This paper demonstrates that energy decay in wavelet scattering networks can be arbitrarily slow, contrasting with previous results for Gabor-based transforms, and depends heavily on signal and filter frequency localization.

Contribution

It proves that energy decay in wavelet scattering transforms can be arbitrarily slow for generic signals, challenging prior assumptions of exponential decay.

Findings

Energy decay can be arbitrarily slow for wavelet scattering.

Rapid decay is unstable and not guaranteed for all signals.

Fast decay is achievable in certain Sobolev spaces with specific frequency localization.

Abstract

We analyze energy decay for deep convolutional neural networks employed as feature extractors, including Mallat's wavelet scattering transform. For time-frequency scattering transforms based on Gabor filters, previous work has established that energy decay is exponential for arbitrary square-integrable input signals. In contrast, our main results allow proving that this is false for wavelet scattering in arbitrary dimensions. Specifically, we show that the energy decay of wavelet and wavelet-like scattering transforms acting on generic square-integrable signals can be arbitrarily slow. Importantly, this slow decay behavior holds for dense subsets of $L^2(\mathbb{R}^d)$, indicating that rapid energy decay is generally an unstable property of signals. We complement these findings with positive results that allow us to infer fast (up to exponential) energy decay for generalized Sobolev spaces tailored to the frequency localization of the underlying filter bank. Both negative and positive results highlight that energy decay in scattering networks critically depends on the interplay between the respective frequency localizations of both the signal and the filters used.