Compressive Sensing and Neural Networks from a Statistical Learning Perspective

TL;DR

This paper analyzes the generalization properties of neural networks inspired by iterative soft-thresholding algorithms for sparse reconstruction, providing theoretical bounds that scale favorably with network depth and measurements.

Contribution

It introduces a class of neural networks based on unfolding ISTA for sparse recovery and derives a novel generalization error bound for these networks.

Findings

Generalization error scales logarithmically with network depth.

Error bound is at most linear in the number of measurements.

Proposes a new approach for dictionary learning via neural network training.

Abstract

Various iterative reconstruction algorithms for inverse problems can be unfolded as neural networks. Empirically, this approach has often led to improved results, but theoretical guarantees are still scarce. While some progress on generalization properties of neural networks have been made, great challenges remain. In this chapter, we discuss and combine these topics to present a generalization error analysis for a class of neural networks suitable for sparse reconstruction from few linear measurements. The hypothesis class considered is inspired by the classical iterative soft-thresholding algorithm (ISTA). The neural networks in this class are obtained by unfolding iterations of ISTA and learning some of the weights. Based on training samples, we aim at learning the optimal network parameters via empirical risk minimization and thereby the optimal network that reconstructs signals from their compressive linear measurements. In particular, we may learn a sparsity basis that is shared by all of the iterations/layers and thereby obtain a new approach for dictionary learning. For this class of networks, we present a generalization bound, which is based on bounding the Rademacher complexity of hypothesis classes consisting of such deep networks via Dudley's integral. Remarkably, under realistic conditions, the generalization error scales only logarithmically in the number of layers, and at most linear in number of measurements.