Deep Learning Meets Sparse Regularization: A Signal Processing Perspective

TL;DR

This paper introduces a mathematical framework rooted in signal processing to better understand deep neural networks, explaining their effectiveness and architectural choices through concepts like sparse regularization and transform-domain analysis.

Contribution

It presents a novel mathematical framework that links deep learning performance to signal processing techniques, providing insights into regularization, architecture, and high-dimensional data handling.

Findings

Explains the role of weight decay regularization in neural networks.

Clarifies the use of skip connections and low-rank matrices.

Highlights the importance of sparsity in high-dimensional problems.

Abstract

Deep learning has been wildly successful in practice and most state-of-the-art machine learning methods are based on neural networks. Lacking, however, is a rigorous mathematical theory that adequately explains the amazing performance of deep neural networks. In this article, we present a relatively new mathematical framework that provides the beginning of a deeper understanding of deep learning. This framework precisely characterizes the functional properties of neural networks that are trained to fit to data. The key mathematical tools which support this framework include transform-domain sparse regularization, the Radon transform of computed tomography, and approximation theory, which are all techniques deeply rooted in signal processing. This framework explains the effect of weight decay regularization in neural network training, the use of skip connections and low-rank weight matrices in network architectures, the role of sparsity in neural networks, and explains why neural networks can perform well in high-dimensional problems.