Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective

TL;DR

This paper explores hidden convex structures in deep neural networks, especially ReLU-based models, by linking them to sparse signal processing, which could improve understanding and training of neural networks.

Contribution

It reveals hidden convexities in ReLU neural networks and connects deep learning with sparse signal processing to enhance theoretical understanding.

Findings

Identification of convex equivalences in ReLU networks

Connection between neural networks and sparse signal models

Potential for improved training and analysis methods

Abstract

Deep neural networks (DNNs), particularly those using Rectified Linear Unit (ReLU) activation functions, have achieved remarkable success across diverse machine learning tasks, including image recognition, audio processing, and language modeling. Despite this success, the non-convex nature of DNN loss functions complicates optimization and limits theoretical understanding. In this paper, we highlight how recently developed convex equivalences of ReLU NNs and their connections to sparse signal processing models can address the challenges of training and understanding NNs. Recent research has uncovered several hidden convexities in the loss landscapes of certain NN architectures, notably two-layer ReLU networks and other deeper or varied architectures. This paper seeks to provide an accessible and educational overview that bridges recent advances in the mathematics of deep learning with traditional signal processing, encouraging broader signal processing applications.