Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via $\ell_1, \ell_0$, and transformed-$\ell_1$ Penalties

TL;DR

This paper introduces a relaxed variable splitting method combining thresholding and gradient descent to efficiently learn sparse neural networks with various penalties, ensuring convergence to the true weights with high probability.

Contribution

It proposes a novel optimization approach that guarantees convergence and effective sparsity promotion for learning neural networks with $\, ext{l}_1$, $\, ext{l}_0$, and transformed-$ ext{l}_1$ penalties.

Findings

High-probability convergence to true weights under different penalties.

Numerical experiments validate theoretical convergence and sparsity results.

Trade-offs between accuracy and sparsity are demonstrated.

Abstract

Sparsification of neural networks is one of the effective complexity reduction methods to improve efficiency and generalizability. We consider the problem of learning a one hidden layer convolutional neural network with ReLU activation function via gradient descent under sparsity promoting penalties. It is known that when the input data is Gaussian distributed, no-overlap networks (without penalties) in regression problems with ground truth can be learned in polynomial time at high probability. We propose a relaxed variable splitting method integrating thresholding and gradient descent to overcome the lack of non-smoothness in the loss function. The sparsity in network weight is realized during the optimization (training) process. We prove that under $\ell_1, \ell_0$; and transformed-$\ell_1$ penalties, no-overlap networks can be learned with high probability, and the iterative weights converge to a global limit which is a transformation of the true weight under a novel thresholding operation. Numerical experiments confirm theoretical findings, and compare the accuracy and sparsity trade-off among the penalties.