Sparse Training with Lipschitz Continuous Loss Functions and a Weighted Group L0-norm Constraint

TL;DR

This paper introduces new algorithms for training deep neural networks with structured sparsity constraints, providing the first non-asymptotic convergence guarantees for Lipschitz continuous loss functions under such constraints.

Contribution

It develops zeroth and first-order algorithms with convergence guarantees for constrained Lipschitz functions, specifically addressing weighted group L0-norm constraints in deep learning.

Findings

Algorithms with non-asymptotic convergence guarantees in expectation.

High-probability convergence guarantees for the proposed methods.

Almost sure asymptotic convergence to stationary points.

Abstract

This paper is motivated by structured sparsity for deep neural network training. We study a weighted group L0-norm constraint, and present the projection and normal cone of this set. Using randomized smoothing, we develop zeroth and first-order algorithms for minimizing a Lipschitz continuous function constrained by any closed set which can be projected onto. Non-asymptotic convergence guarantees are proven in expectation for the proposed algorithms for two related convergence criteria which can be considered as approximate stationary points. Two further methods are given using the proposed algorithms: one with non-asymptotic convergence guarantees in high probability, and the other with asymptotic guarantees to a stationary point almost surely. We believe in particular that these are the first such non-asymptotic convergence results for constrained Lipschitz continuous loss functions.