Stochastic Bregman Subgradient Methods for Nonsmooth Nonconvex Optimization Problems

TL;DR

This paper introduces stochastic Bregman subgradient methods for nonsmooth, nonconvex optimization, providing convergence analysis, practical inexact solutions, momentum integration, and applications to neural network training.

Contribution

It develops a comprehensive framework for stochastic Bregman subgradient methods, including inexact subproblem solutions, momentum incorporation, and convergence guarantees for nonsmooth nonconvex problems.

Findings

Methods effectively train nonsmooth neural networks

Convergence is established via differential inclusion

Numerical experiments validate approach effectiveness

Abstract

This paper focuses on the problem of minimizing a locally Lipschitz continuous function. Motivated by the effectiveness of Bregman gradient methods in training nonsmooth deep neural networks and the recent progress in stochastic subgradient methods for nonsmooth nonconvex optimization problems \cite{bolte2021conservative,bolte2022subgradient,xiao2023adam}, we investigate the long-term behavior of stochastic Bregman subgradient methods in such context, especially when the objective function lacks Clarke regularity. We begin by exploring a general framework for Bregman-type methods, establishing their convergence by a differential inclusion approach. For practical applications, we develop a stochastic Bregman subgradient method that allows the subproblems to be solved inexactly. Furthermore, we demonstrate how a single timescale momentum can be integrated into the Bregman subgradient method with slight modifications to the momentum update. Additionally, we introduce a Bregman proximal subgradient method for solving composite optimization problems possibly with constraints, whose convergence can be guaranteed based on the general framework. Numerical experiments on training nonsmooth neural networks are conducted to validate the effectiveness of our proposed methods.