Preconditioned Primal-Dual Gradient Methods for Nonconvex Composite and Finite-Sum Optimization

TL;DR

This paper introduces a preconditioned primal-dual gradient method for nonconvex composite optimization that avoids proximal calculations, proves convergence under mild conditions, and extends to stochastic finite-sum problems with variance reduction.

Contribution

It presents a novel primal-dual algorithm based on conjugate duality for nonconvex problems, with convergence guarantees and a stochastic variant for finite-sum optimization.

Findings

The algorithm converges to critical points under mild conditions.

Global convergence and rates are established using Kurdyka-ojasiewicz property.

Numerical results show the effectiveness of the proposed methods.

Abstract

In this paper, we first introduce a preconditioned primal-dual gradient algorithm based on conjugate duality theory. This algorithm is designed to solve composite optimization problem whose objective function consists of two summands: a continuously differentiable nonconvex function and the composition of a nonsmooth nonconvex function with a linear operator. In contrast to existing nonconvex primal-dual algorithms, our proposed algorithm, through the utilization of conjugate duality, does not require the calculation of proximal mapping of nonconvex functions. Under mild conditions, we prove that any cluster point of the generated sequence is a critical point of the composite optimization problem. In the context of Kurdyka-\L{}ojasiewicz property, we establish global convergence and convergence rates for the iterates. Secondly, for nonconvex finite-sum optimization, we propose a stochastic algorithm that combines the preconditioned primal-dual gradient algorithm with a class of variance reduced stochastic gradient estimators. Almost sure global convergence and expected convergence rates are derived relying on the Kurdyka-\L{}ojasiewicz inequality. Finally, some preliminary numerical results are presented to demonstrate the effectiveness of the proposed algorithms.