Accelerated nonlinear primal-dual hybrid gradient methods with applications to supervised machine learning

TL;DR

This paper introduces accelerated nonlinear primal-dual hybrid gradient methods that achieve optimal convergence rates with simple, efficiently computable stepsize parameters, making them highly effective for large-scale machine learning problems.

Contribution

The paper proposes accelerated nonlinear PDHG methods with simple stepsize rules, providing rigorous convergence analysis and demonstrating superior performance in machine learning applications.

Findings

Methods are faster than competing algorithms in experiments.

Achieve optimal convergence with simple stepsize parameters.

Applicable to infinite-dimensional Banach spaces.

Abstract

The linear primal-dual hybrid gradient (PDHG) method is a first-order method that splits convex optimization problems with saddle-point structure into smaller subproblems. Unlike those obtained in most splitting methods, these subproblems can generally be solved efficiently because they involve simple operations such as matrix-vector multiplications or proximal mappings that are fast to evaluate numerically. This advantage comes at the price that the linear PDHG method requires precise stepsize parameters for the problem at hand to achieve an optimal convergence rate. Unfortunately, these stepsize parameters are often prohibitively expensive to compute for large-scale optimization problems, such as those in machine learning. This issue makes the otherwise simple linear PDHG method unsuitable for such problems, and it is also shared by most first-order optimization methods as well. To address this issue, we introduce accelerated nonlinear PDHG methods that achieve an optimal convergence rate with stepsize parameters that are simple and efficient to compute. We prove rigorous convergence results, including results for strongly convex or smooth problems posed on infinite-dimensional reflexive Banach spaces. We illustrate the efficiency of our methods on $\ell_{1}$-constrained logistic regression and entropy-regularized matrix games. Our numerical experiments show that the nonlinear PDHG methods are considerably faster than competing methods.