A New Lineserach for Accelerated Composite Minimization

TL;DR

This paper introduces a novel linesearch stepsize rule based on function evaluations for convex optimization, providing convergence guarantees and improved performance across various problem classes.

Contribution

The paper proposes a new adaptive linesearch stepsize method that does not rely on smoothness constants, with theoretical convergence guarantees and extensive empirical benchmarking.

Findings

Demonstrates improved convergence over existing methods

Effective across smooth, composite, and non-convex problems

Benchmarked on diverse applications like logistic regression and portfolio optimization

Abstract

The choice of the stepsize in first-order convex optimization is typically based on the smoothness constant and plays a crucial role in the performance of algorithms. Recently, there has been a resurgent interest in introducing adaptive stepsizes that do not explicitly depend on smooth constant. In this paper, we propose a novel linesearch stepsize rule based on function evaluations (i.e., zero-order information) that enjoys provable convergence guarantees for both accelerated and non-accelerated gradient descent. We further discuss the similarities and differences between the proposed stepsize regimes and the existing stepsize rules (including Polyak and Armijo). We numerically benchmark the performance of our proposed algorithms against state-of-the-art methods across three major problems classes of (1) smooth minimization (logistic regression, quadratic programs, log-sum-exponential, and smooth max-cut relaxation) (2) composite minimization ($\ell_1$-regularized least-squares, $\ell_1$-constrained least-squares, and $\ell_1$-regularized logistic regression), and (3) non-convex minimization (cubic minimization). These classes include a wide range of operations research and management applications such as portfolio optimization, discrete choice models, sparse classification and feature selections, high-order optimization and trust-region subproblems.