Adaptive Catalyst for Smooth Convex Optimization

TL;DR

This paper introduces a flexible framework that accelerates various smooth convex optimization algorithms, featuring adaptive regularization and improved convergence without logarithmic factors, enhancing efficiency and applicability.

Contribution

It proposes a novel adaptive Catalyst framework that accelerates non-accelerated algorithms with verifiable stopping criteria and adaptive regularization tuning.

Findings

Applicable to various inner algorithms like Steepest Descent and Coordinate Descent

Achieves acceleration without the logarithmic factor in non-adaptive cases

Provides a practical, flexible approach for smooth convex optimization

Abstract

In this paper, we present a generic framework that allows accelerating almost arbitrary non-accelerated deterministic and randomized algorithms for smooth convex optimization problems. The main approach of our envelope is the same as in Catalyst (Lin et al., 2015): an accelerated proximal outer gradient method, which is used as an envelope for a non-accelerated inner method for the $\ell_2$ regularized auxiliary problem. Our algorithm has two key differences: 1) easily verifiable stopping criteria for inner algorithm; 2) the regularization parameter can be tunned along the way. As a result, the main contribution of our work is a new framework that applies to adaptive inner algorithms: Steepest Descent, Adaptive Coordinate Descent, Alternating Minimization. Moreover, in the non-adaptive case, our approach allows obtaining Catalyst without a logarithmic factor, which appears in the standard Catalyst (Lin et al., 2015, 2018).