Boosting First-Order Methods by Shifting Objective: New Schemes with Faster Worst-Case Rates

TL;DR

This paper introduces a novel methodology for designing accelerated first-order optimization algorithms by shifting the objective function, leading to methods with improved worst-case convergence rates for strongly convex problems.

Contribution

The paper presents a new framework using shifted objectives and an interpolation condition to simplify analysis and develop faster first-order methods for strongly convex optimization.

Findings

Derived accelerated schemes with faster worst-case rates

Simplified analysis due to the interpolation condition

Experimental validation on machine learning tasks

Abstract

We propose a new methodology to design first-order methods for unconstrained strongly convex problems. Specifically, instead of tackling the original objective directly, we construct a shifted objective function that has the same minimizer as the original objective and encodes both the smoothness and strong convexity of the original objective in an interpolation condition. We then propose an algorithmic template for tackling the shifted objective, which can exploit such a condition. Following this template, we derive several new accelerated schemes for problems that are equipped with various first-order oracles and show that the interpolation condition allows us to vastly simplify and tighten the analysis of the derived methods. In particular, all the derived methods have faster worst-case convergence rates than their existing counterparts. Experiments on machine learning tasks are conducted to evaluate the new methods.