Optimal First-Order Algorithms as a Function of Inequalities

TL;DR

This paper introduces a new methodology for designing optimal first-order algorithms by restricting the set of inequalities used in convergence analysis, leading to state-of-the-art methods with specific desired properties.

Contribution

The paper proposes a novel algorithm design approach based on inequality restrictions, enabling the creation of optimized algorithms with targeted characteristics.

Findings

Developed a methodology for optimal algorithm design using inequality restrictions

Designed new accelerated gradient methods with randomized coordinate updates

Achieved state-of-the-art performance with these methods

Abstract

In this work, we present a novel algorithm design methodology that finds the optimal algorithm as a function of inequalities. Specifically, we restrict convergence analyses of algorithms to use a prespecified subset of inequalities, rather than utilizing all true inequalities, and find the optimal algorithm subject to this restriction. This methodology allows us to design algorithms with certain desired characteristics. As concrete demonstrations of this methodology, we find new state-of-the-art accelerated first-order gradient methods using randomized coordinate updates and backtracking line searches.