On Fundamental Proof Structures in First-Order Optimization

TL;DR

This paper explores the fundamental proof structures behind first-order optimization methods, aiming to systematize and automate the process of deriving convergence guarantees and understanding their underlying principles.

Contribution

It presents a systematic approach to constructing convergence proofs for first-order methods, highlighting their core proof structures and facilitating automation.

Findings

Identifies common proof patterns in first-order optimization

Provides tools for systematic proof construction

Enhances understanding of convergence guarantees

Abstract

First-order optimization methods have attracted a lot of attention due to their practical success in many applications, including in machine learning. Obtaining convergence guarantees and worst-case performance certificates for first-order methods have become crucial for understanding ingredients underlying efficient methods and for developing new ones. However, obtaining, verifying, and proving such guarantees is often a tedious task. Therefore, a few approaches were proposed for rendering this task more systematic, and even partially automated. In addition to helping researchers finding convergence proofs, these tools provide insights on the general structures of such proofs. We aim at presenting those structures, showing how to build convergence guarantees for first-order optimization methods.