Optimization with First-Order Surrogate Functions

TL;DR

This paper presents a unified framework for first-order optimization methods using surrogate functions, introduces a new incremental scheme, and demonstrates its effectiveness on large-scale machine learning problems.

Contribution

It unifies various first-order optimization algorithms under a common surrogate-based perspective and proposes a novel incremental scheme that improves performance on large-scale tasks.

Findings

Unified view of first-order methods like proximal gradient and Frank-Wolfe.

New incremental scheme matches or outperforms existing solvers.

Effective on large-scale machine learning problems.

Abstract

In this paper, we study optimization methods consisting of iteratively minimizing surrogates of an objective function. By proposing several algorithmic variants and simple convergence analyses, we make two main contributions. First, we provide a unified viewpoint for several first-order optimization techniques such as accelerated proximal gradient, block coordinate descent, or Frank-Wolfe algorithms. Second, we introduce a new incremental scheme that experimentally matches or outperforms state-of-the-art solvers for large-scale optimization problems typically arising in machine learning.