Generalized Self-Concordant Analysis of Frank-Wolfe algorithms

TL;DR

This paper develops new Frank-Wolfe algorithms with proven convergence guarantees for optimizing generalized self-concordant functions, which are common in machine learning applications but lack existing theoretical analysis.

Contribution

It introduces the first provably convergent FW algorithms tailored for GSC functions, including methods with linear convergence under certain conditions.

Findings

Established O(1/k) convergence rate for FW on GSC functions

Developed a FW variant with linear convergence rate

Extended FW applicability to functions with unbounded curvature

Abstract

Projection-free optimization via different variants of the Frank-Wolfe (FW) method has become one of the cornerstones in large scale optimization for machine learning and computational statistics. Numerous applications within these fields involve the minimization of functions with self-concordance like properties. Such generalized self-concordant (GSC) functions do not necessarily feature a Lipschitz continuous gradient, nor are they strongly convex. Indeed, in a number of applications, e.g. inverse covariance estimation or distance-weighted discrimination problems in support vector machines, the loss is given by a GSC function having unbounded curvature, implying absence of theoretical guarantees for the existing FW methods. This paper closes this apparent gap in the literature by developing provably convergent FW algorithms with standard O(1/k) convergence rate guarantees. If the problem formulation allows the efficient construction of a local linear minimization oracle, we develop a FW method with linear convergence rate.