Nonsmooth Frank-Wolfe using Uniform Affine Approximations

TL;DR

This paper introduces a modified Frank-Wolfe algorithm that effectively handles nonsmooth functions by optimizing affine approximations over neighborhoods, improving performance in nonsmooth low-rank matrix estimation tasks.

Contribution

It proposes a novel Frank-Wolfe variant that extends applicability to nonsmooth functions through uniform affine approximations, with theoretical analysis and practical benefits.

Findings

Overcomes limitations of existing FW in nonsmooth settings

Provides theoretical guarantees for the proposed method

Demonstrates improved performance in low-rank matrix estimation

Abstract

Frank-Wolfe methods (FW) have gained significant interest in the machine learning community due to its ability to efficiently solve large problems that admit a sparse structure (e.g. sparse vectors and low-rank matrices). However the performance of the existing FW method hinges on the quality of the linear approximation. This typically restricts FW to smooth functions for which the approximation quality, indicated by a global curvature measure, is reasonably good. In this paper, we propose a modified FW algorithm amenable to nonsmooth functions by optimizing for approximation quality over all affine approximations given a neighborhood of interest. We analyze theoretical properties of the proposed algorithm and demonstrate that it overcomes many issues associated with existing methods in the context of nonsmooth low-rank matrix estimation.