Randomized Block Frank-Wolfe for Convergent Large-Scale Learning

TL;DR

This paper introduces a flexible randomized block Frank-Wolfe algorithm with new step-size strategies that ensure convergence and feasibility, applicable to large-scale convex and nonconvex learning problems, with proven theoretical guarantees and practical benefits.

Contribution

It develops novel step-size rules for RB-FW that allow flexible block updates while ensuring convergence and feasibility, extending analysis to nonconvex objectives.

Findings

Convergence rates established for primal and duality gap measures.

Feasibility of iterates guaranteed with new step-size sequences.

Performance improvements demonstrated in electrical vehicle charging and SVM applications.

Abstract

Owing to their low-complexity iterations, Frank-Wolfe (FW) solvers are well suited for various large-scale learning tasks. When block-separable constraints are present, randomized block FW (RB-FW) has been shown to further reduce complexity by updating only a fraction of coordinate blocks per iteration. To circumvent the limitations of existing methods, the present work develops step sizes for RB-FW that enable a flexible selection of the number of blocks to update per iteration while ensuring convergence and feasibility of the iterates. To this end, convergence rates of RB-FW are established through computational bounds on a primal sub-optimality measure and on the duality gap. The novel bounds extend the existing convergence analysis, which only applies to a step-size sequence that does not generally lead to feasible iterates. Furthermore, two classes of step-size sequences that guarantee feasibility of the iterates are also proposed to enhance flexibility in choosing decay rates. The novel convergence results are markedly broadened to encompass also nonconvex objectives, and further assert that RB-FW with exact line-search reaches a stationary point at rate $\mathcal{O}(1/\sqrt{t})$. Performance of RB-FW with different step sizes and number of blocks is demonstrated in two applications, namely charging of electrical vehicles and structural support vector machines. Extensive simulated tests demonstrate the performance improvement of RB-FW relative to existing randomized single-block FW methods.