The smoothed complexity of Frank-Wolfe methods via conditioning of random matrices and polytopes

TL;DR

This paper investigates the smoothed complexity of Frank-Wolfe optimization methods, revealing polynomial bounds for simplexes and exponential bounds for general polytopes, through analysis of random matrix conditioning.

Contribution

It introduces a refined analysis of the condition number's smoothed complexity for polytopes, connecting it to random matrix conditioning and extending to other polytope measures.

Findings

Smoothed condition number is polynomial for simplexes.

Smoothed condition number is exponential for general polytopes.

Results impact understanding of tensor decomposition uniqueness.

Abstract

Frank-Wolfe methods are popular for optimization over a polytope. One of the reasons is because they do not need projection onto the polytope but only linear optimization over it. To understand its complexity, Lacoste-Julien and Jaggi introduced a condition number for polytopes and showed linear convergence for several variations of the method. The actual running time can still be exponential in the worst case (when the condition number is exponential). We study the smoothed complexity of the condition number, namely the condition number of small random perturbations of the input polytope and show that it is polynomial for any simplex and exponential for general polytopes. Our results also apply to other condition measures of polytopes that have been proposed for the analysis of Frank-Wolfe methods: vertex-facet distance (Beck and Shtern) and facial distance (Pe\~na and Rodr\'iguez). Our argument for polytopes is a refinement of an argument that we develop to study the conditioning of random matrices. The basic argument shows that for $c>1$ a $d$-by-$n$ random Gaussian matrix with $n \geq cd$ has a $d$-by-$d$ submatrix with minimum singular value that is exponentially small with high probability. This has consequences on results about the robust uniqueness of tensor decompositions.