Robust Structured Statistical Estimation via Conditional Gradient Type Methods

TL;DR

This paper introduces robust Conditional Gradient methods that are resistant to data corruption and heavy-tailed distributions, with improved sample complexity for high-dimensional sparse problems.

Contribution

It develops a robust, projection-free optimization framework using Pairwise CG methods and introduces the RASC condition for efficient high-dimensional sparse estimation.

Findings

Methods are stable and do not accumulate error.

Guarantees robustness to data corruption and heavy tails.

Achieves linear convergence with sample complexity depending on sparsity.

Abstract

Structured statistical estimation problems are often solved by Conditional Gradient (CG) type methods to avoid the computationally expensive projection operation. However, the existing CG type methods are not robust to data corruption. To address this, we propose to robustify CG type methods against Huber's corruption model and heavy-tailed data. First, we show that the two Pairwise CG methods are stable, i.e., do not accumulate error. Combined with robust mean gradient estimation techniques, we can therefore guarantee robustness to a wide class of problems, but now in a projection-free algorithmic framework. Next, we consider high dimensional problems. Robust mean estimation based approaches may have an unacceptably high sample complexity. When the constraint set is a $\ell_0$ norm ball, Iterative-Hard-Thresholding-based methods have been developed recently. Yet extension is non-trivial even for general sets with $O(d)$ extreme points. For setting where the feasible set has $O(\text{poly}(d))$ extreme points, we develop a novel robustness method, based on a new condition we call the Robust Atom Selection Condition (RASC). When RASC is satisfied, our method converges linearly with a corresponding statistical error, with sample complexity that scales correctly in the sparsity of the problem, rather than the ambient dimension as would be required by any approach based on robust mean estimation.