List Decodable Learning via Sum of Squares

TL;DR

This paper introduces a new framework using the Sum-of-Squares SDP hierarchy for list-decodable learning, providing algorithms for robust linear regression and mean estimation in the presence of many outliers.

Contribution

It develops the first list-decodable linear regression algorithm and extends Sum-of-Squares techniques to robust statistical estimation problems.

Findings

First list-decodable linear regression algorithm.

Algorithms work for distributions with Sum-of-Squares certifiable concentration.

Achieves guarantees comparable to existing robust estimation methods.

Abstract

In the list-decodable learning setup, an overwhelming majority (say a $1-\beta$-fraction) of the input data consists of outliers and the goal of an algorithm is to output a small list $\mathcal{L}$ of hypotheses such that one of them agrees with inliers. We develop a framework for list-decodable learning via the Sum-of-Squares SDP hierarchy and demonstrate it on two basic statistical estimation problems {\it Linear regression:} Suppose we are given labelled examples $\{(X_i,y_i)\}_{i \in [N]}$ containing a subset $S$ of $\beta N$ {\it inliers} $\{X_i \}_{i \in S}$ that are drawn i.i.d. from standard Gaussian distribution $N(0,I)$ in $\mathbb{R}^d$, where the corresponding labels $y_i$ are well-approximated by a linear function $\ell$. We devise an algorithm that outputs a list $\mathcal{L}$ of linear functions such that there exists some $\hat{\ell} \in \mathcal{L}$ that is close to $\ell$. This yields the first algorithm for linear regression in a list-decodable setting. Our results hold for any distribution of examples whose concentration and anticoncentration can be certified by Sum-of-Squares proofs. {\it Mean Estimation:} Given data points $\{X_i\}_{i \in [N]}$ containing a subset $S$ of $\beta N$ {\it inliers} $\{X_i \}_{i \in S}$ that are drawn i.i.d. from a Gaussian distribution $N(\mu,I)$ in $\mathbb{R}^d$, we devise an algorithm that generates a list $\mathcal{L}$ of means such that there exists $\hat{\mu} \in \mathcal{L}$ close to $\mu$. The recovery guarantees of the algorithm are analogous to the existing algorithms for the problem by Diakonikolas \etal and Kothari \etal. In an independent and concurrent work, Karmalkar \etal \cite{KlivansKS19} also obtain an algorithm for list-decodable linear regression using the Sum-of-Squares SDP hierarchy.