Coresets for Multiple $\ell_p$ Regression

TL;DR

This paper introduces nearly size-independent coresets for multiple $ ext{ell}_p$ regression, enabling efficient approximation of regression objectives and subspace problems for various $p$ values, with tight bounds and practical applications.

Contribution

It constructs nearly dimension-free coresets for multiple $ ext{ell}_p$ regression, improving prior bounds and providing tight theoretical guarantees for various $p$ and applications.

Findings

Coresets of size $ ilde O( ext{poly}(1/ extvarepsilon), d)$ for $p<2$ and $p>2$ are nearly tight.

Sample complexity for $ ext{ell}_p$ Euclidean power means is established as tight for different $p$.

Existence of small subsets of rows spanning nearly optimal $ ext{ell}_p$ subspaces for $1<p<2$ is proven.

Abstract

A coreset of a dataset with $n$ examples and $d$ features is a weighted subset of examples that is sufficient for solving downstream data analytic tasks. Nearly optimal constructions of coresets for least squares and $\ell_p$ linear regression with a single response are known in prior work. However, for multiple $\ell_p$ regression where there can be $m$ responses, there are no known constructions with size sublinear in $m$. In this work, we construct coresets of size $\tilde O(\varepsilon^{-2}d)$ for $p<2$ and $\tilde O(\varepsilon^{-p}d^{p/2})$ for $p>2$ independently of $m$ (i.e., dimension-free) that approximate the multiple $\ell_p$ regression objective at every point in the domain up to $(1\pm\varepsilon)$ relative error. If we only need to preserve the minimizer subject to a subspace constraint, we improve these bounds by an $\varepsilon$ factor for all $p>1$. All of our bounds are nearly tight. We give two application of our results. First, we settle the number of uniform samples needed to approximate $\ell_p$ Euclidean power means up to a $(1+\varepsilon)$ factor, showing that $\tilde\Theta(\varepsilon^{-2})$ samples for $p = 1$, $\tilde\Theta(\varepsilon^{-1})$ samples for $1 < p < 2$, and $\tilde\Theta(\varepsilon^{1-p})$ samples for $p>2$ is tight, answering a question of Cohen-Addad, Saulpic, and Schwiegelshohn. Second, we show that for $1<p<2$, every matrix has a subset of $\tilde O(\varepsilon^{-1}k)$ rows which spans a $(1+\varepsilon)$-approximately optimal $k$-dimensional subspace for $\ell_p$ subspace approximation, which is also nearly optimal.