Agnostic Sample Compression Schemes for Regression

TL;DR

This paper introduces new sample compression schemes for agnostic regression with various $\, ext{losses}$, demonstrating their size bounds and limitations, especially for linear models and specific $\, ext{loss functions}$.

Contribution

It constructs the first positive approximate compression schemes for agnostic regression with $\, ext{losses}$ in [1,∞], including linear regression and specific $\, ext{loss}$ cases, and establishes limitations for others.

Findings

Approximate compression of size linear in dimension for linear regression.

Exact compression schemes of size linear in dimension for $\, ext{losses}$ }$\, ext{like}\, ext{L}_1$ and $\, ext{L}_ ext{infinity}$.

Non-existence of bounded size exact schemes for certain $\, ext{losses}$, refining previous results.

Abstract

We obtain the first positive results for bounded sample compression in the agnostic regression setting with the $\ell_p$ loss, where $p\in [1,\infty]$. We construct a generic approximate sample compression scheme for real-valued function classes exhibiting exponential size in the fat-shattering dimension but independent of the sample size. Notably, for linear regression, an approximate compression of size linear in the dimension is constructed. Moreover, for $\ell_1$ and $\ell_\infty$ losses, we can even exhibit an efficient exact sample compression scheme of size linear in the dimension. We further show that for every other $\ell_p$ loss, $p\in (1,\infty)$, there does not exist an exact agnostic compression scheme of bounded size. This refines and generalizes a negative result of David, Moran, and Yehudayoff for the $\ell_2$ loss. We close by posing general open questions: for agnostic regression with $\ell_1$ loss, does every function class admits an exact compression scheme of size equal to its pseudo-dimension? For the $\ell_2$ loss, does every function class admit an approximate compression scheme of polynomial size in the fat-shattering dimension? These questions generalize Warmuth's classic sample compression conjecture for realizable-case classification.