Accurate Coresets for Latent Variable Models and Regularized Regression

TL;DR

This paper introduces a unified framework for constructing accurate coresets applicable to a broad class of machine learning models, including latent variable models and regularized regression, with provably small sizes and extensive experimental validation.

Contribution

It presents a general coreset construction framework and algorithms for latent variable models and regularized regression, improving coreset sizes and applicability over prior limited models.

Findings

Coreset size for latent variable models is polynomial in the number of latent variables.

Coreset size for regularized regression is smaller than d^p, accounting for regularization effects.

Experimental results confirm theoretical size bounds and effectiveness on real datasets.

Abstract

Accurate coresets are a weighted subset of the original dataset, ensuring a model trained on the accurate coreset maintains the same level of accuracy as a model trained on the full dataset. Primarily, these coresets have been studied for a limited range of machine learning models. In this paper, we introduce a unified framework for constructing accurate coresets. Using this framework, we present accurate coreset construction algorithms for general problems, including a wide range of latent variable model problems and $\ell_p$-regularized $\ell_p$-regression. For latent variable models, our coreset size is $O\left(\mathrm{poly}(k)\right)$, where $k$ is the number of latent variables. For $\ell_p$-regularized $\ell_p$-regression, our algorithm captures the reduction of model complexity due to regularization, resulting in a coreset whose size is always smaller than $d^{p}$ for a regularization parameter $\lambda > 0$. Here, $d$ is the dimension of the input points. This inherently improves the size of the accurate coreset for ridge regression. We substantiate our theoretical findings with extensive experimental evaluations on real datasets.