Core-Elements for Large-Scale Least Squares Estimation

TL;DR

This paper introduces a novel element-wise subset selection method called core-elements for large-scale least squares estimation, improving efficiency and robustness, especially with sparse data, and providing theoretical guarantees and superior empirical performance.

Contribution

The paper develops a deterministic core-elements algorithm for efficient, unbiased least squares estimation with theoretical bounds, addressing limitations of existing coreset methods in sparse settings.

Findings

Efficient $O(nnz(X)+rp^2)$ computational cost for large-scale data.

The estimator is unbiased and minimizes an upper bound of variance.

Demonstrates superior performance over existing methods in synthetic and real datasets.

Abstract

The coresets approach, also called subsampling or subset selection, aims to select a subsample as a surrogate for the observed sample and has found extensive applications in large-scale data analysis. Existing coresets methods construct the subsample using a subset of rows from the predictor matrix. Such methods can be significantly inefficient when the predictor matrix is sparse or numerically sparse. To overcome this limitation, we develop a novel element-wise subset selection approach, called core-elements, for large-scale least squares estimation. We provide a deterministic algorithm to construct the core-elements estimator, only requiring an $O(\mathrm{nnz}(X)+rp^2)$ computational cost, where $X$ is an $n\times p$ predictor matrix, $r$ is the number of elements selected from each column of $X$, and $\mathrm{nnz}(\cdot)$ denotes the number of non-zero elements. Theoretically, we show that the proposed estimator is unbiased and approximately minimizes an upper bound of the estimation variance. We also provide an approximation guarantee by deriving a coresets-like finite sample bound for the proposed estimator. To handle potential outliers in the data, we further combine core-elements with the median-of-means procedure, resulting in an efficient and robust estimator with theoretical consistency guarantees. Numerical studies on various synthetic and real-world datasets demonstrate the proposed method's superior performance compared to mainstream competitors.