Suboptimality of Constrained Least Squares and Improvements via Non-Linear Predictors

TL;DR

This paper demonstrates that constrained least squares can be suboptimal in bounded settings and shows that non-linear predictors can achieve optimal rates without distributional assumptions.

Contribution

It proves the suboptimality of constrained least squares in bounded distributions and highlights the effectiveness of non-linear predictors for optimal risk rates.

Findings

Constrained least squares does not always attain the $O(d/n)$ rate.

Constructed bounded distribution causes $ ilde{ ext{Omega}}(d^{3/2}/n)$ excess risk.

Non-linear predictors can achieve the optimal $O(d/n)$ rate without distribution assumptions.

Abstract

We study the problem of predicting as well as the best linear predictor in a bounded Euclidean ball with respect to the squared loss. When only boundedness of the data generating distribution is assumed, we establish that the least squares estimator constrained to a bounded Euclidean ball does not attain the classical $O(d/n)$ excess risk rate, where $d$ is the dimension of the covariates and $n$ is the number of samples. In particular, we construct a bounded distribution such that the constrained least squares estimator incurs an excess risk of order $\Omega(d^{3/2}/n)$ hence refuting a recent conjecture of Ohad Shamir [JMLR 2015]. In contrast, we observe that non-linear predictors can achieve the optimal rate $O(d/n)$ with no assumptions on the distribution of the covariates. We discuss additional distributional assumptions sufficient to guarantee an $O(d/n)$ excess risk rate for the least squares estimator. Among them are certain moment equivalence assumptions often used in the robust statistics literature. While such assumptions are central in the analysis of unbounded and heavy-tailed settings, our work indicates that in some cases, they also rule out unfavorable bounded distributions.