Finite Sample Analysis of Distribution-Free Confidence Ellipsoids for Linear Regression

TL;DR

This paper provides finite-sample, distribution-free guarantees for confidence ellipsoids in linear regression using the Sign-Perturbed Sums method, with bounds that decrease optimally and are validated experimentally.

Contribution

It introduces non-asymptotic confidence ellipsoids with strict guarantees under mild assumptions, extending classical methods with convex optimization-based radii computation.

Findings

High probability bounds for SPS ellipsoid sizes are established.

Ellipsoid volumes decrease at the optimal rate with sample size.

Experimental results validate theoretical bounds and practical effectiveness.

Abstract

The least squares (LS) estimate is the archetypical solution of linear regression problems. The asymptotic Gaussianity of the scaled LS error is often used to construct approximate confidence ellipsoids around the LS estimate, however, for finite samples these ellipsoids do not come with strict guarantees, unless some strong assumptions are made on the noise distributions. The paper studies the distribution-free Sign-Perturbed Sums (SPS) ellipsoidal outer approximation (EOA) algorithm which can construct non-asymptotically guaranteed confidence ellipsoids under mild assumptions, such as independent and symmetric noise terms. These ellipsoids have the same center and orientation as the classical asymptotic ellipsoids, only their radii are different, which radii can be computed by convex optimization. Here, we establish high probability non-asymptotic upper bounds for the sizes of SPS outer ellipsoids for linear regression problems and show that the volumes of these ellipsoids decrease at the optimal rate. Finally, the difference between our theoretical bounds and the empirical sizes of the regions are investigated experimentally.