Distribution-Free Confidence Ellipsoids for Ridge Regression with PAC Bounds

TL;DR

This paper extends the Sign-Perturbed Sums (SPS) confidence ellipsoid method to ridge regression, providing PAC bounds that account for regularization effects and weaker excitation conditions, with practical validation through simulations.

Contribution

It introduces a novel extension of the SPS EOA algorithm to ridge regression and derives PAC bounds that explicitly incorporate the regularization parameter.

Findings

Regularization affects the size of confidence regions.

Tighter bounds are achieved under weaker excitation assumptions.

Simulation experiments demonstrate the practical benefits of regularization.

Abstract

Linearly parametrized models are widely used in control and signal processing, with the least-squares (LS) estimate being the archetypical solution. When the input is insufficiently exciting, the LS problem may be unsolvable or numerically unstable. This issue can be resolved through regularization, typically with ridge regression. Although regularized estimators reduce the variance error, it remains important to quantify their estimation uncertainty. A possible approach for linear regression is to construct confidence ellipsoids with the Sign-Perturbed Sums (SPS) ellipsoidal outer approximation (EOA) algorithm. The SPS EOA builds non-asymptotic confidence ellipsoids under the assumption that the noises are independent and symmetric about zero. This paper introduces an extension of the SPS EOA algorithm to ridge regression, and derives probably approximately correct (PAC) upper bounds for the resulting region sizes. Compared with previous analyses, our result explicitly show how the regularization parameter affects the region sizes, and provide tighter bounds under weaker excitation assumptions. Finally, the practical effect of regularization is also demonstrated via simulation experiments.