Exponential weights in multivariate regression and a low-rankness favoring prior

TL;DR

This paper provides theoretical guarantees for exponential weighting in multivariate regression, showing it can handle dependent noise, unbounded regressors with noise perturbation, and promotes low-rank solutions for matrix parameters.

Contribution

It introduces new risk bounds for exponential weights in multivariate regression, including cases with dependent noise and unbounded regressors, and proposes a low-rank prior for matrix estimation.

Findings

Symmetry condition on noise suffices for sharp risk bounds.

Exponential weights with noise perturbation achieve oracle inequalities.

Low-rank prior leads to optimal estimators for matrix parameters.

Abstract

We establish theoretical guarantees for the expected prediction error of the exponential weighting aggregate in the case of multivariate regression that is when the label vector is multidimensional. We consider the regression model with fixed design and bounded noise. The first new feature uncovered by our guarantees is that it is not necessary to require independence of the observations: a symmetry condition on the noise distribution alone suffices to get a sharp risk bound. This result needs the regression vectors to be bounded. A second curious finding concerns the case of unbounded regression vectors but independent noise. It turns out that applying exponential weights to the label vectors perturbed by a uniform noise leads to an estimator satisfying a sharp oracle inequality. The last contribution is the instantiation of the proposed oracle inequalities to problems in which the unknown parameter is a matrix. We propose a low-rankness favoring prior and show that it leads to an estimator that is optimal under weak assumptions.