Improved PAC-Bayesian Bounds for Linear Regression

TL;DR

This paper presents tighter PAC-Bayesian error bounds for linear regression that also apply to dependent data like time series, enhancing theoretical guarantees for a broader class of models.

Contribution

It introduces improved, tighter PAC-Bayesian bounds for linear regression that are valid for dependent data, including time series, extending previous independent-sample results.

Findings

Bound converges to true generalization loss with optimal temperature.

Applicable to dependent data such as ARX time series models.

Provides theoretical guarantees for non-i.i.d. training data.

Abstract

In this paper, we improve the PAC-Bayesian error bound for linear regression derived in Germain et al. [10]. The improvements are twofold. First, the proposed error bound is tighter, and converges to the generalization loss with a well-chosen temperature parameter. Second, the error bound also holds for training data that are not independently sampled. In particular, the error bound applies to certain time series generated by well-known classes of dynamical models, such as ARX models.