PAC-Bayesian aggregation of affine estimators

TL;DR

This paper introduces a PAC-Bayesian aggregation method for affine estimators in fixed design regression, achieving probabilistic oracle inequalities through overpenalization under sub-Gaussian noise.

Contribution

It presents a novel aggregation approach with probabilistic guarantees, extending PAC-Bayesian techniques to affine estimators with dependent sub-Gaussian noise.

Findings

Achieves oracle inequality in probability for aggregated estimators.

Applicable to a broad class of affine estimators under dependent sub-Gaussian noise.

Uses overpenalization to improve probabilistic bounds.

Abstract

Aggregating estimators using exponential weights depending on their risk appears optimal in expectation but not in probability. We use here a slight overpenalization to obtain oracle inequality in probability for such an explicit aggregation procedure. We focus on the fixed design regression framework and the aggregation of affine estimators and obtain results for a large family of affine estimators under a non necessarily independent sub-Gaussian noise assumptions.