PAC-Bayesian Bounds for Randomized Empirical Risk Minimizers

TL;DR

This paper extends PAC-Bayesian bounds to broader statistical inference problems, providing a framework to control the risk of randomized estimators and aiding in model selection.

Contribution

It generalizes PAC-Bayesian theorems to new inference settings and analyzes randomized estimators near classical ones for risk control.

Findings

Bounded risk deviations for randomized estimators.

Applicable to a wide range of estimation procedures.

Facilitates model selection through risk bounds.

Abstract

The aim of this paper is to generalize the PAC-Bayesian theorems proved by Catoni in the classification setting to more general problems of statistical inference. We show how to control the deviations of the risk of randomized estimators. A particular attention is paid to randomized estimators drawn in a small neighborhood of classical estimators, whose study leads to control the risk of the latter. These results allow to bound the risk of very general estimation procedures, as well as to perform model selection.