Still no free lunches: the price to pay for tighter PAC-Bayes bounds

TL;DR

This paper investigates the fundamental limits of obtaining tight PAC-Bayes bounds in robust settings, especially for models with minimal assumptions, highlighting the trade-offs between bound tightness and model complexity.

Contribution

It characterizes the inherent trade-offs and limits in deriving tight PAC-Bayes bounds for cheap models under robust assumptions.

Findings

Tight PAC-Bayes bounds require paying a certain cost in model assumptions.

There are fundamental limits to how tight bounds can be for models with minimal assumptions.

Robust statistical procedures can improve bounds but cannot eliminate the inherent trade-offs.

Abstract

"No free lunch" results state the impossibility of obtaining meaningful bounds on the error of a learning algorithm without prior assumptions and modelling. Some models are expensive (strong assumptions, such as as subgaussian tails), others are cheap (simply finite variance). As it is well known, the more you pay, the more you get: in other words, the most expensive models yield the more interesting bounds. Recent advances in robust statistics have investigated procedures to obtain tight bounds while keeping the cost minimal. The present paper explores and exhibits what the limits are for obtaining tight PAC-Bayes bounds in a robust setting for cheap models, addressing the question: is PAC-Bayes good value for money?