Overfitting and Generalizing with (PAC) Bayesian Prediction in Noisy Binary Classification

TL;DR

This paper analyzes a PAC-Bayes learning rule for noisy binary classification, showing how the choice of regularization parameter affects overfitting and generalization, and extending previous work to continuous priors and randomized predictions.

Contribution

It extends PAC-Bayes analysis to continuous priors and randomized predictors, providing a detailed characterization of regularization effects on overfitting and generalization.

Findings

Choosing a large regularization parameter ensures vanishing excess loss.

Over-regularization can lead to underfitting, while under-regularization causes overfitting.

The work generalizes previous discrete prior analysis to continuous priors.

Abstract

We consider a PAC-Bayes type learning rule for binary classification, balancing the training error of a randomized ''posterior'' predictor with its KL divergence to a pre-specified ''prior''. This can be seen as an extension of a modified two-part-code Minimum Description Length (MDL) learning rule, to continuous priors and randomized predictions. With a balancing parameter of $\lambda=1$ this learning rule recovers an (empirical) Bayes posterior and a modified variant recovers the profile posterior, linking with standard Bayesian prediction (up to the treatment of the single-parameter noise level). However, from a risk-minimization prediction perspective, this Bayesian predictor overfits and can lead to non-vanishing excess loss in the agnostic case. Instead a choice of $\lambda \gg 1$, which can be seen as using a sample-size-dependent-prior, ensures uniformly vanishing excess loss even in the agnostic case. We precisely characterize the effect of under-regularizing (and over-regularizing) as a function of the balance parameter $\lambda$, understanding the regimes in which this under-regularization is tempered or catastrophic. This work extends previous work by Zhu and Srebro [2025] that considered only discrete priors to PAC Bayes type learning rules and, through their rigorous Bayesian interpretation, to Bayesian prediction more generally.