On Margins and Derandomisation in PAC-Bayes

TL;DR

This paper introduces a general method for derandomising PAC-Bayesian bounds using margins, applicable to various classifiers including neural networks and SVMs, by leveraging concentration properties of predictions.

Contribution

It provides a unified approach to derive margin bounds for diverse classifiers and extends to partially-derandomised predictors with weaker concentration.

Findings

Margin bounds for linear classifiers, neural networks, and deep ReLU networks.

A new derandomisation technique based on prediction concentration.

Extension of bounds to partially-derandomised predictors.

Abstract

We give a general recipe for derandomising PAC-Bayesian bounds using margins, with the critical ingredient being that our randomised predictions concentrate around some value. The tools we develop straightforwardly lead to margin bounds for various classifiers, including linear prediction -- a class that includes boosting and the support vector machine -- single-hidden-layer neural networks with an unusual \(\erf\) activation function, and deep ReLU networks. Further, we extend to partially-derandomised predictors where only some of the randomness is removed, letting us extend bounds to cases where the concentration properties of our predictors are otherwise poor.