Progress in Self-Certified Neural Networks

TL;DR

This paper evaluates progress in self-certified neural networks trained with PAC-Bayes bounds, demonstrating their effectiveness especially in small data regimes and their competitive risk certification compared to traditional test set bounds.

Contribution

It empirically compares classical test set bounds with PAC-Bayes bounds on probabilistic neural networks, highlighting advantages in small data scenarios and the competitiveness of PAC-Bayes certificates.

Findings

PAC-Bayes bounds are close to test set errors on multiple datasets.

Self-certified strategies outperform test set bounds in data-starved regimes.

Probabilistic neural networks with PAC-Bayes bounds provide competitive risk certificates.

Abstract

A learning method is self-certified if it uses all available data to simultaneously learn a predictor and certify its quality with a tight statistical certificate that is valid on unseen data. Recent work has shown that neural network models trained by optimising PAC-Bayes bounds lead not only to accurate predictors, but also to tight risk certificates, bearing promise towards achieving self-certified learning. In this context, learning and certification strategies based on PAC-Bayes bounds are especially attractive due to their ability to leverage all data to learn a posterior and simultaneously certify its risk with a tight numerical certificate. In this paper, we assess the progress towards self-certification in probabilistic neural networks learnt by PAC-Bayes inspired objectives. We empirically compare (on 4 classification datasets) classical test set bounds for deterministic predictors and a PAC-Bayes bound for randomised self-certified predictors. We first show that both of these generalisation bounds are not too far from out-of-sample test set errors. We then show that in data starvation regimes, holding out data for the test set bounds adversely affects generalisation performance, while self-certified strategies based on PAC-Bayes bounds do not suffer from this drawback, proving that they might be a suitable choice for the small data regime. We also find that probabilistic neural networks learnt by PAC-Bayes inspired objectives lead to certificates that can be surprisingly competitive with commonly used test set bounds.