Bound to Disagree: Generalization Bounds via Certifiable Surrogates

TL;DR

This paper introduces disagreement-based certificates for deep learning models that provide tight, computable generalization bounds using surrogate models, applicable without altering the original training process.

Contribution

It proposes a novel disagreement-based approach to certify generalization bounds via surrogate models, applicable across multiple frameworks without modifying target models.

Findings

Certificates are empirically tight.

Versatile approach using different surrogate frameworks.

No modification needed for target models or training procedures.

Abstract

Generalization bounds for deep learning models are typically vacuous, not computable or restricted to specific model classes. In this paper, we tackle these issues by providing new disagreement-based certificates for the gap between the true risk of any two predictors. We then bound the true risk of the predictor of interest via a surrogate model that enjoys tight generalization guarantees, and evaluating our disagreement bound on an unlabeled dataset. We empirically demonstrate the tightness of the obtained certificates and showcase the versatility of the approach by training surrogate models leveraging three different frameworks: sample compression, model compression and PAC-Bayes theory. Importantly, such guarantees are achieved without modifying the target model, nor adapting the training procedure to the generalization framework.