On the Rashomon ratio of infinite hypothesis sets

TL;DR

This paper extends the concept of the Rashomon ratio to infinite classifier families, demonstrating its significance in model selection and generalization, with practical estimation methods and illustrative examples involving neural networks and affine classifiers.

Contribution

It generalizes the Rashomon ratio to infinite hypothesis sets and provides methods to estimate it with guarantees, including examples with neural networks and affine classifiers.

Findings

A large Rashomon ratio indicates robust model selection potential.

Estimation methods for the Rashomon ratio are effective with finite samples.

Examples show the ratio can be large in neural network and affine classifier families.

Abstract

Given a classification problem and a family of classifiers, the Rashomon ratio measures the proportion of classifiers that yield less than a given loss. Previous work has explored the advantage of a large Rashomon ratio in the case of a finite family of classifiers. Here we consider the more general case of an infinite family. We show that a large Rashomon ratio guarantees that choosing the classifier with the best empirical accuracy among a random subset of the family, which is likely to improve generalizability, will not increase the empirical loss too much. We quantify the Rashomon ratio in two examples involving infinite classifier families in order to illustrate situations in which it is large. In the first example, we estimate the Rashomon ratio of the classification of normally distributed classes using an affine classifier. In the second, we obtain a lower bound for the Rashomon ratio of a classification problem with a modified Gram matrix when the classifier family consists of two-layer ReLU neural networks. In general, we show that the Rashomon ratio can be estimated using a training dataset along with random samples from the classifier family and we provide guarantees that such an estimation is close to the true value of the Rashomon ratio.