GRASP: A Goodness-of-Fit Test for Classification Learning

TL;DR

This paper introduces GRASP, a nonparametric goodness-of-fit test for classifiers that assesses how well a model fits the true label distribution without relying on parametric assumptions, applicable in finite samples.

Contribution

The paper proposes a novel, distribution-free goodness-of-fit test called GRASP for binary classifiers, including a model-X version that leverages known feature distributions for improved power.

Findings

GRASP effectively tests classifier fit in finite samples.

Model-X GRASP outperforms in settings with known feature distributions.

Extensive experiments validate the test's robustness and accuracy.

Abstract

Performance of classifiers is often measured in terms of average accuracy on test data. Despite being a standard measure, average accuracy fails in characterizing the fit of the model to the underlying conditional law of labels given the features vector ($Y|X$), e.g. due to model misspecification, over fitting, and high-dimensionality. In this paper, we consider the fundamental problem of assessing the goodness-of-fit for a general binary classifier. Our framework does not make any parametric assumption on the conditional law $Y|X$, and treats that as a black box oracle model which can be accessed only through queries. We formulate the goodness-of-fit assessment problem as a tolerance hypothesis testing of the form \[ H_0: \mathbb{E}\Big[D_f\Big({\sf Bern}(\eta(X))\|{\sf Bern}(\hat{\eta}(X))\Big)\Big]\leq \tau\,, \] where $D_f$ represents an $f$-divergence function, and $\eta(x)$, $\hat{\eta}(x)$ respectively denote the true and an estimate likelihood for a feature vector $x$ admitting a positive label. We propose a novel test, called \grasp for testing $H_0$, which works in finite sample settings, no matter the features (distribution-free). We also propose model-X \grasp designed for model-X settings where the joint distribution of the features vector is known. Model-X \grasp uses this distributional information to achieve better power. We evaluate the performance of our tests through extensive numerical experiments.