Zipper: Addressing degeneracy in algorithm-agnostic inference

TL;DR

The paper introduces Zipper, a novel method that addresses degeneracy in algorithm-agnostic goodness-of-fit tests by overlapping data splits, ensuring valid size control and improved power.

Contribution

Zipper provides a simple, effective device for non-degenerate, asymptotically normal test statistics in model-agnostic goodness-of-fit testing, enhancing power and validity.

Findings

Zipper guarantees asymptotic normality under the null hypothesis.

The method maintains valid size control across various settings.

Finite-sample experiments show strong performance of Zipper.

Abstract

The widespread use of black box prediction methods has sparked an increasing interest in algorithm/model-agnostic approaches for quantifying goodness-of-fit, with direct ties to specification testing, model selection and variable importance assessment. A commonly used framework involves defining a predictiveness criterion, applying a cross-fitting procedure to estimate the predictiveness, and utilizing the difference in estimated predictiveness between two models as the test statistic. However, even after standardization, the test statistic typically fails to converge to a non-degenerate distribution under the null hypothesis of equal goodness, leading to what is known as the degeneracy issue. To addresses this degeneracy issue, we present a simple yet effective device, Zipper. It draws inspiration from the strategy of additional splitting of testing data, but encourages an overlap between two testing data splits in predictiveness evaluation. Zipper binds together the two overlapping splits using a slider parameter that controls the proportion of overlap. Our proposed test statistic follows an asymptotically normal distribution under the null hypothesis for any fixed slider value, guaranteeing valid size control while enhancing power by effective data reuse. Finite-sample experiments demonstrate that our procedure, with a simple choice of the slider, works well across a wide range of settings.