Perturbing the Derivative: Doubly Wild Refitting for Model-Free Evaluation of Opaque Machine Learning Predictors

TL;DR

This paper introduces a model-free method for evaluating the excess risk of opaque machine learning models by perturbing derivatives and refitting, enabling risk bounds without detailed knowledge of the model class.

Contribution

The authors propose a novel wild refitting technique that bounds excess risk using only black-box access, applicable to complex models like deep neural networks.

Findings

Provides a risk upper bound without assuming model class complexity

Applicable to deep neural networks and generative models

Uses derivative perturbation and refitting for evaluation

Abstract

We study the problem of excess risk evaluation for empirical risk minimization (ERM) under convex losses. We show that by leveraging the idea of wild refitting, one can upper bound the excess risk through the so-called "wild optimism," without relying on the global structure of the underlying function class but only assuming black box access to the training algorithm and a single dataset. We begin by generating two sets of artificially modified pseudo-outcomes created by stochastically perturbing the derivatives with carefully chosen scaling. Using these pseudo-labeled datasets, we refit the black-box procedure twice to obtain two wild predictors and derive an efficient excess risk upper bound under the fixed design setting. Requiring no prior knowledge of the complexity of the underlying function class, our method is essentially model-free and holds significant promise for theoretically evaluating modern opaque deep neural networks and generative models, where traditional learning theory could be infeasible due to the extreme complexity of the hypothesis class.