Scaffolding Sets

TL;DR

This paper introduces the concept of scaffolding sets, small collections of sets that ensure a predictor's correctness through multi-calibration, inspired by neural network data representations.

Contribution

It proposes a novel framework for constructing scaffolding sets that guarantee predictor correctness beyond calibration, advancing the understanding of structured data representations.

Findings

Scaffolding sets enable predictors to be correct, not just calibrated.

The approach is inspired by neural network intermediate layer representations.

Provides a new method for ensuring predictor reliability.

Abstract

Predictors map individual instances in a population to the interval $[0,1]$. For a collection $\mathcal C$ of subsets of a population, a predictor is multi-calibrated with respect to $\mathcal C$ if it is simultaneously calibrated on each set in $\mathcal C$. We initiate the study of the construction of scaffolding sets, a small collection $\mathcal S$ of sets with the property that multi-calibration with respect to $\mathcal S$ ensures correctness, and not just calibration, of the predictor. Our approach is inspired by the folk wisdom that the intermediate layers of a neural net learn a highly structured and useful data representation.