Dependence Measures Bounding the Exploration Bias for General Measurements

TL;DR

This paper introduces a new framework and dependence measures to analyze and bound exploration bias in adaptive data analysis, especially for heavy-tailed distributions, extending previous mutual information-based methods.

Contribution

It generalizes existing bias bounds to broader measurement classes and introduces dependence measures better suited for heavy-tailed data.

Findings

New bounds are nearly tight in certain cases.

Dependence measures effectively quantify exploration bias for heavy-tailed distributions.

Framework extends Russo and Zou's 2015 results to more general settings.

Abstract

We propose a framework to analyze and quantify the bias in adaptive data analysis. It generalizes that proposed by Russo and Zou'15, applying to measurements whose moment generating function exists, measurements with a finite $p$-norm, and measurements in general Orlicz spaces. We introduce a new class of dependence measures which retain key properties of mutual information while more effectively quantifying the exploration bias for heavy tailed distributions. We provide examples of cases where our bounds are nearly tight in situations where the original framework of Russo and Zou'15 does not apply.