Post-ADC Inference: Valid Inference After Active Data Collection

TL;DR

This paper introduces a post-ADC inference framework that corrects biases from active data collection and data-dependent target construction, enabling valid statistical inference after adaptive sampling.

Contribution

It proposes a new method based on selective inference to provide valid p-values and confidence intervals for data collected through active strategies like SMBO.

Findings

Valid inference achieved for GP-UCB and TPE data collection methods.

Framework corrects biases from adaptive sampling and target construction.

Empirical results confirm the validity of the proposed inference method.

Abstract

The validity of statistical inference depends critically on how data are collected. When data gathered through active data collection (ADC) are reused for a post-hoc inferential task, conventional inference can fail because the sampling is adaptively biased toward regions favored by the collection strategy. This issue is especially pronounced in black-box optimization, where sequential model-based optimization (SMBO) methods such as the tree-structured Parzen estimator (TPE) and Gaussian process upper confidence bound (GP-UCB) preferentially concentrate evaluations in promising regions. We study statistical inference on actively collected data when the inferential target is constructed in a data-dependent manner after data collection. To enable valid inference in this setting, we propose post-ADC inference, a framework that accounts for the biases arising from both the active data collection process and the subsequent data-driven target construction. Our method builds on selective inference and provides valid $p$-values and confidence intervals that correct for both sources of bias. The framework applies to a broad class of ADC processes by imposing only assumptions on the observation noise, without requiring any assumptions on the underlying black-box function or the surrogate model used by the SMBO algorithm. Empirical results also show that post-ADC inference provides valid inference for data collected by GP-UCB and TPE.