Data-driven Error Estimation: Excess Risk Bounds without Class Complexity as Input

TL;DR

This paper introduces a data-driven method for error estimation that provides high-probability bounds without relying on class complexity, applicable to various estimation tasks and adaptive algorithms.

Contribution

It presents a novel approach to error bounds that adapts to unknown error correlations and does not require class complexity as input.

Findings

Effective in finite and infinite class settings

Applicable to confidence intervals and bandit algorithms

Overcomes limitations of existing complexity-dependent methods

Abstract

Constructing confidence intervals that are simultaneously valid across a class of estimates is central to tasks such as multiple mean estimation, generalization guarantees, and adaptive experimental design. We frame this as an ``error estimation problem," where the goal is to determine a high-probability upper bound on the maximum error for a class of estimates. We propose an entirely data-driven approach that derives such bounds for both finite and infinite class settings, naturally adapting to a potentially unknown correlation structure of random errors. Notably, our method does not require class complexity as an input, overcoming a major limitation of existing approaches. We present our simple yet general solution and demonstrate applications to simultaneous confidence intervals, excess-risk control and optimizing exploration in contextual bandit algorithms.