Taming the Monster Every Context: Complexity Measure and Unified Framework for Offline-Oracle Efficient Contextual Bandits

TL;DR

This paper introduces OE2D, a unified framework for offline oracle-efficient contextual bandits that achieves near-optimal regret with logarithmic calls to regression oracles, supported by a new complexity measure called DOEC.

Contribution

The paper develops OE2D, a general algorithmic framework that reduces contextual bandit problems to offline regression, and introduces the DOEC complexity measure to analyze regret.

Findings

OE2D achieves near-optimal regret with logarithmic oracle calls.

DOEC is bounded in bounded Eluder dimension and smoothed regret settings.

A novel relationship between DOEC and DEC bridges offline and online bandit algorithms.

Abstract

We propose an algorithmic framework, Offline Estimation to Decisions (OE2D), that reduces contextual bandit learning with general reward function approximation to offline regression. The framework allows near-optimal regret for contextual bandits with large action spaces with $O(log(T))$ calls to an offline regression oracle over $T$ rounds, and makes $O(loglog(T))$ calls when $T$ is known. The design of OE2D algorithm generalizes Falcon~\citep{simchi2022bypassing} and its linear reward version~\citep[][Section 4]{xu2020upper} in that it chooses an action distribution that we term ``exploitative F-design'' that simultaneously guarantees low regret and good coverage that trades off exploration and exploitation. Central to our regret analysis is a new complexity measure, the Decision-Offline Estimation Coefficient (DOEC), which we show is bounded in bounded Eluder dimension per-context and smoothed regret settings. We also establish a relationship between DOEC and Decision Estimation Coefficient (DEC)~\citep{foster2021statistical}, bridging the design principles of offline- and online-oracle efficient contextual bandit algorithms for the first time.