Residual Overfit Method of Exploration

TL;DR

The paper introduces ROME, a computationally efficient exploration method for bandit and reinforcement learning algorithms that leverages overfitting estimates to identify uncertain actions, outperforming existing methods.

Contribution

The paper proposes ROME, a novel exploration approach using only two point estimates to efficiently identify uncertain actions without resampling or posterior approximations.

Findings

ROME outperforms established bandit methods on multiple datasets.

The method generalizes across various models and settings.

It reduces computational overhead compared to traditional approaches.

Abstract

Exploration is a crucial aspect of bandit and reinforcement learning algorithms. The uncertainty quantification necessary for exploration often comes from either closed-form expressions based on simple models or resampling and posterior approximations that are computationally intensive. We propose instead an approximate exploration methodology based on fitting only two point estimates, one tuned and one overfit. The approach, which we term the residual overfit method of exploration (ROME), drives exploration towards actions where the overfit model exhibits the most overfitting compared to the tuned model. The intuition is that overfitting occurs the most at actions and contexts with insufficient data to form accurate predictions of the reward. We justify this intuition formally from both a frequentist and a Bayesian information theoretic perspective. The result is a method that generalizes to a wide variety of models and avoids the computational overhead of resampling or posterior approximations. We compare ROME against a set of established contextual bandit methods on three datasets and find it to be one of the best performing.