Sequential Decision Problems with Missing Feedback

TL;DR

This paper addresses the problem of online policy learning with missing feedback, proposing a new nonparametric algorithm that achieves near-optimal regret bounds even when rewards are missing at random.

Contribution

It introduces the Doubly-Robust Upper Confidence Bound (DR-UCB) algorithm that explicitly models missingness and attains nearly-optimal regret in such settings.

Findings

DR-UCB achieves $ ilde{O}( oot{T})$ regret.

Theoretical bounds hold under broad dependence structures.

Simulations validate the theoretical predictions.

Abstract

This paper investigates the challenges of optimal online policy learning under missing data. State-of-the-art algorithms implicitly assume that rewards are always observable. I show that when rewards are missing at random, the Upper Confidence Bound (UCB) algorithm maintains optimal regret bounds; however, it selects suboptimal policies with high probability as soon as this assumption is relaxed. To overcome this limitation, I introduce a fully nonparametric algorithm-Doubly-Robust Upper Confidence Bound (DR-UCB)-which explicitly models the form of missingness through observable covariates and achieves a nearly-optimal worst-case regret rate of $\widetilde{O}(\sqrt{T})$. To prove this result, I derive high-probability bounds for a class of doubly-robust estimators that hold under broad dependence structures. Simulation results closely match the theoretical predictions, validating the proposed framework.