Partially Observable Contextual Bandits with Linear Payoffs

TL;DR

This paper introduces EMKF-Bandit, an algorithm for partially observable contextual bandits with linear payoffs, combining filtering and bandit strategies to handle correlated, incomplete information in decision-making scenarios like finance.

Contribution

It proposes a novel pipeline integrating system identification, filtering, and bandit algorithms for partially observable contexts, with theoretical regret analysis and practical simulations.

Findings

EMKF-Bandit achieves sub-linear regret under certain filtering conditions.

The pipeline effectively handles correlated, incomplete contexts in simulations.

Numerical results demonstrate practical benefits in finance-like scenarios.

Abstract

The standard contextual bandit framework assumes fully observable and actionable contexts. In this work, we consider a new bandit setting with partially observable, correlated contexts and linear payoffs, motivated by the applications in finance where decision making is based on market information that typically displays temporal correlation and is not fully observed. We make the following contributions marrying ideas from statistical signal processing with bandits: (i) We propose an algorithmic pipeline named EMKF-Bandit, which integrates system identification, filtering, and classic contextual bandit algorithms into an iterative method alternating between latent parameter estimation and decision making. (ii) We analyze EMKF-Bandit when we select Thompson sampling as the bandit algorithm and show that it incurs a sub-linear regret under conditions on filtering. (iii) We conduct numerical simulations that demonstrate the benefits and practical applicability of the proposed pipeline.