Contextual Information-Directed Sampling

TL;DR

This paper explores how to optimize information-directed sampling in contextual bandits, demonstrating the benefits of considering context distribution and proposing an efficient Actor-Critic-based method.

Contribution

It introduces a new form of contextual IDS that accounts for context distribution and provides a computationally-efficient Actor-Critic implementation.

Findings

Contextual IDS outperforms conditional IDS in bandit problems.

Considering context distribution improves decision-making.

Proposed Actor-Critic method is effective in neural network bandits.

Abstract

Information-directed sampling (IDS) has recently demonstrated its potential as a data-efficient reinforcement learning algorithm. However, it is still unclear what is the right form of information ratio to optimize when contextual information is available. We investigate the IDS design through two contextual bandit problems: contextual bandits with graph feedback and sparse linear contextual bandits. We provably demonstrate the advantage of contextual IDS over conditional IDS and emphasize the importance of considering the context distribution. The main message is that an intelligent agent should invest more on the actions that are beneficial for the future unseen contexts while the conditional IDS can be myopic. We further propose a computationally-efficient version of contextual IDS based on Actor-Critic and evaluate it empirically on a neural network contextual bandit.