Recurrent Neural-Linear Posterior Sampling for Nonstationary Contextual Bandits

TL;DR

This paper introduces a recurrent neural network-based method for nonstationary contextual bandits that learns relevant historical context directly from raw interaction data, outperforming traditional handcrafted approaches.

Contribution

It proposes a novel combination of recurrent neural networks with posterior sampling for nonstationary bandits, addressing limitations of handcrafted historical contexts.

Findings

Recurrent approach outperforms feedforward models in diverse nonstationary problems.

Method is more widely applicable than traditional nonstationary bandit algorithms.

Provides a new regret bound for linear posterior sampling with measurement error.

Abstract

An agent in a nonstationary contextual bandit problem should balance between exploration and the exploitation of (periodic or structured) patterns present in its previous experiences. Handcrafting an appropriate historical context is an attractive alternative to transform a nonstationary problem into a stationary problem that can be solved efficiently. However, even a carefully designed historical context may introduce spurious relationships or lack a convenient representation of crucial information. In order to address these issues, we propose an approach that learns to represent the relevant context for a decision based solely on the raw history of interactions between the agent and the environment. This approach relies on a combination of features extracted by recurrent neural networks with a contextual linear bandit algorithm based on posterior sampling. Our experiments on a diverse selection of contextual and noncontextual nonstationary problems show that our recurrent approach consistently outperforms its feedforward counterpart, which requires handcrafted historical contexts, while being more widely applicable than conventional nonstationary bandit algorithms. Although it is very difficult to provide theoretical performance guarantees for our new approach, we also prove a novel regret bound for linear posterior sampling with measurement error that may serve as a foundation for future theoretical work.