Thompson Sampling in Dynamic Systems for Contextual Bandit Problems

TL;DR

This paper develops a Thompson Sampling approach for contextual bandit problems in dynamic systems, using Laplace Approximation for nonlinear models and adaptive decay rates to balance exploration and exploitation.

Contribution

It introduces a novel method combining Laplace Approximation with adaptive decay in Thompson Sampling for nonlinear, time-varying systems in contextual bandit settings.

Findings

Effective posterior approximation for nonlinear dynamic models

Adaptive decay rates improve exploration-exploitation balance

Enhanced performance in time-varying contextual bandit problems

Abstract

We consider the multiarm bandit problems in the timevarying dynamic system for rich structural features. For the nonlinear dynamic model, we propose the approximate inference for the posterior distributions based on Laplace Approximation. For the context bandit problems, Thompson Sampling is adopted based on the underlying posterior distributions of the parameters. More specifically, we introduce the discount decays on the previous samples impact and analyze the different decay rates with the underlying sample dynamics. Consequently, the exploration and exploitation is adaptively tradeoff according to the dynamics in the system.