Thompson Sampling for Contextual Bandit Problems with Auxiliary Safety Constraints

TL;DR

This paper introduces a new Thompson sampling algorithm designed for multi-objective contextual bandit problems with auxiliary safety constraints, addressing complex real-world decision-making scenarios.

Contribution

The paper proposes a novel Thompson sampling approach for multi-outcome bandits with auxiliary constraints, extending existing methods to handle multiple objectives and safety considerations.

Findings

Algorithm performs well on synthetic data

Effective in real-world video transcoding application

Provides a practical safety-performance trade-off mechanism

Abstract

Recent advances in contextual bandit optimization and reinforcement learning have garnered interest in applying these methods to real-world sequential decision making problems. Real-world applications frequently have constraints with respect to a currently deployed policy. Many of the existing constraint-aware algorithms consider problems with a single objective (the reward) and a constraint on the reward with respect to a baseline policy. However, many important applications involve multiple competing objectives and auxiliary constraints. In this paper, we propose a novel Thompson sampling algorithm for multi-outcome contextual bandit problems with auxiliary constraints. We empirically evaluate our algorithm on a synthetic problem. Lastly, we apply our method to a real world video transcoding problem and provide a practical way for navigating the trade-off between safety and performance using Bayesian optimization.