TSEC: a framework for online experimentation under experimental constraints

TL;DR

This paper introduces TSEC, a Bayesian Thompson Sampling framework designed for online experimentation with constraints on the number of arms tested simultaneously, improving decision-making in large-scale, resource-limited settings.

Contribution

The paper presents a novel TSEC method that models correlations between arms using a Bayesian interaction model, enabling effective subset selection under experimental constraints.

Findings

TSEC outperforms industry benchmarks in website optimization.

TSEC achieves more consistent wealth accumulation in portfolio management.

Demonstrates effectiveness in simulated and real-world applications.

Abstract

Thompson sampling is a popular algorithm for solving multi-armed bandit problems, and has been applied in a wide range of applications, from website design to portfolio optimization. In such applications, however, the number of choices (or arms) $N$ can be large, and the data needed to make adaptive decisions require expensive experimentation. One is then faced with the constraint of experimenting on only a small subset of $K \ll N$ arms within each time period, which poses a problem for traditional Thompson sampling. We propose a new Thompson Sampling under Experimental Constraints (TSEC) method, which addresses this so-called "arm budget constraint". TSEC makes use of a Bayesian interaction model with effect hierarchy priors, to model correlations between rewards on different arms. This fitted model is then integrated within Thompson sampling, to jointly identify a good subset of arms for experimentation and to allocate resources over these arms. We demonstrate the effectiveness of TSEC in two problems with arm budget constraints. The first is a simulated website optimization study, where TSEC shows noticeable improvements over industry benchmarks. The second is a portfolio optimization application on industry-based exchange-traded funds, where TSEC provides more consistent and greater wealth accumulation over standard investment strategies.