Efficient Multivariate Bandit Algorithm with Path Planning

TL;DR

This paper introduces a novel path planning framework using decision graphs and Thompson sampling to efficiently solve high-dimensional multivariate Multi-Armed Bandit problems, overcoming the curse of dimensionality.

Contribution

It is the first to apply graph path planning and Monte Carlo tree search strategies to multivariate MAB, improving convergence and regret performance.

Findings

Faster convergence speed in simulations

More efficient optimal arm allocation

Lower cumulative regret achieved

Abstract

In this paper, we solve the arms exponential exploding issue in multivariate Multi-Armed Bandit (Multivariate-MAB) problem when the arm dimension hierarchy is considered. We propose a framework called path planning (TS-PP) which utilizes decision graph/trees to model arm reward success rate with m-way dimension interaction, and adopts Thompson sampling (TS) for heuristic search of arm selection. Naturally, it is quite straightforward to combat the curse of dimensionality using a serial processes that operates sequentially by focusing on one dimension per each process. For our best acknowledge, we are the first to solve Multivariate-MAB problem using graph path planning strategy and deploying alike Monte-Carlo tree search ideas. Our proposed method utilizing tree models has advantages comparing with traditional models such as general linear regression. Simulation studies validate our claim by achieving faster convergence speed, better efficient optimal arm allocation and lower cumulative regret.