Fast Online Learning with Gaussian Prior-Driven Hierarchical Unimodal Thompson Sampling

TL;DR

This paper introduces hierarchical Thompson Sampling algorithms for Gaussian-clustered multi-armed bandit problems, achieving lower regret bounds by exploiting the structure and unimodality of rewards, with theoretical and experimental validation.

Contribution

It proposes TSCG and UTSCG algorithms that leverage hierarchical and unimodal structures to improve regret bounds in Gaussian bandit problems.

Findings

TSCG outperforms standard TSG in regret bounds.

UTSCG achieves even lower regret bounds with unimodal rewards.

Numerical experiments confirm the theoretical advantages.

Abstract

We study a type of Multi-Armed Bandit (MAB) problems in which arms with a Gaussian reward feedback are clustered. Such an arm setting finds applications in many real-world problems, for example, mmWave communications and portfolio management with risky assets, as a result of the universality of the Gaussian distribution. Based on the Thompson Sampling algorithm with Gaussian prior (TSG) algorithm for the selection of the optimal arm, we propose our Thompson Sampling with Clustered arms under Gaussian prior (TSCG) specific to the 2-level hierarchical structure. We prove that by utilizing the 2-level structure, we can achieve a lower regret bound than we do with ordinary TSG. In addition, when the reward is Unimodal, we can reach an even lower bound on the regret by our Unimodal Thompson Sampling algorithm with Clustered Arms under Gaussian prior (UTSCG). Each of our proposed algorithms are accompanied by theoretical evaluation of the upper regret bound, and our numerical experiments confirm the advantage of our proposed algorithms.