Efficient and Optimal Policy Gradient Algorithm for Corrupted Multi-armed Bandits

TL;DR

This paper introduces SAMBA, a policy gradient algorithm for corrupted multi-armed bandits, achieving near-optimal regret bounds and outperforming existing methods through theoretical analysis and simulations.

Contribution

The paper proposes SAMBA, a computationally efficient policy gradient algorithm that improves regret bounds for corrupted bandit problems, reducing the logarithmic factor compared to prior algorithms.

Findings

SAMBA achieves a regret bound of O(K log T / Δ + C / Δ).

SAMBA reduces the log T factor in regret compared to CBARBAR.

Simulations show SAMBA outperforms existing algorithms in practice.

Abstract

In this paper, we consider the stochastic multi-armed bandits problem with adversarial corruptions, where the random rewards of the arms are partially modified by an adversary to fool the algorithm. We apply the policy gradient algorithm SAMBA to this setting, and show that it is computationally efficient, and achieves a state-of-the-art $O(K\log T/\Delta) + O(C/\Delta)$ regret upper bound, where $K$ is the number of arms, $C$ is the unknown corruption level, $\Delta$ is the minimum expected reward gap between the best arm and other ones, and $T$ is the time horizon. Compared with the best existing efficient algorithm (e.g., CBARBAR), whose regret upper bound is $O(K\log^2 T/\Delta) + O(C)$, we show that SAMBA reduces one $\log T$ factor in the regret bound, while maintaining the corruption-dependent term to be linear with $C$. This is indeed asymptotically optimal. We also conduct simulations to demonstrate the effectiveness of SAMBA, and the results show that SAMBA outperforms existing baselines.