Revisiting Follow-the-Perturbed-Leader with Unbounded Perturbations in Bandit Problems

TL;DR

This paper advances the theoretical understanding of Follow-the-Perturbed-Leader (FTPL) algorithms in bandit problems by establishing Best-of-Both-Worlds guarantees for unbounded perturbations, including hybrid and symmetric Fréchet-type cases, and explores their limitations.

Contribution

It extends BOBW results for FTPL with broad unbounded perturbations, introduces new insights into perturbation design, and analyzes the connection between Tsallis entropy and Fréchet-type perturbations.

Findings

BOBW guarantees established for asymmetric unbounded Fréchet-type perturbations.

First BOBW guarantee for symmetric unbounded perturbations in two-armed bandits.

Limitations identified for symmetric Fréchet-type perturbations in multi-armed bandit settings.

Abstract

Follow-the-Regularized-Leader (FTRL) policies have achieved Best-of-Both-Worlds (BOBW) results in various settings through hybrid regularizers, whereas analogous results for Follow-the-Perturbed-Leader (FTPL) remain limited due to inherent analytical challenges. To advance the analytical foundations of FTPL, we revisit classical FTRL-FTPL duality for unbounded perturbations and establish BOBW results for FTPL under a broad family of asymmetric unbounded Fr\'echet-type perturbations, including hybrid perturbations combining Gumbel-type and Fr\'echet-type tails. These results not only extend the BOBW results of FTPL but also offer new insights into designing alternative FTPL policies competitive with hybrid regularization approaches. Motivated by earlier observations in two-armed bandits, we further investigate the connection between the $1/2$-Tsallis entropy and a Fr\'echet-type perturbation. Our numerical observations suggest that it corresponds to a symmetric Fr\'echet-type perturbation, and based on this, we establish the first BOBW guarantee for symmetric unbounded perturbations in the two-armed setting. In contrast, in general multi-armed bandits, we find an instance in which symmetric Fr\'echet-type perturbations violate the key condition for standard BOBW analysis, which is a problem not observed with asymmetric or nonnegative Fr\'echet-type perturbations. Although this example does not rule out alternative analyses achieving BOBW results, it suggests the limitations of directly applying the relationship observed in two-armed cases to the general case and thus emphasizes the need for further investigation to fully understand the behavior of FTPL in broader settings.