Vanishing L2 regularization for the softmax Multi Armed Bandit

TL;DR

This paper analyzes the convergence of L2 regularized softmax policies in multi-armed bandit algorithms, providing theoretical proofs and empirical evidence of numerical advantages as regularization vanishes.

Contribution

It introduces a theoretical framework for analyzing the convergence of vanishing L2 regularization in softmax bandit algorithms, which was previously unresolved.

Findings

Theoretical convergence results are established for vanishing L2 regularization.

Empirical experiments show numerical advantages of this regime on benchmarks.

Abstract

Multi Armed Bandit (MAB) algorithms are a cornerstone of reinforcement learning and have been studied both theoretically and numerically. One of the most commonly used implementation uses a softmax mapping to prescribe the optimal policy and served as the foundation for downstream algorithms, including REINFORCE. Distinct from vanilla approaches, we consider here the L2 regularized softmax policy gradient where a quadratic term is subtracted from the mean reward. Previous studies exploiting convexity failed to identify a suitable theoretical framework to analyze its convergence when the regularization parameter vanishes. We prove here theoretical convergence results and confirm empirically that this regime makes the L2 regularization numerically advantageous on standard benchmarks.