Improving Reward-Conditioned Policies for Multi-Armed Bandits using Normalized Weight Functions

TL;DR

This paper enhances reward-conditioned policies for multi-armed bandits by introducing normalized weight functions, improving convergence speed and expected rewards, especially in large action spaces and sparse reward scenarios.

Contribution

It proposes a generalized marginalization technique using normalized weight functions to improve RCP performance in multi-armed bandit problems.

Findings

Improved RCP convergence and rewards with normalized weights.

Competitive performance of RCPs against classic methods like UCB and Thompson sampling.

Effective in large action spaces and sparse reward environments.

Abstract

Recently proposed reward-conditioned policies (RCPs) offer an appealing alternative in reinforcement learning. Compared with policy gradient methods, policy learning in RCPs is simpler since it is based on supervised learning, and unlike value-based methods, it does not require optimization in the action space to take actions. However, for multi-armed bandit (MAB) problems, we find that RCPs are slower to converge and have inferior expected rewards at convergence, compared with classic methods such as the upper confidence bound and Thompson sampling. In this work, we show that the performance of RCPs can be enhanced by constructing policies through the marginalization of rewards using normalized weight functions, whose sum or integral equal $1$, although the function values may be negative. We refer to this technique as generalized marginalization, whose advantage is that negative weights for policies conditioned on low rewards can make the resulting policies more distinct from them. Strategies to perform generalized marginalization in MAB with discrete action spaces are studied. Through simulations, we demonstrate that the proposed technique improves RCPs and makes them competitive with classic methods, showing superior performance on challenging MABs with large action spaces and sparse reward signals.