Exploring Large Action Sets with Hyperspherical Embeddings using von Mises-Fisher Sampling

TL;DR

This paper presents vMF-exp, a scalable exploration method for large action sets in reinforcement learning using hyperspherical embeddings and von Mises-Fisher sampling, outperforming traditional methods in scalability and effectiveness.

Contribution

The paper introduces vMF-exp, a novel scalable exploration technique leveraging von Mises-Fisher distributions for hyperspherical action embeddings, addressing scalability issues of existing methods.

Findings

vMF-exp scales to virtually unlimited actions

Theoretically maintains exploration probabilities similar to Boltzmann Exploration

Empirically effective in simulated, real-world, and large-scale deployment scenarios

Abstract

This paper introduces von Mises-Fisher exploration (vMF-exp), a scalable method for exploring large action sets in reinforcement learning problems where hyperspherical embedding vectors represent these actions. vMF-exp involves initially sampling a state embedding representation using a von Mises-Fisher distribution, then exploring this representation's nearest neighbors, which scales to virtually unlimited numbers of candidate actions. We show that, under theoretical assumptions, vMF-exp asymptotically maintains the same probability of exploring each action as Boltzmann Exploration (B-exp), a popular alternative that, nonetheless, suffers from scalability issues as it requires computing softmax values for each action. Consequently, vMF-exp serves as a scalable alternative to B-exp for exploring large action sets with hyperspherical embeddings. Experiments on simulated data, real-world public data, and the successful large-scale deployment of vMF-exp on the recommender system of a global music streaming service empirically validate the key properties of the proposed method.