Diversity-Driven Selection of Exploration Strategies in Multi-Armed Bandits

TL;DR

This paper introduces a novel diversity-driven approach for selecting exploration strategies in multi-armed bandit settings, demonstrating its effectiveness in a robotic simulation by discriminating strategy quality and achieving competitive performance.

Contribution

It proposes a new strategy-agnostic method that uses diversity as a reward signal for exploration strategy selection in multi-armed bandits.

Findings

Effectively discriminates between strategies of varying quality.

Achieves performance comparable to the best fixed strategy mixture.

Works well even with subtle differences between strategies.

Abstract

We consider a scenario where an agent has multiple available strategies to explore an unknown environment. For each new interaction with the environment, the agent must select which exploration strategy to use. We provide a new strategy-agnostic method that treat the situation as a Multi-Armed Bandits problem where the reward signal is the diversity of effects that each strategy produces. We test the method empirically on a simulated planar robotic arm, and establish that the method is both able discriminate between strategies of dissimilar quality, even when the differences are tenuous, and that the resulting performance is competitive with the best fixed mixture of strategies.