Exploration Potential

TL;DR

This paper introduces exploration potential, a new metric for measuring how well reinforcement learning agents have explored their environment class, considering reward structure to improve asymptotic optimality.

Contribution

The paper proposes exploration potential as a novel exploration measure that accounts for reward structure, enhancing the understanding of exploration-exploitation tradeoffs.

Findings

Exploration potential effectively guides exploration strategies.

Algorithms using exploration potential achieve better asymptotic optimality.

Experimental results in multi-armed bandits demonstrate its practical utility.

Abstract

We introduce exploration potential, a quantity that measures how much a reinforcement learning agent has explored its environment class. In contrast to information gain, exploration potential takes the problem's reward structure into account. This leads to an exploration criterion that is both necessary and sufficient for asymptotic optimality (learning to act optimally across the entire environment class). Our experiments in multi-armed bandits use exploration potential to illustrate how different algorithms make the tradeoff between exploration and exploitation.