Exploration and Incentives in Reinforcement Learning

TL;DR

This paper introduces a novel mechanism design approach for incentivizing exploration among self-interested agents in complex, stateful reinforcement learning environments, leveraging information asymmetry to guide exploration.

Contribution

It is the first to apply mechanism design principles to incentivize exploration in a stateful reinforcement learning setting with multiple agents.

Findings

Algorithm explores all reachable states in the MDP

Provides provable guarantees similar to static exploration problems

First work to consider mechanism design in stateful RL

Abstract

How do you incentivize self-interested agents to $\textit{explore}$ when they prefer to $\textit{exploit}$? We consider complex exploration problems, where each agent faces the same (but unknown) MDP. In contrast with traditional formulations of reinforcement learning, agents control the choice of policies, whereas an algorithm can only issue recommendations. However, the algorithm controls the flow of information, and can incentivize the agents to explore via information asymmetry. We design an algorithm which explores all reachable states in the MDP. We achieve provable guarantees similar to those for incentivizing exploration in static, stateless exploration problems studied previously. To the best of our knowledge, this is the first work to consider mechanism design in a stateful, reinforcement learning setting.