Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning

TL;DR

This paper introduces reward machines, a structured way to represent and exploit reward functions in reinforcement learning, leading to more sample-efficient learning and better policies by leveraging internal reward structure.

Contribution

The paper proposes reward machines, a novel finite state machine framework for explicitly representing reward functions, enabling structured learning and reward shaping in RL.

Findings

Improved sample efficiency across multiple domains

Enhanced policy quality through reward structure exploitation

Supports complex reward specifications like temporal logic

Abstract

Reinforcement learning (RL) methods usually treat reward functions as black boxes. As such, these methods must extensively interact with the environment in order to discover rewards and optimal policies. In most RL applications, however, users have to program the reward function and, hence, there is the opportunity to make the reward function visible -- to show the reward function's code to the RL agent so it can exploit the function's internal structure to learn optimal policies in a more sample efficient manner. In this paper, we show how to accomplish this idea in two steps. First, we propose reward machines, a type of finite state machine that supports the specification of reward functions while exposing reward function structure. We then describe different methodologies to exploit this structure to support learning, including automated reward shaping, task decomposition, and counterfactual reasoning with off-policy learning. Experiments on tabular and continuous domains, across different tasks and RL agents, show the benefits of exploiting reward structure with respect to sample efficiency and the quality of resultant policies. Finally, by virtue of being a form of finite state machine, reward machines have the expressive power of a regular language and as such support loops, sequences and conditionals, as well as the expression of temporally extended properties typical of linear temporal logic and non-Markovian reward specification.