Numeric Reward Machines

TL;DR

This paper extends reward machines in reinforcement learning to handle numeric features like distances, enabling more effective guidance in inherently numeric tasks, and demonstrates significant performance improvements in the Craft domain.

Contribution

The paper introduces two types of reward machines that incorporate numeric features, expanding their applicability to numeric tasks in reinforcement learning.

Findings

Numeric reward machines outperform baseline in Craft domain.

Numeric features improve learning efficiency.

Approaches generalize to various numeric features.

Abstract

Reward machines inform reinforcement learning agents about the reward structure of the environment and often drastically speed up the learning process. However, reward machines only accept Boolean features such as robot-reached-gold. Consequently, many inherently numeric tasks cannot profit from the guidance offered by reward machines. To address this gap, we aim to extend reward machines with numeric features such as distance-to-gold. For this, we present two types of reward machines: numeric-Boolean and numeric. In a numeric-Boolean reward machine, distance-to-gold is emulated by two Boolean features distance-to-gold-decreased and robot-reached-gold. In a numeric reward machine, distance-to-gold is used directly alongside the Boolean feature robot-reached-gold. We compare our new approaches to a baseline reward machine in the Craft domain, where the numeric feature is the agent-to-target distance. We use cross-product Q-learning, Q-learning with counter-factual experiences, and the options framework for learning. Our experimental results show that our new approaches significantly outperform the baseline approach. Extending reward machines with numeric features opens up new possibilities of using reward machines in inherently numeric tasks.