Omega-Regular Reward Machines

TL;DR

This paper introduces omega-regular reward machines, combining reward machines with omega-regular languages to create expressive reward mechanisms for reinforcement learning, supported by a new algorithm and experimental validation.

Contribution

It proposes omega-regular reward machines and a model-free RL algorithm to handle complex non-Markovian rewards in reinforcement learning.

Findings

The algorithm computes epsilon-optimal strategies effectively.

Experimental results demonstrate the approach's practicality.

Omega-regular reward machines enhance reward expressiveness.

Abstract

Reinforcement learning (RL) is a powerful approach for training agents to perform tasks, but designing an appropriate reward mechanism is critical to its success. However, in many cases, the complexity of the learning objectives goes beyond the capabilities of the Markovian assumption, necessitating a more sophisticated reward mechanism. Reward machines and omega-regular languages are two formalisms used to express non-Markovian rewards for quantitative and qualitative objectives, respectively. This paper introduces omega-regular reward machines, which integrate reward machines with omega-regular languages to enable an expressive and effective reward mechanism for RL. We present a model-free RL algorithm to compute epsilon-optimal strategies against omega-egular reward machines and evaluate the effectiveness of the proposed algorithm through experiments.