FORM: Learning Expressive and Transferable First-Order Logic Reward Machines

TL;DR

This paper introduces First-Order Reward Machines (FORMs), which use first-order logic for more expressive and transferable reward representations in reinforcement learning, enabling scalable learning and multi-agent collaboration.

Contribution

The paper proposes FORM, a novel first-order logic-based reward machine, along with a learning method and multi-agent framework to improve transferability and scalability in RL tasks.

Findings

FORMs are more compact and scalable than traditional RMs.

Effective learning of FORMs in complex tasks where traditional RMs fail.

Multi-agent approach enhances transferability and learning speed.

Abstract

Reward machines (RMs) are an effective approach for addressing non-Markovian rewards in reinforcement learning (RL) through finite-state machines. Traditional RMs, which label edges with propositional logic formulae, inherit the limited expressivity of propositional logic. This limitation hinders the learnability and transferability of RMs since complex tasks will require numerous states and edges. To overcome these challenges, we propose First-Order Reward Machines ($\texttt{FORM}$s), which use first-order logic to label edges, resulting in more compact and transferable RMs. We introduce a novel method for $\textbf{learning}$ $\texttt{FORM}$s and a multi-agent formulation for $\textbf{exploiting}$ them and facilitate their transferability, where multiple agents collaboratively learn policies for a shared $\texttt{FORM}$. Our experimental results demonstrate the scalability of $\texttt{FORM}$s with respect to traditional RMs. Specifically, we show that $\texttt{FORM}$s can be effectively learnt for tasks where traditional RM learning approaches fail. We also show significant improvements in learning speed and task transferability thanks to the multi-agent learning framework and the abstraction provided by the first-order language.