Zero-Shot Instruction Following in RL via Structured LTL Representations

TL;DR

This paper introduces a novel method for reinforcement learning agents to follow complex, structured instructions specified in linear temporal logic (LTL), effectively handling multiple interacting high-level events through structured representations and graph neural networks.

Contribution

It presents a new approach that conditions policies on sequences of Boolean formulae aligned with automaton transitions, improving multi-task instruction following in complex environments.

Findings

Effective in complex chess-based environment

Handles multiple high-level events simultaneously

Outperforms previous methods in structured task execution

Abstract

Linear temporal logic (LTL) is a compelling framework for specifying complex, structured tasks for reinforcement learning (RL) agents. Recent work has shown that interpreting LTL instructions as finite automata, which can be seen as high-level programs monitoring task progress, enables learning a single generalist policy capable of executing arbitrary instructions at test time. However, existing approaches fall short in environments where multiple high-level events (i.e., atomic propositions) can be true at the same time and potentially interact in complicated ways. In this work, we propose a novel approach to learning a multi-task policy for following arbitrary LTL instructions that addresses this shortcoming. Our method conditions the policy on sequences of simple Boolean formulae, which directly align with transitions in the automaton, and are encoded via a graph neural network (GNN) to yield structured task representations. Experiments in a complex chess-based environment demonstrate the advantages of our approach.