Semantically Labelled Automata for Multi-Task Reinforcement Learning with LTL Instructions

TL;DR

This paper introduces a novel method for multi-task reinforcement learning using semantically labeled automata derived from LTL instructions, enabling efficient task embedding and improved generalization to complex, unseen tasks.

Contribution

It presents a new semantic LTL-to-automata translation technique that enhances task embedding and policy conditioning in multi-task RL, outperforming existing methods.

Findings

Achieves state-of-the-art performance across various domains.

Scales effectively to complex LTL specifications.

Supports full LTL with rich automaton state information.

Abstract

We study multi-task reinforcement learning (RL), a setting in which an agent learns a single, universal policy capable of generalising to arbitrary, possibly unseen tasks. We consider tasks specified as linear temporal logic (LTL) formulae, which are commonly used in formal methods to specify properties of systems, and have recently been successfully adopted in RL. In this setting, we present a novel task embedding technique leveraging a new generation of semantic LTL-to-automata translations, originally developed for temporal synthesis. The resulting semantically labelled automata contain rich, structured information in each state that allow us to (i) compute the automaton efficiently on-the-fly, (ii) extract expressive task embeddings used to condition the policy, and (iii) naturally support full LTL. Experimental results in a variety of domains demonstrate that our approach achieves state-of-the-art performance and is able to scale to complex specifications where existing methods fail.