DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL

TL;DR

DeepLTL introduces a novel method leveraging B"uchi automata to enable multi-task reinforcement learning agents to efficiently satisfy complex LTL specifications, including unseen ones, across various domains.

Contribution

The paper presents a new learning approach that uses B"uchi automata to handle arbitrary LTL specifications, improving over existing methods in generality and efficiency.

Findings

Successfully satisfies a wide range of LTL specifications in multiple domains.

Outperforms existing methods in satisfaction probability.

Demonstrates zero-shot generalization to unseen specifications.

Abstract

Linear temporal logic (LTL) has recently been adopted as a powerful formalism for specifying complex, temporally extended tasks in multi-task reinforcement learning (RL). However, learning policies that efficiently satisfy arbitrary specifications not observed during training remains a challenging problem. Existing approaches suffer from several shortcomings: they are often only applicable to finite-horizon fragments of LTL, are restricted to suboptimal solutions, and do not adequately handle safety constraints. In this work, we propose a novel learning approach to address these concerns. Our method leverages the structure of B\"uchi automata, which explicitly represent the semantics of LTL specifications, to learn policies conditioned on sequences of truth assignments that lead to satisfying the desired formulae. Experiments in a variety of discrete and continuous domains demonstrate that our approach is able to zero-shot satisfy a wide range of finite- and infinite-horizon specifications, and outperforms existing methods in terms of both satisfaction probability and efficiency. Code available at: https://deep-ltl.github.io/