Directed Exploration in Reinforcement Learning from Linear Temporal Logic

TL;DR

This paper introduces a novel exploration strategy for reinforcement learning with linear temporal logic (LTL) specifications, leveraging automaton-based value estimation and Bayesian methods to improve scalability from simple to complex environments.

Contribution

It proposes a new approach that casts the LTL automaton as a Markov reward process and uses Bayesian inference to generate intrinsic rewards, enhancing exploration in high-dimensional settings.

Findings

Improved exploration in complex RL environments with LTL specifications.

Successful application from tabular to high-dimensional continuous systems.

Enhanced scalability of LTL-based RL algorithms.

Abstract

Linear temporal logic (LTL) is a powerful language for task specification in reinforcement learning, as it allows describing objectives beyond the expressivity of conventional discounted return formulations. Nonetheless, recent works have shown that LTL formulas can be translated into a variable rewarding and discounting scheme, whose optimization produces a policy maximizing a lower bound on the probability of formula satisfaction. However, the synthesized reward signal remains fundamentally sparse, making exploration challenging. We aim to overcome this limitation, which can prevent current algorithms from scaling beyond low-dimensional, short-horizon problems. We show how better exploration can be achieved by further leveraging the LTL specification and casting its corresponding Limit Deterministic B\"uchi Automaton (LDBA) as a Markov reward process, thus enabling a form of high-level value estimation. By taking a Bayesian perspective over LDBA dynamics and proposing a suitable prior distribution, we show that the values estimated through this procedure can be treated as a shaping potential and mapped to informative intrinsic rewards. Empirically, we demonstrate applications of our method from tabular settings to high-dimensional continuous systems, which have so far represented a significant challenge for LTL-based reinforcement learning algorithms.