Live LTL Progress Tracking: Towards Task-Based Exploration

TL;DR

This paper introduces a framework for tracking the progress of reinforcement learning agents on complex tasks specified by linear temporal logic, enabling better exploration and reward shaping.

Contribution

The paper presents a novel Live LTL Progress Tracking framework that encodes task progress in RL using a vector updated at each step, facilitating new metrics and exploration strategies.

Findings

Framework formalized and demonstrated with a simple example

Tracking vector encodes detailed task execution information

Potential for integration into RL for improved exploration and reward shaping

Abstract

Motivated by the challenge presented by non-Markovian objectives in reinforcement learning (RL), we present a novel framework to track and represent the progress of autonomous agents through complex, multi-stage tasks. Given a specification in finite linear temporal logic (LTL), the framework establishes a 'tracking vector' which updates at each time step in a trajectory rollout. The values of the vector represent the status of the specification as the trajectory develops, assigning true, false, or 'open' labels (where 'open' is used for indeterminate cases). Applied to an LTL formula tree, the tracking vector can be used to encode detailed information about how a task is executed over a trajectory, providing a potential tool for new performance metrics, diverse exploration, and reward shaping. In this paper, we formally present the framework and algorithm, collectively named Live LTL Progress Tracking, give a simple working example, and demonstrate avenues for its integration into RL models. Future work will apply the framework to problems such as task-space exploration and diverse solution-finding in RL.