Model-Free Reinforcement Learning for Optimal Control of MarkovDecision Processes Under Signal Temporal Logic Specifications

TL;DR

This paper introduces a model-free reinforcement learning method to synthesize policies for finite-horizon MDPs that satisfy Signal Temporal Logic specifications with probabilistic guarantees, applicable to robotic motion planning under uncertainty.

Contribution

It proposes a novel state augmentation technique and formulates STL constraints as reachability objectives within a constrained MDP framework, enabling model-free RL to handle complex temporal logic specifications.

Findings

Effective in robotic motion planning under uncertainty

Guarantees a lower bound on STL satisfaction probability

Leverages existing RL algorithms for constrained policy synthesis

Abstract

We present a model-free reinforcement learning algorithm to find an optimal policy for a finite-horizon Markov decision process while guaranteeing a desired lower bound on the probability of satisfying a signal temporal logic (STL) specification. We propose a method to effectively augment the MDP state space to capture the required state history and express the STL objective as a reachability objective. The planning problem can then be formulated as a finite-horizon constrained Markov decision process (CMDP). For a general finite horizon CMDP problem with unknown transition probability, we develop a reinforcement learning scheme that can leverage any model-free RL algorithm to provide an approximately optimal policy out of the general space of non-stationary randomized policies. We illustrate the effectiveness of our approach in the context of robotic motion planning for complex missions under uncertainty and performance objectives.