Logically-Constrained Reinforcement Learning

TL;DR

This paper introduces a model-free reinforcement learning algorithm that synthesizes policies satisfying linear temporal logic properties in unknown MDPs, improving efficiency by focusing exploration on relevant states and guaranteeing probabilistic satisfaction.

Contribution

It presents the first RL method that incorporates LTL constraints directly into policy synthesis for unknown MDPs, enhancing scalability and performance.

Findings

Achieves an order of magnitude reduction in synthesis iterations.

Guarantees probabilistic satisfaction of LTL properties if such policies exist.

Produces reasonable policies even when the property cannot be satisfied.

Abstract

We present the first model-free Reinforcement Learning (RL) algorithm to synthesise policies for an unknown Markov Decision Process (MDP), such that a linear time property is satisfied. The given temporal property is converted into a Limit Deterministic Buchi Automaton (LDBA) and a robust reward function is defined over the state-action pairs of the MDP according to the resulting LDBA. With this reward function, the policy synthesis procedure is "constrained" by the given specification. These constraints guide the MDP exploration so as to minimize the solution time by only considering the portion of the MDP that is relevant to satisfaction of the LTL property. This improves performance and scalability of the proposed method by avoiding an exhaustive update over the whole state space while the efficiency of standard methods such as dynamic programming is hindered by excessive memory requirements, caused by the need to store a full-model in memory. Additionally, we show that the RL procedure sets up a local value iteration method to efficiently calculate the maximum probability of satisfying the given property, at any given state of the MDP. We prove that our algorithm is guaranteed to find a policy whose traces probabilistically satisfy the LTL property if such a policy exists, and additionally we show that our method produces reasonable control policies even when the LTL property cannot be satisfied. The performance of the algorithm is evaluated via a set of numerical examples. We observe an improvement of one order of magnitude in the number of iterations required for the synthesis compared to existing approaches.