Probably Approximately Correct MDP Learning and Control With Temporal Logic Constraints

TL;DR

This paper presents a PAC-MDP-based algorithm for synthesizing control policies that maximize the probability of satisfying temporal logic specifications in unknown stochastic environments, with guarantees on near-optimality and polynomial complexity.

Contribution

It introduces a model-based PAC-MDP approach for control synthesis under temporal logic constraints, ensuring near-optimal policies with polynomial sample complexity.

Findings

Algorithm achieves ε-approximate optimality with high probability

Policy construction scales polynomially with MDP and automaton size

Finitely terminating iterative policy updates

Abstract

We consider synthesis of control policies that maximize the probability of satisfying given temporal logic specifications in unknown, stochastic environments. We model the interaction between the system and its environment as a Markov decision process (MDP) with initially unknown transition probabilities. The solution we develop builds on the so-called model-based probably approximately correct Markov decision process (PAC-MDP) methodology. The algorithm attains an $\varepsilon$-approximately optimal policy with probability $1-\delta$ using samples (i.e. observations), time and space that grow polynomially with the size of the MDP, the size of the automaton expressing the temporal logic specification, $\frac{1}{\varepsilon}$, $\frac{1}{\delta}$ and a finite time horizon. In this approach, the system maintains a model of the initially unknown MDP, and constructs a product MDP based on its learned model and the specification automaton that expresses the temporal logic constraints. During execution, the policy is iteratively updated using observation of the transitions taken by the system. The iteration terminates in finitely many steps. With high probability, the resulting policy is such that, for any state, the difference between the probability of satisfying the specification under this policy and the optimal one is within a predefined bound.