Entropy Rate Maximization of Markov Decision Processes under Linear Temporal Logic Tasks

TL;DR

This paper develops a polynomial-time algorithm for synthesizing control policies for Markov decision processes that maximize entropy rate while satisfying linear temporal logic tasks, enhancing unpredictability and security.

Contribution

It introduces a novel approach to maximize entropy rate under LTL constraints, addressing limitations of previous methods that diverged for infinite horizons.

Findings

Algorithm for entropy rate maximization in communicating MDPs.

Polynomial-time solution for LTL-constrained entropy rate maximization.

Validated effectiveness through robot task planning case studies.

Abstract

We investigate the problem of synthesizing optimal control policies for Markov decision processes (MDPs) with both qualitative and quantitative objectives. Specifically, our goal is to achieve a given linear temporal logic (LTL) task with probability one, while maximizing the \emph{entropy rate} of the system. The notion of entropy rate characterizes the long-run average (un)predictability of a stochastic process. Such an optimal policy is of our interest, in particular, from the security point of view, as it not only ensures the completion of tasks, but also maximizes the unpredictability of the system. However, existing works only focus on maximizing the total entropy which may diverge to infinity for infinite horizon. In this paper, we provide a complete solution to the entropy rate maximization problem under LTL constraints. Specifically, we first present an algorithm for synthesizing entropy rate maximizing policies for communicating MDPs. Then based on a new state classification method, we show the entropy rate maximization problem under LTL task can be effectively solved in polynomial-time. We illustrate the proposed algorithm based on two case studies of robot task planning scenario.