Compositional planning in Markov decision processes: Temporal abstraction meets generalized logic composition

TL;DR

This paper introduces a novel compositional reasoning approach for hierarchical planning in Markov decision processes, combining temporal abstraction with generalized logic composition to efficiently synthesize policies satisfying complex temporal logic constraints.

Contribution

It proposes a new method for policy composition based on generalized logic, enabling efficient hierarchical planning under temporal logic constraints in MDPs.

Findings

The method efficiently synthesizes policies for complex tasks.

It guarantees correctness and optimality in satisfying temporal logic specifications.

Demonstrated effectiveness in stochastic planning examples.

Abstract

In hierarchical planning for Markov decision processes (MDPs), temporal abstraction allows planning with macro-actions that take place at different time scale in form of sequential composition. In this paper, we propose a novel approach to compositional reasoning and hierarchical planning for MDPs under temporal logic constraints. In addition to sequential composition, we introduce a composition of policies based on generalized logic composition: Given sub-policies for sub-tasks and a new task expressed as logic compositions of subtasks, a semi-optimal policy, which is optimal in planning with only sub-policies, can be obtained by simply composing sub-polices. Thus, a synthesis algorithm is developed to compute optimal policies efficiently by planning with primitive actions, policies for sub-tasks, and the compositions of sub-policies, for maximizing the probability of satisfying temporal logic specifications. We demonstrate the correctness and efficiency of the proposed method in stochastic planning examples with a single agent and multiple task specifications.