Decision-Theoretic Planning with Concurrent Temporally Extended Actions

TL;DR

This paper introduces a planning model that allows multiple temporally extended actions to be executed concurrently, leveraging a semi-Markov decision process framework to improve planning efficiency in uncertain environments.

Contribution

It extends the options framework to include concurrent actions and demonstrates how to compute value functions using SMDP algorithms for faster planning.

Findings

Concurrent options enable faster planning in navigation tasks.

The model supports overlapping execution of actions for complex tasks.

Using SMDP algorithms facilitates efficient computation of value functions.

Abstract

We investigate a model for planning under uncertainty with temporallyextended actions, where multiple actions can be taken concurrently at each decision epoch. Our model is based on the options framework, and combines it with factored state space models,where the set of options can be partitioned into classes that affectdisjoint state variables. We show that the set of decisionepochs for concurrent options defines a semi-Markov decisionprocess, if the underlying temporally extended actions being parallelized arerestricted to Markov options. This property allows us to use SMDPalgorithms for computing the value function over concurrentoptions. The concurrent options model allows overlapping execution ofoptions in order to achieve higher performance or in order to performa complex task. We describe a simple experiment using a navigationtask which illustrates how concurrent options results in a faster planwhen compared to the case when only one option is taken at a time.