Tree Search-Based Policy Optimization under Stochastic Execution Delay

TL;DR

This paper introduces stochastic delayed execution MDPs and a model-based policy optimization algorithm, DEZ, that effectively handles stochastic delays, outperforming baselines in Atari experiments.

Contribution

The paper formalizes stochastic delayed execution MDPs and develops DEZ, a novel algorithm that optimizes policies considering stochastic delays without state augmentation.

Findings

DEZ outperforms baselines in stochastic delay scenarios

Naive methods underperform with stochastic delays

The approach maintains sample efficiency similar to EfficientZero

Abstract

The standard formulation of Markov decision processes (MDPs) assumes that the agent's decisions are executed immediately. However, in numerous realistic applications such as robotics or healthcare, actions are performed with a delay whose value can even be stochastic. In this work, we introduce stochastic delayed execution MDPs, a new formalism addressing random delays without resorting to state augmentation. We show that given observed delay values, it is sufficient to perform a policy search in the class of Markov policies in order to reach optimal performance, thus extending the deterministic fixed delay case. Armed with this insight, we devise DEZ, a model-based algorithm that optimizes over the class of Markov policies. DEZ leverages Monte-Carlo tree search similar to its non-delayed variant EfficientZero to accurately infer future states from the action queue. Thus, it handles delayed execution while preserving the sample efficiency of EfficientZero. Through a series of experiments on the Atari suite, we demonstrate that although the previous baseline outperforms the naive method in scenarios with constant delay, it underperforms in the face of stochastic delays. In contrast, our approach significantly outperforms the baselines, for both constant and stochastic delays. The code is available at http://github.com/davidva1/Delayed-EZ .