Revisiting State Augmentation methods for Reinforcement Learning with Stochastic Delays

TL;DR

This paper introduces a novel framework for reinforcement learning in environments with stochastic delays, transforming delayed MDPs into standard MDPs to enable near-optimal learning with reduced complexity.

Contribution

It formalizes delayed MDPs, proposes a delay-resolved RL framework, and demonstrates improved performance and efficiency over existing methods.

Findings

Delay-resolved DQN achieves near-optimal rewards

Framework simplifies handling of stochastic delays

Outperforms existing algorithms in various environments

Abstract

Several real-world scenarios, such as remote control and sensing, are comprised of action and observation delays. The presence of delays degrades the performance of reinforcement learning (RL) algorithms, often to such an extent that algorithms fail to learn anything substantial. This paper formally describes the notion of Markov Decision Processes (MDPs) with stochastic delays and shows that delayed MDPs can be transformed into equivalent standard MDPs (without delays) with significantly simplified cost structure. We employ this equivalence to derive a model-free Delay-Resolved RL framework and show that even a simple RL algorithm built upon this framework achieves near-optimal rewards in environments with stochastic delays in actions and observations. The delay-resolved deep Q-network (DRDQN) algorithm is bench-marked on a variety of environments comprising of multi-step and stochastic delays and results in better performance, both in terms of achieving near-optimal rewards and minimizing the computational overhead thereof, with respect to the currently established algorithms.