DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delays

TL;DR

DEER is a framework that improves reinforcement learning in environments with variable delays by using a pretrained encoder to map delayed states into meaningful representations, enhancing interpretability and delay resilience.

Contribution

The paper introduces DEER, a novel delay-resilient framework employing a pretrained encoder trained on delay-free data to handle variable delays in RL tasks.

Findings

DEER outperforms state-of-the-art RL algorithms in delayed scenarios.

The pretrained encoder effectively maps delayed states to improve learning stability.

DEER seamlessly integrates with existing RL algorithms without modifications.

Abstract

Classic reinforcement learning (RL) frequently confronts challenges in tasks involving delays, which cause a mismatch between received observations and subsequent actions, thereby deviating from the Markov assumption. Existing methods usually tackle this issue with end-to-end solutions using state augmentation. However, these black-box approaches often involve incomprehensible processes and redundant information in the information states, causing instability and potentially undermining the overall performance. To alleviate the delay challenges in RL, we propose $\textbf{DEER (Delay-resilient Encoder-Enhanced RL)}$, a framework designed to effectively enhance the interpretability and address the random delay issues. DEER employs a pretrained encoder to map delayed states, along with their variable-length past action sequences resulting from different delays, into hidden states, which is trained on delay-free environment datasets. In a variety of delayed scenarios, the trained encoder can seamlessly integrate with standard RL algorithms without requiring additional modifications and enhance the delay-solving capability by simply adapting the input dimension of the original algorithms. We evaluate DEER through extensive experiments on Gym and Mujoco environments. The results confirm that DEER is superior to state-of-the-art RL algorithms in both constant and random delay settings.