Dynamic Deep-Reinforcement-Learning Algorithm in Partially Observable Markov Decision Processes

TL;DR

This paper introduces three novel deep reinforcement learning algorithms using LSTM networks to better handle partially observable environments by incorporating action information, demonstrating improved performance and computational efficiency.

Contribution

It investigates the impact of including action data and network architecture choices in RNN-based RL, proposing three new algorithms with improved efficiency and effectiveness.

Findings

Inclusion of action trajectories improves learning performance.

H-TD3 reduces computational time while maintaining performance.

LSTM architectures effectively summarize hidden state trajectories.

Abstract

Recent studies have greatly improved reinforcement learning, and an increased interest in real-world implementation has emerged. In many cases, the implementation is challenged by time-varying disturbances as it introduces hidden states, which makes the problem best described with Partially Observable Markov Decision Processes. An effective approach to address this problem is to introduce a Recurrent Neural Network (RNN) in place of a state estimator. However, only a few studies have investigated the types of information to be supplied to the RNN and the network architecture to handle them. This study discusses the effectiveness of the inclusion of action along with observation and the impact of network architecture to handle them by providing interpretations of how the trajectories are summarized at LSTM networks. Specifically, three novel approaches with different architectures are introduced. All algorithms demonstrated the effectiveness of the inclusion of the action trajectories in simulation environments. In particular, one of the developed algorithms, H-TD3, differs from the typical actor and critic network as the critic network is trained by utilizing the hidden states generated by the actor network as the summarized trajectory information. This novel approach exhibited the potential improvement of the computational time while maintaining the performance.