Reward-Punishment Reinforcement Learning with Maximum Entropy

TL;DR

This paper presents softDMP, a reinforcement learning algorithm that enhances reward-punishment learning by integrating entropy optimization, leading to improved sample efficiency and robustness in navigation tasks.

Contribution

The paper introduces softDMP, a novel method that smooths traditional operators in reward-punishment RL and addresses data collection issues for better learning stability.

Findings

SoftDMP improves sample efficiency in discrete MDPs.

The probabilistic classifier effectively separates roll-outs for reward and punishment updates.

Superior performance demonstrated in Turtlebot 3 maze navigation tasks.

Abstract

We introduce the ``soft Deep MaxPain'' (softDMP) algorithm, which integrates the optimization of long-term policy entropy into reward-punishment reinforcement learning objectives. Our motivation is to facilitate a smoother variation of operators utilized in the updating of action values beyond traditional ``max'' and ``min'' operators, where the goal is enhancing sample efficiency and robustness. We also address two unresolved issues from the previous Deep MaxPain method. Firstly, we investigate how the negated (``flipped'') pain-seeking sub-policy, derived from the punishment action value, collaborates with the ``min'' operator to effectively learn the punishment module and how softDMP's smooth learning operator provides insights into the ``flipping'' trick. Secondly, we tackle the challenge of data collection for learning the punishment module to mitigate inconsistencies arising from the involvement of the ``flipped'' sub-policy (pain-avoidance sub-policy) in the unified behavior policy. We empirically explore the first issue in two discrete Markov Decision Process (MDP) environments, elucidating the crucial advancements of the DMP approach and the necessity for soft treatments on the hard operators. For the second issue, we propose a probabilistic classifier based on the ratio of the pain-seeking sub-policy to the sum of the pain-seeking and goal-reaching sub-policies. This classifier assigns roll-outs to separate replay buffers for updating reward and punishment action-value functions, respectively. Our framework demonstrates superior performance in Turtlebot 3's maze navigation tasks under the ROS Gazebo simulation.