Deep Reinforcement Learning with a Stage Incentive Mechanism of Dense Reward for Robotic Trajectory Planning

TL;DR

This paper introduces dense reward functions and a stage incentive mechanism to enhance deep reinforcement learning efficiency in robotic trajectory planning, achieving faster convergence and higher success rates in manipulator tasks.

Contribution

It proposes novel dense reward functions and a stage incentive mechanism inspired by human cognition, significantly improving learning speed and stability in DRL-based robot trajectory planning.

Findings

Soft stage incentive reward improves convergence rate by up to 46.9%.

Success rate of trajectory planning reaches 99.6%.

Reductions in standard deviation of rewards indicate more stable learning.

Abstract

(This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.) To improve the efficiency of deep reinforcement learning (DRL)-based methods for robot manipulator trajectory planning in random working environments, we present three dense reward functions. These rewards differ from the traditional sparse reward. First, a posture reward function is proposed to speed up the learning process with a more reasonable trajectory by modeling the distance and direction constraints, which can reduce the blindness of exploration. Second, a stride reward function is proposed to improve the stability of the learning process by modeling the distance and movement distance of joint constraints. Finally, in order to further improve learning efficiency, we are inspired by the cognitive process of human behavior and propose a stage incentive mechanism, including a hard stage incentive reward function and a soft stage incentive reward function. Extensive experiments show that the soft stage incentive reward function is able to improve the convergence rate by up to 46.9% with the state-of-the-art DRL methods. The percentage increase in the convergence mean reward was 4.4-15.5% and the percentage decreases with respect to standard deviation were 21.9-63.2%. In the evaluation experiments, the success rate of trajectory planning for a robot manipulator reached 99.6%.