Stable In-hand Manipulation with Finger Specific Multi-agent Shadow Reward

TL;DR

This paper introduces FMSR, a dense reward method for stable in-hand manipulation with multi-agent reinforcement learning, improving convergence speed and stability over traditional sparse rewards.

Contribution

The novel FMSR method determines stable manipulation constraints via dense rewards based on state-action occupancy, enhanced by information sharing among agents.

Findings

FMSR+IS converges faster than traditional methods.

FMSR+IS achieves higher success rates and stability.

Dense rewards improve manipulation stability over sparse rewards.

Abstract

Deep Reinforcement Learning has shown its capability to solve the high degrees of freedom in control and the complex interaction with the object in the multi-finger dexterous in-hand manipulation tasks. Current DRL approaches prefer sparse rewards to dense rewards for the ease of training but lack behavior constraints during the manipulation process, leading to aggressive and unstable policies that are insufficient for safety-critical in-hand manipulation tasks. Dense rewards can regulate the policy to learn stable manipulation behaviors with continuous reward constraints but are hard to empirically define and slow to converge optimally. This work proposes the Finger-specific Multi-agent Shadow Reward (FMSR) method to determine the stable manipulation constraints in the form of dense reward based on the state-action occupancy measure, a general utility of DRL that is approximated during the learning process. Information Sharing (IS) across neighboring agents enables consensus training to accelerate the convergence. The methods are evaluated in two in-hand manipulation tasks on the Shadow Hand. The results show FMSR+IS converges faster in training, achieving a higher task success rate and better manipulation stability than conventional dense reward. The comparison indicates FMSR+IS achieves a comparable success rate even with the behavior constraint but much better manipulation stability than the policy trained with a sparse reward.