Robot Policy Learning from Demonstration Using Advantage Weighting and Early Termination

TL;DR

This paper introduces AWET, a novel reinforcement learning algorithm that leverages offline expert demonstrations with advantage weighting and early termination to improve robotic policy learning efficiency and performance.

Contribution

The paper proposes AWET, combining advantage-weighted critic loss and automatic early termination to enhance offline and online robotic policy learning from demonstrations.

Findings

AWET outperforms state-of-the-art baselines on four robotic tasks.

Advantage weighting improves data efficiency and policy quality.

Early termination prevents policy drift from expert trajectories.

Abstract

Learning robotic tasks in the real world is still highly challenging and effective practical solutions remain to be found. Traditional methods used in this area are imitation learning and reinforcement learning, but they both have limitations when applied to real robots. Combining reinforcement learning with pre-collected demonstrations is a promising approach that can help in learning control policies to solve robotic tasks. In this paper, we propose an algorithm that uses novel techniques to leverage offline expert data using offline and online training to obtain faster convergence and improved performance. The proposed algorithm (AWET) weights the critic losses with a novel agent advantage weight to improve over the expert data. In addition, AWET makes use of an automatic early termination technique to stop and discard policy rollouts that are not similar to expert trajectories -- to prevent drifting far from the expert data. In an ablation study, AWET showed improved and promising performance when compared to state-of-the-art baselines on four standard robotic tasks.