Real-Robot Deep Reinforcement Learning: Improving Trajectory Tracking of Flexible-Joint Manipulator with Reference Correction

TL;DR

This paper presents a deep reinforcement learning approach for real-robot trajectory tracking of flexible-joint manipulators, achieving rapid learning and improved accuracy over traditional controllers.

Contribution

The work introduces a model-free, off-policy RL method with reference correction and informed initialization for efficient real-robot learning.

Findings

Learned policy in under two hours on real robot

Significantly improved trajectory tracking accuracy

Generalizes to unseen payloads

Abstract

Flexible-joint manipulators are governed by complex nonlinear dynamics, defining a challenging control problem. In this work, we propose an approach to learn an outer-loop joint trajectory tracking controller with deep reinforcement learning. The controller represented by a stochastic policy is learned in under two hours directly on the real robot. This is achieved through bounded reference correction actions and use of a model-free off-policy learning method. In addition, an informed policy initialization is proposed, where the agent is pre-trained in a learned simulation. We test our approach on the 7 DOF manipulator of a Baxter robot. We demonstrate that the proposed method is capable of consistent learning across multiple runs when applied directly on the real robot. Our method yields a policy which significantly improves the trajectory tracking accuracy in comparison to the vendor-provided controller, generalizing to an unseen payload.