Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertainty

TL;DR

This paper introduces a novel reinforcement learning approach that modifies feedback signals in existing controllers to improve contact-rich manipulation tasks under uncertainty, surpassing traditional residual policy learning limitations.

Contribution

It proposes a new formulation that adjusts feedback signals with RL, enabling better task learning even with black-box controllers and internal feedback constraints.

Findings

Superior performance on peg-insertion task under uncertainty

Effective modification of feedback signals with RL

Demonstrated advantages over standard RPL methods

Abstract

While classic control theory offers state of the art solutions in many problem scenarios, it is often desired to improve beyond the structure of such solutions and surpass their limitations. To this end, residual policy learning (RPL) offers a formulation to improve existing controllers with reinforcement learning (RL) by learning an additive "residual" to the output of a given controller. However, the applicability of such an approach highly depends on the structure of the controller. Often, internal feedback signals of the controller limit an RL algorithm to adequately change the policy and, hence, learn the task. We propose a new formulation that addresses these limitations by also modifying the feedback signals to the controller with an RL policy and show superior performance of our approach on a contact-rich peg-insertion task under position and orientation uncertainty. In addition, we use a recent Cartesian impedance control architecture as the control framework which can be available to us as a black-box while assuming no knowledge about its input/output structure, and show the difficulties of standard RPL. Furthermore, we introduce an adaptive curriculum for the given task to gradually increase the task difficulty in terms of position and orientation uncertainty. A video showing the results can be found at https://youtu.be/SAZm_Krze7U .