Proactive Action Visual Residual Reinforcement Learning for Contact-Rich Tasks Using a Torque-Controlled Robot

TL;DR

This paper introduces a reinforcement learning approach that integrates visual and haptic data with proactive actions to enhance contact-rich manipulation tasks on torque-controlled robots, improving adaptability and sample efficiency.

Contribution

It proposes a novel RL method combining visual, haptic, and proactive actions to address partial observability and environment variability in contact-rich tasks.

Findings

The method achieves high success rates in inserting RAMs with torque-controlled robots.

It demonstrates robustness to environmental variations.

The approach improves sample efficiency in policy learning.

Abstract

Contact-rich manipulation tasks are commonly found in modern manufacturing settings. However, manually designing a robot controller is considered hard for traditional control methods as the controller requires an effective combination of modalities and vastly different characteristics. In this paper, we firstly consider incorporating operational space visual and haptic information into reinforcement learning(RL) methods to solve the target uncertainty problem in unstructured environments. Moreover, we propose a novel idea of introducing a proactive action to solve the partially observable Markov decision process problem. Together with these two ideas, our method can either adapt to reasonable variations in unstructured environments and improve the sample efficiency of policy learning. We evaluated our method on a task that involved inserting a random-access memory using a torque-controlled robot, and we tested the success rates of the different baselines used in the traditional methods. We proved that our method is robust and can tolerate environmental variations very well.