"What, not how": Solving an under-actuated insertion task from scratch

TL;DR

This paper presents a reinforcement learning approach to solve a complex, under-actuated peg-in-hole manipulation task involving diverse skills and sparse rewards, demonstrating success both in simulation and on a real robot with limited data.

Contribution

It introduces a novel multi-task RL framework combining SAC-X and RHPO to learn complex manipulation skills from scratch in a highly under-actuated setting.

Findings

Successfully solves the task in simulation.

Achieves real robot insertion with limited data.

Demonstrates emergence of complex behaviors.

Abstract

Robot manipulation requires a complex set of skills that need to be carefully combined and coordinated to solve a task. Yet, most ReinforcementLearning (RL) approaches in robotics study tasks which actually consist only of a single manipulation skill, such as grasping an object or inserting a pre-grasped object. As a result the skill ('how' to solve the task) but not the actual goal of a complete manipulation ('what' to solve) is specified. In contrast, we study a complex manipulation goal that requires an agent to learn and combine diverse manipulation skills. We propose a challenging, highly under-actuated peg-in-hole task with a free, rotational asymmetrical peg, requiring a broad range of manipulation skills. While correct peg (re-)orientation is a requirement for successful insertion, there is no reward associated with it. Hence an agent needs to understand this pre-condition and learn the skill to fulfil it. The final insertion reward is sparse, allowing freedom in the solution and leading to complex emerging behaviour not envisioned during the task design. We tackle the problem in a multi-task RL framework using Scheduled Auxiliary Control (SAC-X) combined with Regularized Hierarchical Policy Optimization (RHPO) which successfully solves the task in simulation and from scratch on a single robot where data is severely limited.