Learning Long-Horizon Robot Manipulation Skills via Privileged Action

TL;DR

This paper introduces a structured framework using privileged actions and curriculum learning in simulation to efficiently train robot manipulation skills for complex long-horizon tasks, with successful transfer to real-world scenarios.

Contribution

The paper presents a novel simulation-based training method leveraging privileged actions and curriculum learning, enabling efficient acquisition of complex manipulation skills without extensive reward engineering.

Findings

Successfully trained policies for multi-stage long-horizon tasks

Achieved robust and diverse behaviors across environments

Demonstrated transferability of skills to real-world robots

Abstract

Long-horizon contact-rich tasks are challenging to learn with reinforcement learning, due to ineffective exploration of high-dimensional state spaces with sparse rewards. The learning process often gets stuck in local optimum and demands task-specific reward fine-tuning for complex scenarios. In this work, we propose a structured framework that leverages privileged actions with curriculum learning, enabling the policy to efficiently acquire long-horizon skills without relying on extensive reward engineering or reference trajectories. Specifically, we use privileged actions in simulation with a general training procedure that would be infeasible to implement in real-world scenarios. These privileges include relaxed constraints and virtual forces that enhance interaction and exploration with objects. Our results successfully achieve complex multi-stage long-horizon tasks that naturally combine non-prehensile manipulation with grasping to lift objects from non-graspable poses. We demonstrate generality by maintaining a parsimonious reward structure and showing convergence to diverse and robust behaviors across various environments. Additionally, real-world experiments further confirm that the skills acquired using our approach are transferable to real-world environments, exhibiting robust and intricate performance. Our approach outperforms state-of-the-art methods in these tasks, converging to solutions where others fail.