Plan-Guided Reinforcement Learning for Whole-Body Manipulation

TL;DR

This paper introduces a plan-guided reinforcement learning framework that combines model-based planning with RL to enable robust whole-body manipulation with minimal human supervision, demonstrated on a humanoid robot handling large objects.

Contribution

The proposed PGRL method integrates model-based plans with RL, reducing supervision and improving robustness in complex manipulation tasks.

Findings

Effective in guiding RL exploration with minimal supervision

Improves robustness of manipulation policies

Preliminary results on humanoid robot are promising

Abstract

Synthesizing complex whole-body manipulation behaviors has fundamental challenges due to the rapidly growing combinatorics inherent to contact interaction planning. While model-based methods have shown promising results in solving long-horizon manipulation tasks, they often work under strict assumptions, such as known model parameters, oracular observation of the environment state, and simplified dynamics, resulting in plans that cannot easily transfer to hardware. Learning-based approaches, such as imitation learning (IL) and reinforcement learning (RL), have been shown to be robust when operating over in-distribution states; however, they need heavy human supervision. Specifically, model-free RL requires a tedious reward-shaping process. IL methods, on the other hand, rely on human demonstrations that involve advanced teleoperation methods. In this work, we propose a plan-guided reinforcement learning (PGRL) framework to combine the advantages of model-based planning and reinforcement learning. Our method requires minimal human supervision because it relies on plans generated by model-based planners to guide the exploration in RL. In exchange, RL derives a more robust policy thanks to domain randomization. We test this approach on a whole-body manipulation task on Punyo, an upper-body humanoid robot with compliant, air-filled arm coverings, to pivot and lift a large box. Our preliminary results indicate that the proposed methodology is promising to address challenges that remain difficult for either model- or learning-based strategies alone.