Constrained Sampling to Guide Universal Manipulation RL

TL;DR

This paper introduces Sample-Guided RL, a method that uses model-based constraint solvers to efficiently sample feasible states, guiding reinforcement learning to discover complex manipulation strategies in contact-rich environments.

Contribution

The paper presents a novel approach combining model-based sampling with RL to improve exploration and policy learning in complex manipulation tasks.

Findings

Achieves high success rates in a double sphere manipulation setting.

Demonstrates complex manipulation strategies in a panda arm environment.

Outperforms baseline methods in success rate in contact-rich tasks.

Abstract

We consider how model-based solvers can be leveraged to guide training of a universal policy to control from any feasible start state to any feasible goal in a contact-rich manipulation setting. While Reinforcement Learning (RL) has demonstrated its strength in such settings, it may struggle to sufficiently explore and discover complex manipulation strategies, especially in sparse-reward settings. Our approach is based on the idea of a lower-dimensional manifold of feasible, likely-visited states during such manipulation and to guide RL with a sampler from this manifold. We propose Sample-Guided RL, which uses model-based constraint solvers to efficiently sample feasible configurations (satisfying differentiable collision, contact, and force constraints) and leverage them to guide RL for universal (goal-conditioned) manipulation policies. We study using this data directly to bias state visitation, as well as using black-box optimization of open-loop trajectories between random configurations to impose a state bias and optionally add a behavior cloning loss. In a minimalistic double sphere manipulation setting, Sample-Guided RL discovers complex manipulation strategies and achieves high success rates in reaching any statically stable state. In a more challenging panda arm setting, our approach achieves a significant success rate over a near-zero baseline, and demonstrates a breadth of complex whole-body-contact manipulation strategies.