DexReMoE:In-hand Reorientation of General Object via Mixtures of Experts

TL;DR

DexReMoE introduces a mixture-of-experts framework trained via reinforcement learning to enable general in-hand object reorientation across diverse complex shapes, significantly improving success rates and robustness.

Contribution

The paper presents DexReMoE, a novel mixture-of-experts approach that generalizes in-hand reorientation to complex objects by integrating multiple specialized policies and object category information.

Findings

Achieves an average success count of 19.5 across 150 objects.

Improves worst-case performance from 0.69 to 6.05.

Demonstrates scalability and adaptability in simulation.

Abstract

In hand object reorientation provides capability for dexterous manipulation, requiring robust control policies to manage diverse object geometries, maintain stable grasps, and execute precise complex orientation trajectories. However, prior works focus on single objects or simple geometries and struggle to generalize to complex shapes. In this work, we introduce DexReMoE (Dexterous Reorientation Mixture-of-Experts), in which multiple expert policies are trained for different complex shapes and integrated within a Mixture-of-Experts (MoE) framework, making the approach capable of generalizing across a wide range of objects. Additionally, we incorporate object category information as privileged inputs to enhance shape representation. Our framework is trained in simulation using reinforcement learning (RL) and evaluated on novel out-of-distribution objects in the most challenging scenario of reorienting objects held in the air by a downward-facing hand. In terms of the average consecutive success count, DexReMoE achieves a score of 19.5 across a diverse set of 150 objects. In comparison to the baselines, it also enhances the worst-case performance, increasing it from 0.69 to 6.05. These results underscore the scalability and adaptability of the DexReMoE framework for general-purpose in-hand reorientation.