HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning

TL;DR

HumanoidGen introduces an automated framework using LLM reasoning and atomic operations to generate high-quality data for bimanual humanoid robot manipulation, addressing the lack of datasets and benchmarks.

Contribution

It presents a novel data generation method leveraging LLMs and atomic dexterous operations for humanoid robot manipulation tasks.

Findings

Generated datasets improve 2D and 3D policy performance.

Enhanced planning with Monte Carlo tree search boosts task execution.

Benchmark creation facilitates evaluation of humanoid manipulation data quality.

Abstract

For robotic manipulation, existing robotics datasets and simulation benchmarks predominantly cater to robot-arm platforms. However, for humanoid robots equipped with dual arms and dexterous hands, simulation tasks and high-quality demonstrations are notably lacking. Bimanual dexterous manipulation is inherently more complex, as it requires coordinated arm movements and hand operations, making autonomous data collection challenging. This paper presents HumanoidGen, an automated task creation and demonstration collection framework that leverages atomic dexterous operations and LLM reasoning to generate relational constraints. Specifically, we provide spatial annotations for both assets and dexterous hands based on the atomic operations, and perform an LLM planner to generate a chain of actionable spatial constraints for arm movements based on object affordances and scenes. To further improve planning ability, we employ a variant of Monte Carlo tree search to enhance LLM reasoning for long-horizon tasks and insufficient annotation. In experiments, we create a novel benchmark with augmented scenarios to evaluate the quality of the collected data. The results show that the performance of the 2D and 3D diffusion policies can scale with the generated dataset. Project page is https://openhumanoidgen.github.io.