World In Your Hands: A Large-Scale and Open-Source Ecosystem for Learning Human-Centric Manipulation in the Wild

TL;DR

The paper presents WIYH, a comprehensive open-source ecosystem with extensive real-world human manipulation data, hardware, and benchmarks to enhance robotic manipulation and human-centric learning.

Contribution

It introduces a large-scale, multimodal dataset and hardware tools for real-world human manipulation, enabling improved robotic learning in diverse scenarios.

Findings

Data integration boosts robotic success rates from 8% to 60%.

The ecosystem supports diverse manipulation skills in real-world settings.

Open-source hardware and data facilitate research in human-centric robotics.

Abstract

We introduce World In Your Hands (WIYH), a large-scale open-source ecosystem comprising over 1,000 hours of human manipulation data collected in-the-wild with millimeter-scale motion accuracy. Specifically, WIYH includes (1) the Oracle Suite, a wearable data collection kit with an auto-labeling pipeline for accurate motion capture; (2) the WIYH Dataset, featuring over 1,000 hours of multimodal manipulation data across hundreds of skills in diverse real-world scenarios; and (3) extensive annotations and benchmarks supporting tasks from perception to action. Furthermore, experiments based on the WIYH ecosystem show that integrating WIYH's human-centric data improves robotic manipulation success rates from 8% to 60% in cluttered scenes. World In Your Hands provides a foundation for advancing human-centric data collection and cross-embodiment policy learning. All data and hardware design will be open-source.