RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual Dexterous Robot Hands

TL;DR

This paper introduces RP1M, a large-scale dataset of over one million robot piano playing trajectories, to advance imitation learning for dexterous bi-manual robot hands in complex, contact-rich tasks.

Contribution

The creation of the RP1M dataset and the formulation of finger placement as an optimal transport problem are novel contributions that facilitate scalable imitation learning for robot piano playing.

Findings

Imitation learning approaches achieve state-of-the-art performance using RP1M.

Optimal transport formulation enables automatic annotation of unlabeled songs.

The dataset supports scalable training for complex bi-manual manipulation tasks.

Abstract

It has been a long-standing research goal to endow robot hands with human-level dexterity. Bi-manual robot piano playing constitutes a task that combines challenges from dynamic tasks, such as generating fast while precise motions, with slower but contact-rich manipulation problems. Although reinforcement learning based approaches have shown promising results in single-task performance, these methods struggle in a multi-song setting. Our work aims to close this gap and, thereby, enable imitation learning approaches for robot piano playing at scale. To this end, we introduce the Robot Piano 1 Million (RP1M) dataset, containing bi-manual robot piano playing motion data of more than one million trajectories. We formulate finger placements as an optimal transport problem, thus, enabling automatic annotation of vast amounts of unlabeled songs. Benchmarking existing imitation learning approaches shows that such approaches reach state-of-the-art robot piano playing performance by leveraging RP1M.