Dexterous Robotic Piano Playing at Scale

TL;DR

This paper introduces OmniPianist, a scalable robotic system capable of performing nearly a thousand piano pieces autonomously, using novel optimal transport-based fingering, large-scale reinforcement learning, and imitation learning techniques.

Contribution

The paper presents the first scalable robotic piano playing system that learns without demonstrations, combining optimal transport fingering, large-scale RL, and transformer-based imitation learning.

Findings

Successfully performed nearly 1,000 music pieces.

Trained over 2,000 specialized agents with 1 million trajectories.

Demonstrated effectiveness and scalability through experiments.

Abstract

Endowing robot hands with human-level dexterity has been a long-standing goal in robotics. Bimanual robotic piano playing represents a particularly challenging task: it is high-dimensional, contact-rich, and requires fast, precise control. We present OmniPianist, the first agent capable of performing nearly one thousand music pieces via scalable, human-demonstration-free learning. Our approach is built on three core components. First, we introduce an automatic fingering strategy based on Optimal Transport (OT), allowing the agent to autonomously discover efficient piano-playing strategies from scratch without demonstrations. Second, we conduct large-scale Reinforcement Learning (RL) by training more than 2,000 agents, each specialized in distinct music pieces, and aggregate their experience into a dataset named RP1M++, consisting of over one million trajectories for robotic piano playing. Finally, we employ a Flow Matching Transformer to leverage RP1M++ through large-scale imitation learning, resulting in the OmniPianist agent capable of performing a wide range of musical pieces. Extensive experiments and ablation studies highlight the effectiveness and scalability of our approach, advancing dexterous robotic piano playing at scale.