Dexterous Robotic Piano Playing at Scale
Le Chen, Yi Zhao, Jan Schneider, Quankai Gao, Simon Guist, Cheng Qian, Juho Kannala, Bernhard Sch\"olkopf, Joni Pajarinen, and Dieter B\"uchler

TL;DR
This paper introduces OmniPianist, a scalable robotic system capable of performing nearly a thousand piano pieces autonomously, using novel optimal transport-based fingering, large-scale reinforcement learning, and imitation learning techniques.
Contribution
The paper presents the first scalable robotic piano playing system that learns without demonstrations, combining optimal transport fingering, large-scale RL, and transformer-based imitation learning.
Findings
Successfully performed nearly 1,000 music pieces.
Trained over 2,000 specialized agents with 1 million trajectories.
Demonstrated effectiveness and scalability through experiments.
Abstract
Endowing robot hands with human-level dexterity has been a long-standing goal in robotics. Bimanual robotic piano playing represents a particularly challenging task: it is high-dimensional, contact-rich, and requires fast, precise control. We present OmniPianist, the first agent capable of performing nearly one thousand music pieces via scalable, human-demonstration-free learning. Our approach is built on three core components. First, we introduce an automatic fingering strategy based on Optimal Transport (OT), allowing the agent to autonomously discover efficient piano-playing strategies from scratch without demonstrations. Second, we conduct large-scale Reinforcement Learning (RL) by training more than 2,000 agents, each specialized in distinct music pieces, and aggregate their experience into a dataset named RP1M++, consisting of over one million trajectories for robotic piano…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMusic Technology and Sound Studies · Robot Manipulation and Learning · Advanced Sensor and Energy Harvesting Materials
