MUSIC: Learning Muscle-Driven Dexterous Hand Control
Pei Xu, Yufei Ye, Shuchun Sun, Yu Ding, Elizabeth Schumann, C. Karen Liu

TL;DR
This paper introduces a hierarchical, muscle-driven control system for physics-based hand models, enabling realistic and precise piano playing with novel compositions, validated through diverse musical styles.
Contribution
It presents a novel hierarchical control architecture combining reinforcement learning, VAEs, and a new biomechanical hand model for realistic, precise, and versatile musical performance.
Findings
Achieves state-of-the-art piano playing performance in physics-based models.
Demonstrates superior biomechanical stability and tracking accuracy.
Produces physiologically plausible muscle activation patterns aligned with EMG data.
Abstract
We present a data-driven approach for physics-based, muscle-driven dexterous control that enables musculoskeletal hands to perform precise piano playing for novel pieces of music outside the reference dataset. Our approach combines high-frequency muscle-level control with low-frequency latent-space coordination in a hierarchical architecture. At the low level, general single-hand policies are trained via reinforcement learning to generate dynamic muscle-tendon activations while tracking trajectories from a large reference motion dataset. The resulting tracking policies are then distilled into variational autoencoder (VAE) models, yielding smooth and structured latent spaces that abstract away low-level muscle dynamics. For the high level, we train piece-specific policies to operate in this latent space, coordinating bimanual motions based on specific goals, denoted by note events…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
