From Score to Sound: An End-to-End MIDI-to-Motion Pipeline for Robotic Cello Performance

TL;DR

This paper presents an end-to-end MIDI-to-motion pipeline enabling a robot cellist to perform with human-like sound and expression, using collision-aware control and a novel benchmarking approach involving human evaluations.

Contribution

It introduces a novel MIDI-to-motion pipeline for robotic cello, eliminating the need for motion capture, and establishes a new benchmark with human evaluations and labeled performance data.

Findings

Robot achieves human-like sound quality

Introduces a new benchmark with human participant evaluations

Provides labeled robotic performance data for research

Abstract

Robot musicians require precise control to obtain proper note accuracy, sound quality, and musical expression. Performance of string instruments, such as violin and cello, presents a significant challenge due to the precise control required over bow angle and pressure to produce the desired sound. While prior robotic cellists focus on accurate bowing trajectories, these works often rely on expensive motion capture techniques, and fail to sightread music in a human-like way. We propose a novel end-to-end MIDI score to robotic motion pipeline which converts musical input directly into collision-aware bowing motions for a UR5e robot cellist. Through use of Universal Robot Freedrive feature, our robotic musician can achieve human-like sound without the need for motion capture. Additionally, this work records live joint data via Real-Time Data Exchange (RTDE) as the robot plays, providing labeled robotic playing data from a collection of five standard pieces to the research community. To demonstrate the effectiveness of our method in comparison to human performers, we introduce the Musical Turing Test, in which a collection of 132 human participants evaluate our robot's performance against a human baseline. Human reference recordings are also released, enabling direct comparison for future studies. This evaluation technique establishes the first benchmark for robotic cello performance. Finally, we outline a residual reinforcement learning methodology to improve upon baseline robotic controls, highlighting future opportunities for improved string-crossing efficiency and sound quality.