Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks

TL;DR

This paper introduces the CoP Benchmark Dataset, a comprehensive multimodal benchmark with detailed annotations and evaluation tools, aimed at advancing high-quality video-guided piano music generation.

Contribution

It presents the first open-source, multimodal benchmark with step-by-step Chain-of-Perform guidance for evaluating video-to-piano music generation methods.

Findings

Provides detailed multimodal annotations for precise alignment.

Offers a versatile evaluation framework for various generation tasks.

Includes a publicly available dataset and ongoing leaderboard.

Abstract

Generating high-quality piano audio from video requires precise synchronization between visual cues and musical output, ensuring accurate semantic and temporal alignment.However, existing evaluation datasets do not fully capture the intricate synchronization required for piano music generation. A comprehensive benchmark is essential for two primary reasons: (1) existing metrics fail to reflect the complexity of video-to-piano music interactions, and (2) a dedicated benchmark dataset can provide valuable insights to accelerate progress in high-quality piano music generation. To address these challenges, we introduce the CoP Benchmark Dataset-a fully open-sourced, multimodal benchmark designed specifically for video-guided piano music generation. The proposed Chain-of-Perform (CoP) benchmark offers several compelling features: (1) detailed multimodal annotations, enabling precise semantic and temporal alignment between video content and piano audio via step-by-step Chain-of-Perform guidance; (2) a versatile evaluation framework for rigorous assessment of both general-purpose and specialized video-to-piano generation tasks; and (3) full open-sourcing of the dataset, annotations, and evaluation protocols. The dataset is publicly available at https://github.com/acappemin/Video-to-Audio-and-Piano, with a continuously updated leaderboard to promote ongoing research in this domain.