Combining Genre Classification and Harmonic-Percussive Features with Diffusion Models for Music-Video Generation

TL;DR

This paper introduces a diffusion model-based method for generating music visualisers that combines audio features and user artwork, achieving high audio-visual synchrony and broad application potential.

Contribution

It presents a novel diffusion model approach integrating genre classification, artistic style retrieval, and audio-visual synchronization metrics for music-video generation.

Findings

Higher Audio-Visual Synchrony (AVS) scores with the proposed method

Effective interpolation of frames using diffusion models

Potential applications in music and film industries

Abstract

This study presents a novel method for generating music visualisers using diffusion models, combining audio input with user-selected artwork. The process involves two main stages: image generation and video creation. First, music captioning and genre classification are performed, followed by the retrieval of artistic style descriptions. A diffusion model then generates images based on the user's input image and the derived artistic style descriptions. The video generation stage utilises the same diffusion model to interpolate frames, controlled by audio energy vectors derived from key musical features of harmonics and percussives. The method demonstrates promising results across various genres, and a new metric, Audio-Visual Synchrony (AVS), is introduced to quantitatively evaluate the synchronisation between visual and audio elements. Comparative analysis shows significantly higher AVS values for videos generated using the proposed method with audio energy vectors, compared to linear interpolation. This approach has potential applications in diverse fields, including independent music video creation, film production, live music events, and enhancing audio-visual experiences in public spaces.