V2Meow: Meowing to the Visual Beat via Video-to-Music Generation

TL;DR

V2Meow is a novel video-to-music generation system that learns globally aligned video-acoustic signatures from paired data, producing high-quality, style-controllable music conditioned on general visual features without domain-specific modeling.

Contribution

It introduces V2Meow, a multi-stage autoregressive model trained on diverse in-the-wild video-music pairs, capable of zero-shot high-fidelity music synthesis from videos with style control.

Findings

Outperforms existing systems in visual-audio correspondence

Generates high-quality music conditioned on general visual features

Effective style control via text prompts

Abstract

Video-to-music generation demands both a temporally localized high-quality listening experience and globally aligned video-acoustic signatures. While recent music generation models excel at the former through advanced audio codecs, the exploration of video-acoustic signatures has been confined to specific visual scenarios. In contrast, our research confronts the challenge of learning globally aligned signatures between video and music directly from paired music and videos, without explicitly modeling domain-specific rhythmic or semantic relationships. We propose V2Meow, a video-to-music generation system capable of producing high-quality music audio for a diverse range of video input types using a multi-stage autoregressive model. Trained on 5k hours of music audio clips paired with video frames mined from in-the-wild music videos, V2Meow is competitive with previous domain-specific models when evaluated in a zero-shot manner. It synthesizes high-fidelity music audio waveforms solely by conditioning on pre-trained general-purpose visual features extracted from video frames, with optional style control via text prompts. Through both qualitative and quantitative evaluations, we demonstrate that our model outperforms various existing music generation systems in terms of visual-audio correspondence and audio quality. Music samples are available at tinyurl.com/v2meow.