Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation

TL;DR

Video-Robin is a novel text-conditioned video-to-music generation model that combines autoregressive planning with diffusion-based synthesis for high-quality, controllable, and semantically aligned music creation from videos.

Contribution

It introduces a new model that integrates autoregressive planning with diffusion synthesis, enabling semantic control and improved quality in video-to-music generation.

Findings

Outperforms video-only baselines on multiple benchmarks.

Achieves 2.21x faster inference than state-of-the-art methods.

Enables fine-grained control without sacrificing audio realism.

Abstract

Video-to-music (V2M) is the fundamental task of creating background music for an input video. Recent V2M models achieve audiovisual alignment by typically relying on visual conditioning alone and provide limited semantic and stylistic controllability to the end user. In this paper, we present Video-Robin, a novel text-conditioned video-to-music generation model that enables fast, high-quality, semantically aligned music generation for video content. To balance musical fidelity and semantic understanding, Video-Robin integrates autoregressive planning with diffusion-based synthesis. Specifically, an autoregressive module models global structure by semantically aligning visual and textual inputs to produce high-level music latents. These latents are subsequently refined into coherent, high-fidelity music using local Diffusion Transformers. By factoring semantically driven planning into diffusion-based synthesis, Video-Robin enables fine-grained creator control without sacrificing audio realism. Our proposed model outperforms baselines that solely accept video input and additional feature conditioned baselines on both in-distribution and out-of-distribution benchmarks with a 2.21x speed in inference compared to SOTA. We will open-source everything upon paper acceptance.