Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models

TL;DR

This paper introduces MMHNet, a hierarchical model that enables long-duration video-to-audio generation, demonstrating the ability to generate over 5-minute audio clips from videos, surpassing prior methods.

Contribution

The paper presents MMHNet, a novel hierarchical network with non-causal Mamba, allowing models trained on short videos to generalize to longer durations without additional training.

Findings

Significantly improves long audio generation up to over 5 minutes.

Training on short videos suffices for testing on longer videos without retraining.

Achieves state-of-the-art results on long-video to audio benchmarks.

Abstract

Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.