Learning to Highlight Audio by Watching Movies

TL;DR

This paper introduces a transformer-based multimodal framework for visually-guided acoustic highlighting, using a new movie-based dataset and pseudo-data generation to improve audio-visual harmony in content creation.

Contribution

It presents a novel task of audio highlighting guided by video, along with a new dataset and a pseudo-data generation process for training and evaluation.

Findings

Our model outperforms baselines in quantitative metrics.

Subjective evaluations favor our approach for audio-visual harmony.

Different contextual guidance impacts highlighting effectiveness.

Abstract

Recent years have seen a significant increase in video content creation and consumption. Crafting engaging content requires the careful curation of both visual and audio elements. While visual cue curation, through techniques like optimal viewpoint selection or post-editing, has been central to media production, its natural counterpart, audio, has not undergone equivalent advancements. This often results in a disconnect between visual and acoustic saliency. To bridge this gap, we introduce a novel task: visually-guided acoustic highlighting, which aims to transform audio to deliver appropriate highlighting effects guided by the accompanying video, ultimately creating a more harmonious audio-visual experience. We propose a flexible, transformer-based multimodal framework to solve this task. To train our model, we also introduce a new dataset -- the muddy mix dataset, leveraging the meticulous audio and video crafting found in movies, which provides a form of free supervision. We develop a pseudo-data generation process to simulate poorly mixed audio, mimicking real-world scenarios through a three-step process -- separation, adjustment, and remixing. Our approach consistently outperforms several baselines in both quantitative and subjective evaluation. We also systematically study the impact of different types of contextual guidance and difficulty levels of the dataset. Our project page is here: https://wikichao.github.io/VisAH/.