Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videos

TL;DR

This paper introduces AV-LDM, a novel ambient-aware model for generating realistic action sounds from egocentric videos, effectively disentangling foreground and background sounds and enabling controllable, semantically aligned audio synthesis.

Contribution

The paper presents AV-LDM, the first model to focus on faithful video-to-audio generation from uncurated in-the-wild videos, with a new audio-conditioning mechanism and retrieval-augmented generation.

Findings

Outperforms existing methods in generating action sounds.

Enables controllable ambient sound generation.

Shows potential for application to computer graphics clips.

Abstract

Generating realistic audio for human actions is important for many applications, such as creating sound effects for films or virtual reality games. Existing approaches implicitly assume total correspondence between the video and audio during training, yet many sounds happen off-screen and have weak to no correspondence with the visuals -- resulting in uncontrolled ambient sounds or hallucinations at test time. We propose a novel ambient-aware audio generation model, AV-LDM. We devise a novel audio-conditioning mechanism to learn to disentangle foreground action sounds from the ambient background sounds in in-the-wild training videos. Given a novel silent video, our model uses retrieval-augmented generation to create audio that matches the visual content both semantically and temporally. We train and evaluate our model on two in-the-wild egocentric video datasets, Ego4D and EPIC-KITCHENS, and we introduce Ego4D-Sounds -- 1.2M curated clips with action-audio correspondence. Our model outperforms an array of existing methods, allows controllable generation of the ambient sound, and even shows promise for generalizing to computer graphics game clips. Overall, our approach is the first to focus video-to-audio generation faithfully on the observed visual content despite training from uncurated clips with natural background sounds.