VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement

TL;DR

VarietySound introduces a novel method for video-to-sound generation that allows control over the sound's timbre by disentangling audio into components and using reference audio, improving quality and timbre accuracy.

Contribution

This paper presents a new approach for controllable video-to-sound generation through unsupervised disentanglement of audio components, enabling timbre control based on reference audio.

Findings

High-quality sound generation with good video synchronization

Effective timbre control matching reference audio

Superior results on the VAS dataset

Abstract

Video to sound generation aims to generate realistic and natural sound given a video input. However, previous video-to-sound generation methods can only generate a random or average timbre without any controls or specializations of the generated sound timbre, leading to the problem that people cannot obtain the desired timbre under these methods sometimes. In this paper, we pose the task of generating sound with a specific timbre given a video input and a reference audio sample. To solve this task, we disentangle each target sound audio into three components: temporal information, acoustic information, and background information. We first use three encoders to encode these components respectively: 1) a temporal encoder to encode temporal information, which is fed with video frames since the input video shares the same temporal information as the original audio; 2) an acoustic encoder to encode timbre information, which takes the original audio as input and discards its temporal information by a temporal-corrupting operation; and 3) a background encoder to encode the residual or background sound, which uses the background part of the original audio as input. To make the generated result achieve better quality and temporal alignment, we also adopt a mel discriminator and a temporal discriminator for the adversarial training. Our experimental results on the VAS dataset demonstrate that our method can generate high-quality audio samples with good synchronization with events in video and high timbre similarity with the reference audio.