Listen and Move: Improving GANs Coherency in Agnostic Sound-to-Video Generation

TL;DR

This paper introduces novel techniques to improve the quality and temporal consistency of sound-to-video generative adversarial networks, addressing the challenge of smooth video dynamics in audiovisual synthesis.

Contribution

It proposes three innovative features—sound routing, multi-scale recurrent sound analysis, and a new convolutional layer—that enhance image quality and temporal coherence in sound-to-video GANs.

Findings

Enhanced video quality and coherency demonstrated

Improved temporal dynamics in generated videos

Baseline architecture performance significantly increased

Abstract

Deep generative models have demonstrated the ability to create realistic audiovisual content, sometimes driven by domains of different nature. However, smooth temporal dynamics in video generation is a challenging problem. This work focuses on generic sound-to-video generation and proposes three main features to enhance both image quality and temporal coherency in generative adversarial models: a triple sound routing scheme, a multi-scale residual and dilated recurrent network for extended sound analysis, and a novel recurrent and directional convolutional layer for video prediction. Each of the proposed features improves, in both quality and coherency, the baseline neural architecture typically used in the SoTA, with the video prediction layer providing an extra temporal refinement.