VQCPC-GAN: Variable-Length Adversarial Audio Synthesis Using Vector-Quantized Contrastive Predictive Coding

TL;DR

VQCPC-GAN introduces a novel adversarial framework that enables the generation of variable-length audio by leveraging vector-quantized contrastive predictive coding tokens as conditional inputs, maintaining temporal consistency.

Contribution

The paper proposes VQCPC-GAN, a new method for variable-length audio synthesis using VQCPC tokens, which is a novel approach in adversarial audio generation.

Findings

VQCPC-GAN achieves comparable performance to strong baselines in variable-length audio synthesis.

The model maintains temporal consistency across generated audio segments.

Experimental results demonstrate the effectiveness of using VQCPC tokens as conditional inputs.

Abstract

Influenced by the field of Computer Vision, Generative Adversarial Networks (GANs) are often adopted for the audio domain using fixed-size two-dimensional spectrogram representations as the "image data". However, in the (musical) audio domain, it is often desired to generate output of variable duration. This paper presents VQCPC-GAN, an adversarial framework for synthesizing variable-length audio by exploiting Vector-Quantized Contrastive Predictive Coding (VQCPC). A sequence of VQCPC tokens extracted from real audio data serves as conditional input to a GAN architecture, providing step-wise time-dependent features of the generated content. The input noise z (characteristic in adversarial architectures) remains fixed over time, ensuring temporal consistency of global features. We evaluate the proposed model by comparing a diverse set of metrics against various strong baselines. Results show that, even though the baselines score best, VQCPC-GAN achieves comparable performance even when generating variable-length audio. Numerous sound examples are provided in the accompanying website, and we release the code for reproducibility.