Comparing Representations for Audio Synthesis Using Generative Adversarial Networks

TL;DR

This paper compares various audio representations for GAN-based audio synthesis, finding that complex-valued and certain time-frequency features produce the best quality and efficiency.

Contribution

It systematically evaluates different audio signal representations for GANs, highlighting the effectiveness of complex-valued and specific time-frequency features for synthesis.

Findings

Complex-valued and magnitude/Instantaneous Frequency of STFT perform best.

These representations enable faster generation and inversion.

The study provides a benchmark for audio representation choices in GANs.

Abstract

In this paper, we compare different audio signal representations, including the raw audio waveform and a variety of time-frequency representations, for the task of audio synthesis with Generative Adversarial Networks (GANs). We conduct the experiments on a subset of the NSynth dataset. The architecture follows the benchmark Progressive Growing Wasserstein GAN. We perform experiments both in a fully non-conditional manner as well as conditioning the network on the pitch information. We quantitatively evaluate the generated material utilizing standard metrics for assessing generative models, and compare training and sampling times. We show that complex-valued as well as the magnitude and Instantaneous Frequency of the Short-Time Fourier Transform achieve the best results, and yield fast generation and inversion times. The code for feature extraction, training and evaluating the model is available online.