Audio inpainting with generative adversarial network

TL;DR

This paper explores using a novel Wasserstein GAN architecture for long-range audio inpainting, demonstrating improved quality and frequency content reconstruction over classical models across different instruments and contexts.

Contribution

Introduces a new WGAN architecture with short- and long-range border handling for improved long-range audio inpainting.

Findings

Proposed model outperforms classical WGAN in audio inpainting quality.

Better reconstruction of high-frequency content.

More effective for instruments with lower frequency spectra.

Abstract

We study the ability of Wasserstein Generative Adversarial Network (WGAN) to generate missing audio content which is, in context, (statistically similar) to the sound and the neighboring borders. We deal with the challenge of audio inpainting long range gaps (500 ms) using WGAN models. We improved the quality of the inpainting part using a new proposed WGAN architecture that uses a short-range and a long-range neighboring borders compared to the classical WGAN model. The performance was compared with two different audio instruments (piano and guitar) and on virtuoso pianists together with a string orchestra. The objective difference grading (ODG) was used to evaluate the performance of both architectures. The proposed model outperforms the classical WGAN model and improves the reconstruction of high-frequency content. Further, we got better results for instruments where the frequency spectrum is mainly in the lower range where small noises are less annoying for human ear and the inpainting part is more perceptible. Finally, we could show that better test results for audio dataset were reached where a particular instrument is accompanist by other instruments if we train the network only on this particular instrument neglecting the other instruments.