Upsampling artifacts in neural audio synthesis

TL;DR

This paper investigates the causes of upsampling artifacts in neural audio synthesis, identifying key sources like filtering issues and spectral replicas, and compares different upsampling methods to mitigate these artifacts.

Contribution

It is the first to analyze upsampling artifacts in audio from a signal processing perspective and compares various upsampling layers for artifact reduction.

Findings

Nearest neighbor upsampling reduces artifacts compared to transposed convolutions.

Problematic upsampling operators introduce tonal and filtering artifacts.

Spectral replicas emerge during upsampling, affecting audio quality.

Abstract

A number of recent advances in neural audio synthesis rely on upsampling layers, which can introduce undesired artifacts. In computer vision, upsampling artifacts have been studied and are known as checkerboard artifacts (due to their characteristic visual pattern). However, their effect has been overlooked so far in audio processing. Here, we address this gap by studying this problem from the audio signal processing perspective. We first show that the main sources of upsampling artifacts are: (i) the tonal and filtering artifacts introduced by problematic upsampling operators, and (ii) the spectral replicas that emerge while upsampling. We then compare different upsampling layers, showing that nearest neighbor upsamplers can be an alternative to the problematic (but state-of-the-art) transposed and subpixel convolutions which are prone to introduce tonal artifacts.