NoiseBandNet: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks

TL;DR

NoiseBandNet is a novel neural synthesis architecture that filters white noise through a filterbank to generate and control diverse, inharmonic sound effects with high temporal and spectral resolution, surpassing previous DDSP methods.

Contribution

It introduces NoiseBandNet, a flexible, controllable neural sound effect synthesizer that models inharmonic sounds without harmonic assumptions, improving over existing DDSP-based systems.

Findings

NoiseBandNet outperforms four DDSP variants in 9 of 10 evaluation metrics.

It effectively models various sound effects like footsteps and thunderstorm.

The system enables creative sound variations and user-defined control curves.

Abstract

Controllable neural audio synthesis of sound effects is a challenging task due to the potential scarcity and spectro-temporal variance of the data. Differentiable digital signal processing (DDSP) synthesisers have been successfully employed to model and control musical and harmonic signals using relatively limited data and computational resources. Here we propose NoiseBandNet, an architecture capable of synthesising and controlling sound effects by filtering white noise through a filterbank, thus going further than previous systems that make assumptions about the harmonic nature of sounds. We evaluate our approach via a series of experiments, modelling footsteps, thunderstorm, pottery, knocking, and metal sound effects. Comparing NoiseBandNet audio reconstruction capabilities to four variants of the DDSP-filtered noise synthesiser, NoiseBandNet scores higher in nine out of ten evaluation categories, establishing a flexible DDSP method for generating time-varying, inharmonic sound effects of arbitrary length with both good time and frequency resolution. Finally, we introduce some potential creative uses of NoiseBandNet, by generating variations, performing loudness transfer, and by training it on user-defined control curves.