Differentiable Black-box and Gray-box Modeling of Nonlinear Audio Effects

TL;DR

This paper compares black-box and gray-box neural models for nonlinear audio effects, introduces new models and datasets, and evaluates their performance across a wide range of audio devices and effects.

Contribution

It provides a comprehensive comparison of modeling approaches for nonlinear audio effects, introduces time-varying gray-box models, and releases a large open dataset for community research.

Findings

Gray-box models outperform black-box in many effects.

Proposed models effectively capture compressor, distortion, and fuzz effects.

Extensive subjective evaluation validates model performance.

Abstract

Audio effects are extensively used at every stage of audio and music content creation. The majority of differentiable audio effects modeling approaches fall into the black-box or gray-box paradigms; and most models have been proposed and applied to nonlinear effects like guitar amplifiers, overdrive, distortion, fuzz and compressor. Although a plethora of architectures have been introduced for the task at hand there is still lack of understanding on the state of the art, since most publications experiment with one type of nonlinear audio effect and a very small number of devices. In this work we aim to shed light on the audio effects modeling landscape by comparing black-box and gray-box architectures on a large number of nonlinear audio effects, identifying the most suitable for a wide range of devices. In the process, we also: introduce time-varying gray-box models and propose models for compressor, distortion and fuzz, publish a large dataset for audio effects research - ToneTwist AFx https://github.com/mcomunita/tonetwist-afx-dataset - that is also the first open to community contributions, evaluate models on a variety of metrics and conduct extensive subjective evaluation. Code https://github.com/mcomunita/nablafx and supplementary material https://github.com/mcomunita/nnlinafx-supp-material are also available.