Neutone SDK: An Open Source Framework for Neural Audio Processing

TL;DR

Neutone SDK is an open source framework that simplifies deploying neural audio models in real-time and offline audio applications, addressing integration challenges with a unified, model-agnostic interface.

Contribution

It introduces a comprehensive, Python-based SDK that streamlines neural audio model deployment within digital audio workstations and other applications.

Findings

Enables real-time neural audio processing in DAWs

Supports diverse applications like effect emulation and timbre transfer

Adopted by researchers, educators, and artists

Abstract

Neural audio processing has unlocked novel methods of sound transformation and synthesis, yet integrating deep learning models into digital audio workstations (DAWs) remains challenging due to real-time / neural network inference constraints and the complexities of plugin development. In this paper, we introduce the Neutone SDK: an open source framework that streamlines the deployment of PyTorch-based neural audio models for both real-time and offline applications. By encapsulating common challenges such as variable buffer sizes, sample rate conversion, delay compensation, and control parameter handling within a unified, model-agnostic interface, our framework enables seamless interoperability between neural models and host plugins while allowing users to work entirely in Python. We provide a technical overview of the interfaces needed to accomplish this, as well as the corresponding SDK implementations. We also demonstrate the SDK's versatility across applications such as audio effect emulation, timbre transfer, and sample generation, as well as its adoption by researchers, educators, companies, and artists alike. The Neutone SDK is available at https://github.com/Neutone/neutone_sdk