Automatic multitrack mixing with a differentiable mixing console of neural audio effects

TL;DR

This paper introduces a novel deep learning model for automatic multitrack mixing that learns from real-world data at the waveform level, producing human-readable parameters and outperforming baselines in perceptual evaluations.

Contribution

It presents the first waveform-level multitrack mixing model that is data-efficient, permutation invariant, and produces adjustable, human-readable mixing parameters.

Findings

Outperforms baseline approaches in perceptual evaluations

Can be trained with limited data and arbitrary number of sources

Produces human-readable mixing parameters for manual refinement

Abstract

Applications of deep learning to automatic multitrack mixing are largely unexplored. This is partly due to the limited available data, coupled with the fact that such data is relatively unstructured and variable. To address these challenges, we propose a domain-inspired model with a strong inductive bias for the mixing task. We achieve this with the application of pre-trained sub-networks and weight sharing, as well as with a sum/difference stereo loss function. The proposed model can be trained with a limited number of examples, is permutation invariant with respect to the input ordering, and places no limit on the number of input sources. Furthermore, it produces human-readable mixing parameters, allowing users to manually adjust or refine the generated mix. Results from a perceptual evaluation involving audio engineers indicate that our approach generates mixes that outperform baseline approaches. To the best of our knowledge, this work demonstrates the first approach in learning multitrack mixing conventions from real-world data at the waveform level, without knowledge of the underlying mixing parameters.