Differentiable Digital Signal Processing Mixture Model for Synthesis Parameter Extraction from Mixture of Harmonic Sounds

TL;DR

This paper introduces a differentiable digital signal processing mixture model that enables extraction of synthesis parameters from mixtures of harmonic sounds, improving over previous monophonic models by handling multiple sources.

Contribution

It proposes a novel DDSP mixture model that represents sound mixtures as sums of pretrained DDSP autoencoders, allowing direct estimation of individual source parameters.

Findings

High and stable performance in parameter extraction from mixtures

Outperforms straightforward source separation followed by DDSP autoencoding

Effective for mixtures of harmonic sounds

Abstract

A differentiable digital signal processing (DDSP) autoencoder is a musical sound synthesizer that combines a deep neural network (DNN) and spectral modeling synthesis. It allows us to flexibly edit sounds by changing the fundamental frequency, timbre feature, and loudness (synthesis parameters) extracted from an input sound. However, it is designed for a monophonic harmonic sound and cannot handle mixtures of harmonic sounds. In this paper, we propose a model (DDSP mixture model) that represents a mixture as the sum of the outputs of multiple pretrained DDSP autoencoders. By fitting the output of the proposed model to the observed mixture, we can directly estimate the synthesis parameters of each source. Through synthesis parameter extraction experiments, we show that the proposed method has high and stable performance compared with a straightforward method that applies the DDSP autoencoder to the signals separated by an audio source separation method.