Conditional End-to-End Audio Transforms

TL;DR

This paper introduces a versatile end-to-end audio transformation model that can convert speech and music into different styles by conditioning on speaker identities and instruments, using a fully-differentiable neural network architecture.

Contribution

It presents a novel, unified sequence-to-sequence model capable of transforming audio styles for speech and music with minimal post-processing.

Findings

Achieves realistic audio transformations with minimal post-processing.

Successfully separates speaker and instrument properties from content.

Performs competitively on standard datasets.

Abstract

We present an end-to-end method for transforming audio from one style to another. For the case of speech, by conditioning on speaker identities, we can train a single model to transform words spoken by multiple people into multiple target voices. For the case of music, we can specify musical instruments and achieve the same result. Architecturally, our method is a fully-differentiable sequence-to-sequence model based on convolutional and hierarchical recurrent neural networks. It is designed to capture long-term acoustic dependencies, requires minimal post-processing, and produces realistic audio transforms. Ablation studies confirm that our model can separate speaker and instrument properties from acoustic content at different receptive fields. Empirically, our method achieves competitive performance on community-standard datasets.