FM Tone Transfer with Envelope Learning

TL;DR

This paper introduces Envelope Learning, a new method for Tone Transfer that improves sound diversity and transient rendering, enabling more expressive real-time musical performances.

Contribution

It proposes Envelope Learning, a novel architecture for Tone Transfer that enhances articulation and dynamic control in synthetic audio.

Findings

Accurate rendering of note beginnings and endings.

Improved sound diversity in Tone Transfer.

Real-time implementation as a VST plugin.

Abstract

Tone Transfer is a novel deep-learning technique for interfacing a sound source with a synthesizer, transforming the timbre of audio excerpts while keeping their musical form content. Due to its good audio quality results and continuous controllability, it has been recently applied in several audio processing tools. Nevertheless, it still presents several shortcomings related to poor sound diversity, and limited transient and dynamic rendering, which we believe hinder its possibilities of articulation and phrasing in a real-time performance context. In this work, we present a discussion on current Tone Transfer architectures for the task of controlling synthetic audio with musical instruments and discuss their challenges in allowing expressive performances. Next, we introduce Envelope Learning, a novel method for designing Tone Transfer architectures that map musical events using a training objective at the synthesis parameter level. Our technique can render note beginnings and endings accurately and for a variety of sounds; these are essential steps for improving musical articulation, phrasing, and sound diversity with Tone Transfer. Finally, we implement a VST plugin for real-time live use and discuss possibilities for improvement.