Real-time Timbre Remapping with Differentiable DSP
Jordie Shier, Charalampos Saitis, Andrew Robertson, and Andrew, McPherson

TL;DR
This paper introduces a real-time method for timbre remapping using differentiable digital signal processing, enabling nuanced control of synthesizer parameters based on input timbral features, demonstrated with snare drum to TR-808 synthesis.
Contribution
It presents a novel differentiable DSP approach with a feature difference loss for direct optimization of timbral control parameters in real-time.
Findings
Effective real-time timbre remapping demonstrated
Subtle timbral modulations captured and translated
Applicable to diverse musical contexts
Abstract
Timbre is a primary mode of expression in diverse musical contexts. However, prevalent audio-driven synthesis methods predominantly rely on pitch and loudness envelopes, effectively flattening timbral expression from the input. Our approach draws on the concept of timbre analogies and investigates how timbral expression from an input signal can be mapped onto controls for a synthesizer. Leveraging differentiable digital signal processing, our method facilitates direct optimization of synthesizer parameters through a novel feature difference loss. This loss function, designed to learn relative timbral differences between musical events, prioritizes the subtleties of graded timbre modulations within phrases, allowing for meaningful translations in a timbre space. Using snare drum performances as a case study, where timbral expression is central, we demonstrate real-time timbre remapping…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsWood Treatment and Properties · Structural Engineering and Vibration Analysis · Infrastructure Maintenance and Monitoring
