DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks

TL;DR

DiffSound is a novel differentiable framework for physics-based modal sound synthesis and inverse problems, enabling accurate sound reproduction and physical property estimation from real-world recordings.

Contribution

It introduces a differentiable sound rendering pipeline with an implicit shape representation and high-order finite element analysis for the first time.

Findings

Accurately reproduces target sounds in experiments.

Enables physical parameter estimation and shape reasoning.

Demonstrates effectiveness in diverse sound synthesis tasks.

Abstract

Accurately estimating and simulating the physical properties of objects from real-world sound recordings is of great practical importance in the fields of vision, graphics, and robotics. However, the progress in these directions has been limited -- prior differentiable rigid or soft body simulation techniques cannot be directly applied to modal sound synthesis due to the high sampling rate of audio, while previous audio synthesizers often do not fully model the accurate physical properties of the sounding objects. We propose DiffSound, a differentiable sound rendering framework for physics-based modal sound synthesis, which is based on an implicit shape representation, a new high-order finite element analysis module, and a differentiable audio synthesizer. Our framework can solve a wide range of inverse problems thanks to the differentiability of the entire pipeline, including physical parameter estimation, geometric shape reasoning, and impact position prediction. Experimental results demonstrate the effectiveness of our approach, highlighting its ability to accurately reproduce the target sound in a physics-based manner. DiffSound serves as a valuable tool for various sound synthesis and analysis applications.