TL;DR
This paper introduces a neural network-based method to predict and synthesize realistic sounds from silent videos of objects being hit or scratched, revealing physical properties and actions.
Contribution
It presents a novel algorithm that predicts sound features from videos and synthesizes realistic audio, advancing multimodal understanding of physical interactions in visual scenes.
Findings
Predicted sounds can fool participants in a realism test.
Synthesized sounds convey key material and interaction information.
The approach demonstrates effective cross-modal prediction from video to sound.
Abstract
Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Visually Indicated Sounds | Two Minute Papers #79· youtube
Visually Indicated Sounds· youtube
