Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes

TL;DR

This paper introduces a method to generate realistic sounds of human hand interactions with 3D scenes by training a flow model on recorded action-sound pairs, enabling interactive sound prediction for virtual environments.

Contribution

The paper presents a novel approach to synthesize plausible interaction sounds in 3D scenes using a rectified flow model trained on hand action and sound data.

Findings

Generated sounds accurately reflect material properties.

Sounds are often indistinguishable from real recordings.

Method enables interactive sound prediction in 3D environments.

Abstract

We study the problem of making 3D scene reconstructions interactive by asking the following question: can we predict the sounds of human hands physically interacting with a scene? First, we record a video of a human manipulating objects within a 3D scene using their hands. We then use these action-sound pairs to train a rectified flow model to map 3D hand trajectories to their corresponding audio. At test time, a user can query the model for other actions, parameterized as sequences of hand poses, to estimate their corresponding sounds. In our experiments, we find that our generated sounds accurately convey material properties and actions, and that they are often indistinguishable to human observers from real sounds. Project page: https://www.yimingdou.com/hearing_hands/