The Sound of Water: Inferring Physical Properties from Pouring Liquids

TL;DR

This paper demonstrates that physical properties of liquids and containers can be inferred from pouring sounds using a physics-inspired learning approach, with a new dataset and strong real-world generalization.

Contribution

It introduces a novel method combining theory, simulation, and real data to infer physical properties from audio-visual cues in pouring activities.

Findings

Model accurately infers liquid level and container shape from sound.

Strong generalization to different containers and real-world videos.

New large dataset of pouring videos for research.

Abstract

We study the connection between audio-visual observations and the underlying physics of a mundane yet intriguing everyday activity: pouring liquids. Given only the sound of liquid pouring into a container, our objective is to automatically infer physical properties such as the liquid level, the shape and size of the container, the pouring rate and the time to fill. To this end, we: (i) show in theory that these properties can be determined from the fundamental frequency (pitch); (ii) train a pitch detection model with supervision from simulated data and visual data with a physics-inspired objective; (iii) introduce a new large dataset of real pouring videos for a systematic study; (iv) show that the trained model can indeed infer these physical properties for real data; and finally, (v) we demonstrate strong generalization to various container shapes, other datasets, and in-the-wild YouTube videos. Our work presents a keen understanding of a narrow yet rich problem at the intersection of acoustics, physics, and learning. It opens up applications to enhance multisensory perception in robotic pouring.