Event2Audio: Event-Based Optical Vibration Sensing

TL;DR

This paper introduces Event2Audio, a method that uses event-based cameras to passively and actively sense vibrations for audio recovery, achieving real-time performance and robustness to environmental distortions.

Contribution

It advances vibration sensing by leveraging event-based cameras for faster, more efficient audio recovery from vibrations, even with multiple sources and environmental noise.

Findings

Achieves near real-time audio reconstruction from vibrations.

Handles multiple simultaneous vibration sources effectively.

Maintains high reconstruction quality despite environmental distortions.

Abstract

Small vibrations observed in video can unveil information beyond what is visual, such as sound and material properties. It is possible to passively record these vibrations when they are visually perceptible, or actively amplify their visual contribution with a laser beam when they are not perceptible. In this paper, we improve upon the active sensing approach by leveraging event-based cameras, which are designed to efficiently capture fast motion. We demonstrate our method experimentally by recovering audio from vibrations, even for multiple simultaneous sources, and in the presence of environmental distortions. Our approach matches the state-of-the-art reconstruction quality at much faster speeds, approaching real-time processing.