Echo-Reconstruction: Audio-Augmented 3D Scene Reconstruction
Justin Wilson, Nicholas Rewkowski, Ming C. Lin, Henry Fuchs

TL;DR
This paper introduces Echo-Reconstruction, an audio-visual method that leverages sound reflections to improve 3D scene reconstruction, especially around reflective and textureless surfaces, enhancing visual and audio fidelity in AR/VR applications.
Contribution
It presents a novel audio-visual neural network approach that uses sound reflections to improve depth estimation and scene reconstruction involving challenging surfaces.
Findings
High accuracy in material classification
Effective depth estimation around reflective surfaces
Significant improvement in 3D scene quality
Abstract
Reflective and textureless surfaces such as windows, mirrors, and walls can be a challenge for object and scene reconstruction. These surfaces are often poorly reconstructed and filled with depth discontinuities and holes, making it difficult to cohesively reconstruct scenes that contain these planar discontinuities. We propose Echoreconstruction, an audio-visual method that uses the reflections of sound to aid in geometry and audio reconstruction for virtual conferencing, teleimmersion, and other AR/VR experience. The mobile phone prototype emits pulsed audio, while recording video for RGB-based 3D reconstruction and audio-visual classification. Reflected sound and images from the video are input into our audio (EchoCNN-A) and audio-visual (EchoCNN-AV) convolutional neural networks for surface and sound source detection, depth estimation, and material classification. The inferences…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMusic and Audio Processing · Speech and Audio Processing · Hearing Loss and Rehabilitation
