Sonicmesh: Enhancing 3D Human Mesh Reconstruction in Vision-Impaired Environments With Acoustic Signals

TL;DR

SonicMesh leverages acoustic signals combined with RGB images to improve 3D human mesh reconstruction in environments with poor lighting, occlusions, or privacy constraints, addressing limitations of traditional RGB-based methods.

Contribution

Introduces SonicMesh, a novel method integrating acoustic signals with RGB images using modified HRNet and feature embedding for robust 3D human mesh reconstruction.

Findings

Accurately reconstructs 3D human mesh in occluded and non-line-of-sight environments

Effectively combines acoustic signals with RGB data for enhanced robustness

Demonstrates high accuracy in challenging visual conditions

Abstract

3D Human Mesh Reconstruction (HMR) from 2D RGB images faces challenges in environments with poor lighting, privacy concerns, or occlusions. These weaknesses of RGB imaging can be complemented by acoustic signals, which are widely available, easy to deploy, and capable of penetrating obstacles. However, no existing methods effectively combine acoustic signals with RGB data for robust 3D HMR. The primary challenges include the low-resolution images generated by acoustic signals and the lack of dedicated processing backbones. We introduce SonicMesh, a novel approach combining acoustic signals with RGB images to reconstruct 3D human mesh. To address the challenges of low resolution and the absence of dedicated processing backbones in images generated by acoustic signals, we modify an existing method, HRNet, for effective feature extraction. We also integrate a universal feature embedding technique to enhance the precision of cross-dimensional feature alignment, enabling SonicMesh to achieve high accuracy. Experimental results demonstrate that SonicMesh accurately reconstructs 3D human mesh in challenging environments such as occlusions, non-line-of-sight scenarios, and poor lighting.