ImmFusion: Robust mmWave-RGB Fusion for 3D Human Body Reconstruction in All Weather Conditions

TL;DR

ImmFusion is a novel mmWave-RGB fusion method that robustly reconstructs 3D human bodies in all weather conditions by effectively combining data from both modalities using a Transformer-based approach.

Contribution

It introduces the first mmWave-RGB fusion framework for all-weather 3D human reconstruction, utilizing a Transformer module for dynamic token fusion.

Findings

Outperforms state-of-the-art fusion methods in accuracy.

Demonstrates robustness across diverse weather conditions.

Efficiently utilizes multimodal data for improved reconstruction.

Abstract

3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. Complementary, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for robust all-weather 3D human reconstruction is still an open challenge, given the sparse nature of mmWave and the vulnerability of RGB images. In this paper, we present ImmFusion, the first mmWave-RGB fusion solution to reconstruct 3D human bodies in all weather conditions robustly. Specifically, our ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments on a large-scale dataset, mmBody, captured in various environments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve a robust 3D human body reconstruction in all weather conditions. In addition, our method's accuracy is significantly superior to that of state-of-the-art Transformer-based LiDAR-camera fusion methods.