FusionSense: Bridging Common Sense, Vision, and Touch for Robust Sparse-View Reconstruction

TL;DR

FusionSense is a novel 3D reconstruction framework that combines common-sense priors with sparse vision and tactile data, enabling robots to perceive complex objects more robustly and efficiently.

Contribution

It introduces a hierarchical optimization approach using 3D Gaussian Splatting to fuse priors with sparse sensory observations for improved 3D reconstruction.

Findings

Outperforms state-of-the-art sparse-view reconstruction methods

Enables robust perception of transparent, reflective, and dark objects

Facilitates downstream manipulation and navigation tasks

Abstract

Humans effortlessly integrate common-sense knowledge with sensory input from vision and touch to understand their surroundings. Emulating this capability, we introduce FusionSense, a novel 3D reconstruction framework that enables robots to fuse priors from foundation models with highly sparse observations from vision and tactile sensors. FusionSense addresses three key challenges: (i) How can robots efficiently acquire robust global shape information about the surrounding scene and objects? (ii) How can robots strategically select touch points on the object using geometric and common-sense priors? (iii) How can partial observations such as tactile signals improve the overall representation of the object? Our framework employs 3D Gaussian Splatting as a core representation and incorporates a hierarchical optimization strategy involving global structure construction, object visual hull pruning and local geometric constraints. This advancement results in fast and robust perception in environments with traditionally challenging objects that are transparent, reflective, or dark, enabling more downstream manipulation or navigation tasks. Experiments on real-world data suggest that our framework outperforms previously state-of-the-art sparse-view methods. All code and data are open-sourced on the project website.