Touch-Augmented Gaussian Splatting for Enhanced 3D Scene Reconstruction

TL;DR

This paper introduces a touch-augmented 3D Gaussian Splatting framework that incorporates tactile signals to significantly improve scene reconstruction accuracy in challenging conditions like occlusion and low lighting.

Contribution

It presents a novel multimodal approach integrating touch signals into 3D Gaussian Splatting, with a two-stage sampling scheme and geometric loss for enhanced geometry and appearance.

Findings

Over 15x reduction in Chamfer Distance under severe occlusion

Consistent improvements in geometric accuracy across scenarios

Maintains a fully online reconstruction pipeline

Abstract

This paper presents a multimodal framework that integrates touch signals (contact points and surface normals) into 3D Gaussian Splatting (3DGS). Our approach enhances scene reconstruction, particularly under challenging conditions like low lighting, limited camera viewpoints, and occlusions. Different from the visual-only method, the proposed approach incorporates spatially selective touch measurements to refine both the geometry and appearance of the 3D Gaussian representation. To guide the touch exploration, we introduce a two-stage sampling scheme that initially probes sparse regions and then concentrates on high-uncertainty boundaries identified from the reconstructed mesh. A geometric loss is proposed to ensure surface smoothness, resulting in improved geometry. Experimental results across diverse scenarios show consistent improvements in geometric accuracy. In the most challenging case with severe occlusion, the Chamfer Distance is reduced by over 15x, demonstrating the effectiveness of integrating touch cues into 3D Gaussian Splatting. Furthermore, our approach maintains a fully online pipeline, underscoring its feasibility in visually degraded environments.