Monocular Reconstruction of Neural Tactile Fields

TL;DR

This paper introduces neural tactile fields, a novel 3D representation predicting tactile responses from a single RGB image, enabling robots to navigate complex, deformable environments more effectively.

Contribution

The paper presents the first method to predict neural tactile fields from monocular images, enhancing robot interaction-aware navigation in deformable environments.

Findings

Improves volumetric 3D reconstruction by 85.8%.

Enhances surface reconstruction accuracy by 26.7%.

Enables path planning that considers tactile properties of objects.

Abstract

Robots operating in the real world must plan through environments that deform, yield, and reconfigure under contact, requiring interaction-aware 3D representations that extend beyond static geometric occupancy. To address this, we introduce neural tactile fields, a novel 3D representation that maps spatial locations to the expected tactile response upon contact. Our model predicts these neural tactile fields from a single monocular RGB image -- the first method to do so. When integrated with off-the-shelf path planners, neural tactile fields enable robots to generate paths that avoid high-resistance objects while deliberately routing through low-resistance regions (e.g. foliage), rather than treating all occupied space as equally impassable. Empirically, our learning framework improves volumetric 3D reconstruction by $85.8\%$ and surface reconstruction by $26.7\%$ compared to state-of-the-art monocular 3D reconstruction methods (LRM and Direct3D).