NeuralTouch: Neural Descriptors for Precise Sim-to-Real Tactile Robot Control

TL;DR

NeuralTouch combines neural descriptor fields and tactile sensing with reinforcement learning to enhance the accuracy and robustness of robotic grasping, enabling precise manipulation across various objects without fine-tuning.

Contribution

This work introduces NeuralTouch, a novel multimodal framework that integrates NDF and tactile feedback with deep RL for improved, generalizable robotic grasping.

Findings

Significantly improves grasping accuracy over baseline methods.

Enables zero-shot transfer from simulation to real-world tasks.

Demonstrates robustness in complex manipulation scenarios like lid opening.

Abstract

Grasping accuracy is a critical prerequisite for precise object manipulation, often requiring careful alignment between the robot hand and object. Neural Descriptor Fields (NDF) offer a promising vision-based method to generate grasping poses that generalize across object categories. However, NDF alone can produce inaccurate poses due to imperfect camera calibration, incomplete point clouds, and object variability. Meanwhile, tactile sensing enables more precise contact, but existing approaches typically learn policies limited to simple, predefined contact geometries. In this work, we introduce NeuralTouch, a multimodal framework that integrates NDF and tactile sensing to enable accurate, generalizable grasping through gentle physical interaction. Our approach leverages NDF to implicitly represent the target contact geometry, from which a deep reinforcement learning (RL) policy is trained to refine the grasp using tactile feedback. This policy is conditioned on the neural descriptors and does not require explicit specification of contact types. We validate NeuralTouch through ablation studies in simulation and zero-shot transfer to real-world manipulation tasks--such as peg-out-in-hole and bottle lid opening--without additional fine-tuning. Results show that NeuralTouch significantly improves grasping accuracy and robustness over baseline methods, offering a general framework for precise, contact-rich robotic manipulation.