FingerEye: Continuous and Unified Vision-Tactile Sensing for Dexterous Manipulation

TL;DR

FingerEye is a novel vision-tactile sensor that provides continuous feedback during manipulation, enabling more adaptive and robust dexterous robotic behaviors through imitation learning and digital twin simulation.

Contribution

The paper introduces FingerEye, a compact sensor that offers seamless vision and tactile feedback throughout interaction, enhancing dexterous manipulation capabilities.

Findings

FingerEye enables continuous perception from pre-contact to post-contact.

The learned policies demonstrate robustness across diverse object interactions.

Digital twin simulation improves policy generalization and robustness.

Abstract

Dexterous robotic manipulation requires comprehensive perception across all phases of interaction: pre-contact, contact initiation, and post-contact. Such continuous feedback allows a robot to adapt its actions throughout interaction. However, many existing tactile sensors, such as GelSight and its variants, only provide feedback after contact is established, limiting a robot's ability to precisely initiate contact. We introduce FingerEye, a compact and cost-effective sensor that provides continuous vision-tactile feedback throughout the interaction process. FingerEye integrates binocular RGB cameras to provide close-range visual perception with implicit stereo depth. Upon contact, external forces and torques deform a compliant ring structure; these deformations are captured via marker-based pose estimation and serve as a proxy for contact wrench sensing. This design enables a perception stream that smoothly transitions from pre-contact visual cues to post-contact tactile feedback. Building on this sensing capability, we develop a vision-tactile imitation learning policy that fuses signals from multiple FingerEye sensors to learn dexterous manipulation behaviors from limited real-world data. We further develop a digital twin of our sensor and robot platform to improve policy generalization. By combining real demonstrations with visually augmented simulated observations for representation learning, the learned policies become more robust to object appearance variations. Together, these design aspects enable dexterous manipulation across diverse object properties and interaction regimes, including coin standing, chip picking, letter retrieving, and syringe manipulation. The hardware design, code, appendix, and videos are available on our project website: https://nus-lins-lab.github.io/FingerEyeWeb/