Using Collocated Vision and Tactile Sensors for Visual Servoing and Localization

TL;DR

This paper explores how combining collocated vision and tactile sensors enhances robotic visual servoing and localization by providing better object pose estimates, reducing occlusion, and refining contact points, especially for textured objects.

Contribution

It introduces a method for integrating vision and tactile sensing at the sensor level to improve robot manipulation and localization accuracy, highlighting the importance of prior visual information.

Findings

Optic flow requires significant camera movement to predict motion direction.

State-of-the-art vision algorithms struggle to localize tactile images without prior visual context.

Collocated sensors improve object pose estimation and tactile localization accuracy.

Abstract

Coordinating proximity and tactile imaging by collocating cameras with tactile sensors can 1) provide useful information before contact such as object pose estimates and visually servo a robot to a target with reduced occlusion and higher resolution compared to head-mounted or external depth cameras, 2) simplify the contact point and pose estimation problems and help tactile sensing avoid erroneous matches when a surface does not have significant texture or has repetitive texture with many possible matches, and 3) use tactile imaging to further refine contact point and object pose estimation. We demonstrate our results with objects that have more surface texture than most objects in standard manipulation datasets. We learn that optic flow needs to be integrated over a substantial amount of camera travel to be useful in predicting movement direction. Most importantly, we also learn that state of the art vision algorithms do not do a good job localizing tactile images on object models, unless a reasonable prior can be provided from collocated cameras.