VisuoTactile 6D Pose Estimation of an In-Hand Object using Vision and Tactile Sensor Data

TL;DR

This paper introduces a method combining vision and tactile data for 6D object pose estimation in robotic in-hand manipulation, improving accuracy and generalization from synthetic to real-world scenarios.

Contribution

The paper proposes a novel sensor fusion approach using point clouds and a dense fusion network architecture for improved in-hand 6D pose estimation.

Findings

Tactile data enhances pose estimation accuracy.

Synthetic data training generalizes well to real robots.

The method outperforms vision-only approaches.

Abstract

Knowledge of the 6D pose of an object can benefit in-hand object manipulation. In-hand 6D object pose estimation is challenging because of heavy occlusion produced by the robot's grippers, which can have an adverse effect on methods that rely on vision data only. Many robots are equipped with tactile sensors at their fingertips that could be used to complement vision data. In this paper, we present a method that uses both tactile and vision data to estimate the pose of an object grasped in a robot's hand. To address challenges like lack of standard representation for tactile data and sensor fusion, we propose the use of point clouds to represent object surfaces in contact with the tactile sensor and present a network architecture based on pixel-wise dense fusion. We also extend NVIDIA's Deep Learning Dataset Synthesizer to produce synthetic photo-realistic vision data and corresponding tactile point clouds. Results suggest that using tactile data in addition to vision data improves the 6D pose estimate, and our network generalizes successfully from synthetic training to real physical robots.