Deep Gated Multi-modal Learning: In-hand Object Pose Changes Estimation using Tactile and Image Data

TL;DR

This paper introduces a deep gated multi-modal learning approach that automatically determines the reliability of tactile and visual data to accurately estimate object pose changes during in-hand manipulation, even with occlusions and noise.

Contribution

It presents a novel end-to-end deep learning model that self-adjusts modality reliability, improving in-hand object pose estimation using combined tactile and image data.

Findings

Effective estimation of pose changes for 15 objects during grasping.

Reliability values adapt to noise and sensor failure, enhancing robustness.

Accurate pose change estimation for unknown objects.

Abstract

For in-hand manipulation, estimation of the object pose inside the hand is one of the important functions to manipulate objects to the target pose. Since in-hand manipulation tends to cause occlusions by the hand or the object itself, image information only is not sufficient for in-hand object pose estimation. Multiple modalities can be used in this case, the advantage is that other modalities can compensate for occlusion, noise, and sensor malfunctions. Even though deciding the utilization rate of a modality (referred to as reliability value) corresponding to the situations is important, the manual design of such models is difficult, especially for various situations. In this paper, we propose deep gated multi-modal learning, which self-determines the reliability value of each modality through end-to-end deep learning. For the experiments, an RGB camera and a GelSight tactile sensor were attached to the parallel gripper of the Sawyer robot, and the object pose changes were estimated during grasping. A total of 15 objects were used in the experiments. In the proposed model, the reliability values of the modalities were determined according to the noise level and failure of each modality, and it was confirmed that the pose change was estimated even for unknown objects.