Collision-aware In-hand 6D Object Pose Estimation using Multiple Vision-based Tactile Sensors

TL;DR

This paper presents a novel method for estimating the 6D pose of objects held in hand using multiple vision-based tactile sensors, combining geometric reasoning, CNN filtering, and collision-aware optimization.

Contribution

It introduces a collision-aware in-hand 6D pose estimation framework that integrates tactile sensor data, geometric filtering, and neural network-based hypothesis validation.

Findings

Achieves 87.5% accuracy in pose estimation in simulation.

Maintains an average positional error of about 2 centimeters.

Demonstrates effectiveness with real tactile sensors.

Abstract

In this paper, we address the problem of estimating the in-hand 6D pose of an object in contact with multiple vision-based tactile sensors. We reason on the possible spatial configurations of the sensors along the object surface. Specifically, we filter contact hypotheses using geometric reasoning and a Convolutional Neural Network (CNN), trained on simulated object-agnostic images, to promote those that better comply with the actual tactile images from the sensors. We use the selected sensors configurations to optimize over the space of 6D poses using a Gradient Descent-based approach. We finally rank the obtained poses by penalizing those that are in collision with the sensors. We carry out experiments in simulation using the DIGIT vision-based sensor with several objects, from the standard YCB model set. The results demonstrate that our approach estimates object poses that are compatible with actual object-sensor contacts in $87.5\%$ of cases while reaching an average positional error in the order of $2$ centimeters. Our analysis also includes qualitative results of experiments with a real DIGIT sensor.