Learning Tactile Models for Factor Graph-based Estimation

TL;DR

This paper introduces a method to estimate object poses during planar pushing using tactile sensors by learning local observation models and integrating them into a factor graph for reliable tracking.

Contribution

It proposes a novel two-stage approach that learns tactile observation models directly from data and incorporates them into a factor graph for object pose estimation.

Findings

Achieved reliable object tracking with tactile feedback in 150 real-world sequences.

Demonstrated robustness across different object shapes and trajectories.

Integrated tactile models with physics and geometric factors for improved estimation.

Abstract

We're interested in the problem of estimating object states from touch during manipulation under occlusions. In this work, we address the problem of estimating object poses from touch during planar pushing. Vision-based tactile sensors provide rich, local image measurements at the point of contact. A single such measurement, however, contains limited information and multiple measurements are needed to infer latent object state. We solve this inference problem using a factor graph. In order to incorporate tactile measurements in the graph, we need local observation models that can map high-dimensional tactile images onto a low-dimensional state space. Prior work has used low-dimensional force measurements or engineered functions to interpret tactile measurements. These methods, however, can be brittle and difficult to scale across objects and sensors. Our key insight is to directly learn tactile observation models that predict the relative pose of the sensor given a pair of tactile images. These relative poses can then be incorporated as factors within a factor graph. We propose a two-stage approach: first we learn local tactile observation models supervised with ground truth data, and then integrate these models along with physics and geometric factors within a factor graph optimizer. We demonstrate reliable object tracking using only tactile feedback for 150 real-world planar pushing sequences with varying trajectories across three object shapes. Supplementary video: https://youtu.be/y1kBfSmi8w0