Simultaneous Extrinsic Contact and In-Hand Pose Estimation via Distributed Tactile Sensing

TL;DR

This paper introduces a factor graph-based method that combines tactile sensing and physical constraints to accurately estimate object pose and contacts during in-hand manipulation, outperforming existing approaches especially with tactile data alone.

Contribution

The work presents a novel factor graph formulation that integrates tactile observations with physical constraints for simultaneous contact and pose estimation.

Findings

Outperforms existing geometric and contact-informed methods

Effective with tactile data alone, reducing reliance on visual feedback

Demonstrates improved accuracy in in-hand manipulation scenarios

Abstract

Prehensile autonomous manipulation, such as peg insertion, tool use, or assembly, require precise in-hand understanding of the object pose and the extrinsic contacts made during interactions. Providing accurate estimation of pose and contacts is challenging. Tactile sensors can provide local geometry at the sensor and force information about the grasp, but the locality of sensing means resolving poses and contacts from tactile alone is often an ill-posed problem, as multiple configurations can be consistent with the observations. Adding visual feedback can help resolve ambiguities, but can suffer from noise and occlusions. In this work, we propose a method that pairs local observations from sensing with the physical constraints of contact. We propose a set of factors that ensure local consistency with tactile observations as well as enforcing physical plausibility, namely, that the estimated pose and contacts must respect the kinematic and force constraints of quasi-static rigid body interactions. We formalize our problem as a factor graph, allowing for efficient estimation. In our experiments, we demonstrate that our method outperforms existing geometric and contact-informed estimation pipelines, especially when only tactile information is available. Video results can be found at https://tacgraph.github.io/.