TEXterity -- Tactile Extrinsic deXterity: Simultaneous Tactile Estimation and Control for Extrinsic Dexterity

TL;DR

This paper presents a novel tactile estimation and control framework for in-hand object manipulation that combines kinematic and tactile sensor data to improve pose estimation and manipulation accuracy in real-time.

Contribution

It introduces a combined discrete and continuous pose estimation approach with receding horizon control, enhancing manipulation precision under occlusion and limited dexterity.

Findings

Outperforms single-shot methods in estimation accuracy

Successfully manipulates diverse objects and configurations

Enables closed-loop control for precise in-hand manipulation

Abstract

We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans in a receding horizon fashion to control the pose of a grasped object. This approach consists of a discrete pose estimator that tracks the most likely sequence of object poses in a coarsely discretized grid, and a continuous pose estimator-controller to refine the pose estimate and accurately manipulate the pose of the grasped object. Our method is tested on diverse objects and configurations, achieving desired manipulation objectives and outperforming single-shot methods in estimation accuracy. The proposed approach holds potential for tasks requiring precise manipulation and limited intrinsic in-hand dexterity under visual occlusion, laying the foundation for closed-loop behavior in applications such as regrasping, insertion, and tool use. Please see https://sites.google.com/view/texterity for videos of real-world demonstrations.