Realtime State Estimation with Tactile and Visual sensing. Application to Planar Manipulation

TL;DR

This paper presents a real-time object state estimation method combining tactile and visual sensing using a SLAM framework, improving robustness in cluttered environments for planar manipulation tasks.

Contribution

It introduces a novel integration of tactile and visual data within the iSAM framework tailored for pushing scenarios, enhancing accuracy and robustness in state estimation.

Findings

Improved estimation accuracy in cluttered scenes

Robustness across different object shapes and surfaces

Effective fusion of tactile and visual data

Abstract

Accurate and robust object state estimation enables successful object manipulation. Visual sensing is widely used to estimate object poses. However, in a cluttered scene or in a tight workspace, the robot's end-effector often occludes the object from the visual sensor. The robot then loses visual feedback and must fall back on open-loop execution. In this paper, we integrate both tactile and visual input using a framework for solving the SLAM problem, incremental smoothing and mapping (iSAM), to provide a fast and flexible solution. Visual sensing provides global pose information but is noisy in general, whereas contact sensing is local, but its measurements are more accurate relative to the end-effector. By combining them, we aim to exploit their advantages and overcome their limitations. We explore the technique in the context of a pusher-slider system. We adapt iSAM's measurement cost and motion cost to the pushing scenario, and use an instrumented setup to evaluate the estimation quality with different object shapes, on different surface materials, and under different contact modes.