# Joint Inference of Kinematic and Force Trajectories with Visuo-Tactile Sensing

## TL;DR

This paper presents a probabilistic inference framework for joint estimation of kinematic and force trajectories during planar pushing tasks, integrating visuo-tactile data to improve robustness and accuracy in state estimation.

## Contribution

It extends previous methods by fusing visual and tactile sensor data for joint state and force estimation using factor graphs, applicable in both batch and incremental settings.

## Key findings

- Improved state estimation accuracy with multi-modal sensor fusion.
- Performance degradation under sensor occlusion and noise analyzed.
- Outperforms prior methods on multiple datasets.

## Abstract

To perform complex tasks, robots must be able to interact with and manipulate their surroundings. One of the key challenges in accomplishing this is robust state estimation during physical interactions, where the state involves not only the robot and the object being manipulated, but also the state of the contact itself. In this work, within the context of planar pushing, we extend previous inference-based approaches to state estimation in several ways. We estimate the robot, object, and the contact state on multiple manipulation platforms configured with a vision-based articulated model tracker, and either a biomimetic tactile sensor or a force-torque sensor. We show how to fuse raw measurements from the tracker and tactile sensors to jointly estimate the trajectory of the kinematic states and the forces in the system via probabilistic inference on factor graphs, in both batch and incremental settings. We perform several benchmarks with our framework and show how performance is affected by incorporating various geometric and physics based constraints, occluding vision sensors, or injecting noise in tactile sensors. We also compare with prior work on multiple datasets and demonstrate that our approach can effectively optimize over multi-modal sensor data and reduce uncertainty to find better state estimates.