Online Estimation of Articulated Objects with Factor Graphs using Vision and Proprioceptive Sensing

TL;DR

This paper presents an online method combining vision-based priors and kinematic sensing for robots to estimate and manipulate unknown articulated objects, achieving high success in real-world experiments.

Contribution

It introduces a novel online estimation approach that merges neural network predictions with real-time kinematic data for better articulation modeling.

Findings

Achieved 80% success rate in autonomous opening of unknown objects.

Effectively combined vision and proprioceptive sensing for real-time articulation estimation.

Demonstrated system on a real robot with closed-loop experiments.

Abstract

From dishwashers to cabinets, humans interact with articulated objects every day, and for a robot to assist in common manipulation tasks, it must learn a representation of articulation. Recent deep learning learning methods can provide powerful vision-based priors on the affordance of articulated objects from previous, possibly simulated, experiences. In contrast, many works estimate articulation by observing the object in motion, requiring the robot to already be interacting with the object. In this work, we propose to use the best of both worlds by introducing an online estimation method that merges vision-based affordance predictions from a neural network with interactive kinematic sensing in an analytical model. Our work has the benefit of using vision to predict an articulation model before touching the object, while also being able to update the model quickly from kinematic sensing during the interaction. In this paper, we implement a full system using shared autonomy for robotic opening of articulated objects, in particular objects in which the articulation is not apparent from vision alone. We implemented our system on a real robot and performed several autonomous closed-loop experiments in which the robot had to open a door with unknown joint while estimating the articulation online. Our system achieved an 80% success rate for autonomous opening of unknown articulated objects.