Multi-Modal Learning of Keypoint Predictive Models for Visual Object Manipulation

TL;DR

This paper introduces a self-supervised multi-modal approach for robots to learn visual keypoints and extend their kinematic models during object manipulation, enhancing generalization and manipulation capabilities.

Contribution

It presents a novel autoencoder-based multi-modal keypoint detector and a method to extend robot kinematics using visual keypoints, enabling better manipulation in new environments.

Findings

The approach accurately predicts visual keypoints on grasped objects.

It successfully extends the robot's kinematic chain with minimal visual data.

The extended kinematic model improves object placement tasks in simulation and hardware.

Abstract

Humans have impressive generalization capabilities when it comes to manipulating objects and tools in completely novel environments. These capabilities are, at least partially, a result of humans having internal models of their bodies and any grasped object. How to learn such body schemas for robots remains an open problem. In this work, we develop an self-supervised approach that can extend a robot's kinematic model when grasping an object from visual latent representations. Our framework comprises two components: (1) we present a multi-modal keypoint detector: an autoencoder architecture trained by fusing proprioception and vision to predict visual key points on an object; (2) we show how we can use our learned keypoint detector to learn an extension of the kinematic chain by regressing virtual joints from the predicted visual keypoints. Our evaluation shows that our approach learns to consistently predict visual keypoints on objects in the manipulator's hand, and thus can easily facilitate learning an extended kinematic chain to include the object grasped in various configurations, from a few seconds of visual data. Finally we show that this extended kinematic chain lends itself for object manipulation tasks such as placing a grasped object and present experiments in simulation and on hardware.