Self-supervised learning of object pose estimation using keypoint prediction

TL;DR

This paper introduces a self-supervised learning method for object pose estimation from single images, utilizing keypoint prediction on category-specific deformable shapes, achieving significant improvements over existing methods.

Contribution

A novel self-supervised approach for camera pose prediction using keypoints on deformable shapes, trained with proxy ground-truth heatmaps from category-specific mean shapes.

Findings

Significant improvement over state-of-the-art methods in camera pose prediction

Effective online inference of 3D objects from 2D video frames

Trained on one dataset and tested across multiple datasets

Abstract

This paper describes recent developments in object specific pose and shape prediction from single images. The main contribution is a new approach to camera pose prediction by self-supervised learning of keypoints corresponding to locations on a category specific deformable shape. We designed a network to generate a proxy ground-truth heatmap from a set of keypoints distributed all over the category-specific mean shape, where each is represented by a unique color on a labeled texture. The proxy ground-truth heatmap is used to train a deep keypoint prediction network, which can be used in online inference. The proposed approach to camera pose prediction show significant improvements when compared with state-of-the-art methods. Our approach to camera pose prediction is used to infer 3D objects from 2D image frames of video sequences online. To train the reconstruction model, it receives only a silhouette mask from a single frame of a video sequence in every training step and a category-specific mean object shape. We conducted experiments using three different datasets representing the bird category: the CUB [51] image dataset, YouTubeVos and the Davis video datasets. The network is trained on the CUB dataset and tested on all three datasets. The online experiments are demonstrated on YouTubeVos and Davis [56] video sequences using a network trained on the CUB training set.