Joint Viewpoint and Keypoint Estimation with Real and Synthetic Data

TL;DR

This paper introduces a joint CNN approach for estimating viewpoints and keypoints of objects, leveraging synthetic data and multi-task training to improve accuracy over independent methods.

Contribution

It presents a novel multi-task CNN that jointly estimates viewpoints and keypoints, utilizing synthetic data for training and enabling flexible datasets.

Findings

Joint training improves accuracy of both tasks.

Synthetic dataset accelerates annotation process.

Outperforms previous independent-task methods.

Abstract

The estimation of viewpoints and keypoints effectively enhance object detection methods by extracting valuable traits of the object instances. While the output of both processes differ, i.e., angles vs. list of characteristic points, they indeed share the same focus on how the object is placed in the scene, inducing that there is a certain level of correlation between them. Therefore, we propose a convolutional neural network that jointly computes the viewpoint and keypoints for different object categories. By training both tasks together, each task improves the accuracy of the other. Since the labelling of object keypoints is very time consuming for human annotators, we also introduce a new synthetic dataset with automatically generated viewpoint and keypoints annotations. Our proposed network can also be trained on datasets that contain viewpoint and keypoints annotations or only one of them. The experiments show that the proposed approach successfully exploits this implicit correlation between the tasks and outperforms previous techniques that are trained independently.