iSPA-Net: Iterative Semantic Pose Alignment Network

TL;DR

iSPA-Net is an iterative deep learning framework that improves 3D object pose estimation from monocular images by leveraging semantic 3D structure, reducing data requirements, and refining predictions through iterative pose alignment.

Contribution

The paper introduces iSPA-Net, a novel iterative pose alignment network that exploits semantic 3D structure and correspondence for fine-grained pose estimation with minimal annotations.

Findings

Achieves state-of-the-art performance on real image viewpoint datasets.

Effectively refines pose estimates through iterative alignment.

Demonstrates applications in active viewpoint localization and unsupervised part segmentation.

Abstract

Understanding and extracting 3D information of objects from monocular 2D images is a fundamental problem in computer vision. In the task of 3D object pose estimation, recent data driven deep neural network based approaches suffer from scarcity of real images with 3D keypoint and pose annotations. Drawing inspiration from human cognition, where the annotators use a 3D CAD model as structural reference to acquire ground-truth viewpoints for real images; we propose an iterative Semantic Pose Alignment Network, called iSPA-Net. Our approach focuses on exploiting semantic 3D structural regularity to solve the task of fine-grained pose estimation by predicting viewpoint difference between a given pair of images. Such image comparison based approach also alleviates the problem of data scarcity and hence enhances scalability of the proposed approach for novel object categories with minimal annotation. The fine-grained object pose estimator is also aided by correspondence of learned spatial descriptor of the input image pair. The proposed pose alignment framework enjoys the faculty to refine its initial pose estimation in consecutive iterations by utilizing an online rendering setup along with effectiveness of a non-uniform bin classification of pose-difference. This enables iSPA-Net to achieve state-of-the-art performance on various real image viewpoint estimation datasets. Further, we demonstrate effectiveness of the approach for multiple applications. First, we show results for active object viewpoint localization to capture images from similar pose considering only a single image as pose reference. Second, we demonstrate the ability of the learned semantic correspondence to perform unsupervised part-segmentation transfer using only a single part-annotated 3D template model per object class. To encourage reproducible research, we have released the codes for our proposed algorithm.