Equivariant Single View Pose Prediction Via Induced and Restricted Representations

TL;DR

This paper introduces a novel neural network architecture leveraging SO(2)-equivariance constraints to learn 3D object pose from 2D images, achieving state-of-the-art results on pose estimation benchmarks.

Contribution

The paper formulates geometric consistency constraints for 3D pose prediction from 2D images and constructs architectures based on induced and restricted SO(2) representations, unifying previous methods.

Findings

Achieves state-of-the-art results on PASCAL3D+ and SYMSOL datasets.

Unifies and generalizes previous 3D pose prediction architectures.

Proposes a learnable algorithm respecting geometric consistency constraints.

Abstract

Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in three dimensions to make predictions about novel images. However, imposing SO(3)-equivariance on two-dimensional inputs is difficult because the group of three-dimensional rotations does not have a natural action on the two-dimensional plane. Specifically, it is possible that an element of SO(3) will rotate an image out of plane. We show that an algorithm that learns a three-dimensional representation of the world from two dimensional images must satisfy certain geometric consistency properties which we formulate as SO(2)-equivariance constraints. We use the induced and restricted representations of SO(2) on SO(3) to construct and classify architectures which satisfy these geometric consistency constraints. We prove that any architecture which respects said consistency constraints can be realized as an instance of our construction. We show that three previously proposed neural architectures for 3D pose prediction are special cases of our construction. We propose a new algorithm that is a learnable generalization of previously considered methods. We test our architecture on three pose predictions task and achieve SOTA results on both the PASCAL3D+ and SYMSOL pose estimation tasks.