Image to Icosahedral Projection for $\mathrm{SO}(3)$ Object Reasoning from Single-View Images

TL;DR

This paper introduces a novel neural network architecture that projects 2D images onto an icosahedral surface to enable approximate rotation equivariance in 3D object reasoning, improving pose estimation and classification.

Contribution

It proposes a new icosahedral projection method for 2D images to achieve approximate SO(3) equivariance in 3D object tasks.

Findings

Outperforms baseline models in pose estimation

Achieves better shape classification accuracy

Demonstrates effective rotation reasoning from single images

Abstract

Reasoning about 3D objects based on 2D images is challenging due to variations in appearance caused by viewing the object from different orientations. Tasks such as object classification are invariant to 3D rotations and other such as pose estimation are equivariant. However, imposing equivariance as a model constraint is typically not possible with 2D image input because we do not have an a priori model of how the image changes under out-of-plane object rotations. The only $\mathrm{SO}(3)$-equivariant models that currently exist require point cloud or voxel input rather than 2D images. In this paper, we propose a novel architecture based on icosahedral group convolutions that reasons in $\mathrm{SO(3)}$ by learning a projection of the input image onto an icosahedron. The resulting model is approximately equivariant to rotation in $\mathrm{SO}(3)$. We apply this model to object pose estimation and shape classification tasks and find that it outperforms reasonable baselines. Project website: \url{https://dmklee.github.io/image2icosahedral}