Multi-view Convolutional Neural Networks for 3D Shape Recognition

TL;DR

This paper demonstrates that view-based 2D image descriptors learned through CNNs can effectively recognize 3D shapes, outperforming traditional 3D descriptors, especially when multiple views are used, and introduces a novel multi-view CNN architecture.

Contribution

The paper introduces a new multi-view CNN architecture that combines multiple 2D views into a compact shape descriptor, significantly improving 3D shape recognition performance.

Findings

Single view recognition outperforms traditional 3D descriptors.

Multiple views increase recognition accuracy.

The proposed architecture improves recognition performance with fewer views.

Abstract

A longstanding question in computer vision concerns the representation of 3D shapes for recognition: should 3D shapes be represented with descriptors operating on their native 3D formats, such as voxel grid or polygon mesh, or can they be effectively represented with view-based descriptors? We address this question in the context of learning to recognize 3D shapes from a collection of their rendered views on 2D images. We first present a standard CNN architecture trained to recognize the shapes' rendered views independently of each other, and show that a 3D shape can be recognized even from a single view at an accuracy far higher than using state-of-the-art 3D shape descriptors. Recognition rates further increase when multiple views of the shapes are provided. In addition, we present a novel CNN architecture that combines information from multiple views of a 3D shape into a single and compact shape descriptor offering even better recognition performance. The same architecture can be applied to accurately recognize human hand-drawn sketches of shapes. We conclude that a collection of 2D views can be highly informative for 3D shape recognition and is amenable to emerging CNN architectures and their derivatives.