MVTN: Learning Multi-View Transformations for 3D Understanding

TL;DR

This paper introduces MVTN, a learnable multi-view transformation network that optimizes camera viewpoints for 3D shape recognition, achieving state-of-the-art results and improved robustness in 3D classification tasks.

Contribution

We propose MVTN, a differentiable network that learns optimal view-points for multi-view 3D recognition, integrated into an adaptive pipeline for meshes and point clouds.

Findings

State-of-the-art performance on ModelNet40, ScanObjectNN, ShapeNet Core55

Improved robustness to occlusion in 3D shape recognition

Effective integration with multi-view networks for end-to-end training

Abstract

Multi-view projection techniques have shown themselves to be highly effective in achieving top-performing results in the recognition of 3D shapes. These methods involve learning how to combine information from multiple view-points. However, the camera view-points from which these views are obtained are often fixed for all shapes. To overcome the static nature of current multi-view techniques, we propose learning these view-points. Specifically, we introduce the Multi-View Transformation Network (MVTN), which uses differentiable rendering to determine optimal view-points for 3D shape recognition. As a result, MVTN can be trained end-to-end with any multi-view network for 3D shape classification. We integrate MVTN into a novel adaptive multi-view pipeline that is capable of rendering both 3D meshes and point clouds. Our approach demonstrates state-of-the-art performance in 3D classification and shape retrieval on several benchmarks (ModelNet40, ScanObjectNN, ShapeNet Core55). Further analysis indicates that our approach exhibits improved robustness to occlusion compared to other methods. We also investigate additional aspects of MVTN, such as 2D pretraining and its use for segmentation. To support further research in this area, we have released MVTorch, a PyTorch library for 3D understanding and generation using multi-view projections.