VSFormer: Mining Correlations in Flexible View Set for Multi-view 3D Shape Understanding

TL;DR

VSFormer introduces a flexible, set-based approach with explicit correlation learning for multi-view 3D shape understanding, outperforming previous methods in accuracy and efficiency.

Contribution

It proposes a novel permutation-invariant view set and a Transformer model to explicitly learn multi-view correlations, improving flexibility and performance.

Findings

Achieves state-of-the-art results on ModelNet40, ScanObjectNN, and RGBD datasets.

Establishes new records on the SHREC'17 retrieval benchmark.

Demonstrates better flexibility and inference efficiency.

Abstract

View-based methods have demonstrated promising performance in 3D shape understanding. However, they tend to make strong assumptions about the relations between views or learn the multi-view correlations indirectly, which limits the flexibility of exploring inter-view correlations and the effectiveness of target tasks. To overcome the above problems, this paper investigates flexible organization and explicit correlation learning for multiple views. In particular, we propose to incorporate different views of a 3D shape into a permutation-invariant set, referred to as \emph{View Set}, which removes rigid relation assumptions and facilitates adequate information exchange and fusion among views. Based on that, we devise a nimble Transformer model, named \emph{VSFormer}, to explicitly capture pairwise and higher-order correlations of all elements in the set. Meanwhile, we theoretically reveal a natural correspondence between the Cartesian product of a view set and the correlation matrix in the attention mechanism, which supports our model design. Comprehensive experiments suggest that VSFormer has better flexibility, efficient inference efficiency and superior performance. Notably, VSFormer reaches state-of-the-art results on various 3d recognition datasets, including ModelNet40, ScanObjectNN and RGBD. It also establishes new records on the SHREC'17 retrieval benchmark. The code and datasets are available at \url{https://github.com/auniquesun/VSFormer}.