UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction

TL;DR

UMIFormer introduces a transformer-based approach that mines correlations between similar tokens across unstructured views to improve multi-view 3D reconstruction, outperforming state-of-the-art methods on ShapeNet.

Contribution

The paper proposes UMIFormer, a novel transformer network that effectively mines inter-view token correlations for unstructured multi-view 3D reconstruction.

Findings

Outperforms existing SOTA methods on ShapeNet

Effectively mines correlations between tokens from different views

Demonstrates adaptability to unstructured multiple images

Abstract

In recent years, many video tasks have achieved breakthroughs by utilizing the vision transformer and establishing spatial-temporal decoupling for feature extraction. Although multi-view 3D reconstruction also faces multiple images as input, it cannot immediately inherit their success due to completely ambiguous associations between unstructured views. There is not usable prior relationship, which is similar to the temporally-coherence property in a video. To solve this problem, we propose a novel transformer network for Unstructured Multiple Images (UMIFormer). It exploits transformer blocks for decoupled intra-view encoding and designed blocks for token rectification that mine the correlation between similar tokens from different views to achieve decoupled inter-view encoding. Afterward, all tokens acquired from various branches are compressed into a fixed-size compact representation while preserving rich information for reconstruction by leveraging the similarities between tokens. We empirically demonstrate on ShapeNet and confirm that our decoupled learning method is adaptable for unstructured multiple images. Meanwhile, the experiments also verify our model outperforms existing SOTA methods by a large margin. Code will be available at https://github.com/GaryZhu1996/UMIFormer.