GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff

TL;DR

This paper introduces GARNet, a global-aware multi-view 3D reconstruction network that improves feature fusion using global correlation, with a novel loss and training strategy, outperforming state-of-the-art methods while reducing parameters.

Contribution

The paper proposes a global-aware attention fusion approach, a new loss function, and a dynamic training strategy for multi-view 3D reconstruction, enhancing accuracy and efficiency.

Findings

Outperforms existing SOTA methods on ShapeNet

Uses fewer parameters than comparable algorithms

Provides a view-reduction method balancing cost and performance

Abstract

Deep learning technology has made great progress in multi-view 3D reconstruction tasks. At present, most mainstream solutions establish the mapping between views and shape of an object by assembling the networks of 2D encoder and 3D decoder as the basic structure while they adopt different approaches to obtain aggregation of features from several views. Among them, the methods using attention-based fusion perform better and more stable than the others, however, they still have an obvious shortcoming -- the strong independence of each view during predicting the weights for merging leads to a lack of adaption of the global state. In this paper, we propose a global-aware attention-based fusion approach that builds the correlation between each branch and the global to provide a comprehensive foundation for weights inference. In order to enhance the ability of the network, we introduce a novel loss function to supervise the shape overall and propose a dynamic two-stage training strategy that can effectively adapt to all reconstructors with attention-based fusion. Experiments on ShapeNet verify that our method outperforms existing SOTA methods while the amount of parameters is far less than the same type of algorithm, Pix2Vox++. Furthermore, we propose a view-reduction method based on maximizing diversity and discuss the cost-performance tradeoff of our model to achieve a better performance when facing heavy input amount and limited computational cost.