Multi-View Stereo Using Perspective-Aware Features and Metadata to Improve Cost Volume

TL;DR

This paper introduces a new method for 3D reconstruction from images that improves accuracy in challenging areas like reflections and texture-less surfaces.

Contribution

The novel approach combines perspective-aware convolutions and metadata to enhance feature matching in multi-view stereo.

Findings

PAC-MVSNet improves feature matching in texture-less and reflective regions using perspective-aware convolutions.

The method integrates metadata like camera pose distance to guide geometric reasoning during cost aggregation.

The proposed network outperforms existing methods on multiple benchmark datasets.

Abstract

Feature matching is pivotal when using multi-view stereo (MVS) to reconstruct dense 3D models from calibrated images. This paper proposes PAC-MVSNet, which integrates perspective-aware convolution (PAC) and metadata-enhanced cost volumes to address the challenges in reflective and texture-less regions. PAC dynamically aligns convolutional kernels with scene perspective lines, while the use of metadata (e.g., camera pose distance) enables geometric reasoning during cost aggregation. In PAC-MVSNet, we introduce feature matching with long-range tracking that utilizes both internal and external focuses to integrate extensive contextual data within individual images as well as across multiple images. To enhance the performance of the feature matching with long-range tracking, we also propose a perspective-aware convolution module that directs the convolutional kernel to capture features along the perspective lines. This enables the module to extract perspective-aware features from images, improving the feature matching. Finally, we crafted a specific 2D CNN that fuses image priors, thereby integrating keyframes and geometric metadata within the cost volume to evaluate depth planes. Our method represents the first attempt to embed the existing physical model knowledge into a network for completing MVS tasks, which achieved optimal performance using multiple benchmark datasets.

Figures