Out of Sight, Out of Mind: A Source-View-Wise Feature Aggregation for Multi-View Image-Based Rendering

TL;DR

This paper introduces a robust source-view-wise feature aggregation method for multi-view image-based rendering, improving consensus detection among source features by leveraging local structures and learnable similarity functions, leading to state-of-the-art results.

Contribution

The paper proposes a novel source-view-wise feature aggregation technique that enhances robustness against outliers by using local structure analysis and learnable similarity mappings.

Findings

Significant performance improvement over existing methods.

Achieved state-of-the-art results on benchmark datasets.

Robustness to occlusions and outliers in source features.

Abstract

To estimate the volume density and color of a 3D point in the multi-view image-based rendering, a common approach is to inspect the consensus existence among the given source image features, which is one of the informative cues for the estimation procedure. To this end, most of the previous methods utilize equally-weighted aggregation features. However, this could make it hard to check the consensus existence when some outliers, which frequently occur by occlusions, are included in the source image feature set. In this paper, we propose a novel source-view-wise feature aggregation method, which facilitates us to find out the consensus in a robust way by leveraging local structures in the feature set. We first calculate the source-view-wise distance distribution for each source feature for the proposed aggregation. After that, the distance distribution is converted to several similarity distributions with the proposed learnable similarity mapping functions. Finally, for each element in the feature set, the aggregation features are extracted by calculating the weighted means and variances, where the weights are derived from the similarity distributions. In experiments, we validate the proposed method on various benchmark datasets, including synthetic and real image scenes. The experimental results demonstrate that incorporating the proposed features improves the performance by a large margin, resulting in the state-of-the-art performance.