ReTR: Modeling Rendering Via Transformer for Generalizable Neural Surface Reconstruction

TL;DR

ReTR introduces a transformer-based rendering framework that enhances neural surface reconstruction by modeling complex interactions, leading to improved accuracy and generalization over existing methods.

Contribution

The paper proposes a novel transformer-based rendering process with a learnable meta-ray token, improving surface reconstruction quality and robustness.

Findings

Outperforms state-of-the-art methods in reconstruction quality.

Demonstrates superior generalization across datasets.

Operates effectively within a high-dimensional feature space.

Abstract

Generalizable neural surface reconstruction techniques have attracted great attention in recent years. However, they encounter limitations of low confidence depth distribution and inaccurate surface reasoning due to the oversimplified volume rendering process employed. In this paper, we present Reconstruction TRansformer (ReTR), a novel framework that leverages the transformer architecture to redesign the rendering process, enabling complex render interaction modeling. It introduces a learnable $\textit{meta-ray token}$ and utilizes the cross-attention mechanism to simulate the interaction of rendering process with sampled points and render the observed color. Meanwhile, by operating within a high-dimensional feature space rather than the color space, ReTR mitigates sensitivity to projected colors in source views. Such improvements result in accurate surface assessment with high confidence. We demonstrate the effectiveness of our approach on various datasets, showcasing how our method outperforms the current state-of-the-art approaches in terms of reconstruction quality and generalization ability. $\textit{Our code is available at }$ https://github.com/YixunLiang/ReTR.