RPBG: Towards Robust Neural Point-based Graphics in the Wild

TL;DR

RPBG introduces a robust neural point-based graphics method that improves view synthesis quality in real-world, noisy, and unbounded scenes by enhancing the renderer with attention-based corrections and inpainting, outperforming existing methods.

Contribution

The paper proposes RPBG, a robust extension of NPBG, with novel attention-based visibility correction and inpainting techniques to handle real-world scene variations.

Findings

RPBG significantly outperforms the baseline NPBG on diverse datasets.

RPBG demonstrates superior robustness compared to state-of-the-art NeRF variants.

The method maintains high-quality rendering with acceptable computational overhead.

Abstract

Point-based representations have recently gained popularity in novel view synthesis, for their unique advantages, e.g., intuitive geometric representation, simple manipulation, and faster convergence. However, based on our observation, these point-based neural re-rendering methods are only expected to perform well under ideal conditions and suffer from noisy, patchy points and unbounded scenes, which are challenging to handle but defacto common in real applications. To this end, we revisit one such influential method, known as Neural Point-based Graphics (NPBG), as our baseline, and propose Robust Point-based Graphics (RPBG). We in-depth analyze the factors that prevent NPBG from achieving satisfactory renderings on generic datasets, and accordingly reform the pipeline to make it more robust to varying datasets in-the-wild. Inspired by the practices in image restoration, we greatly enhance the neural renderer to enable the attention-based correction of point visibility and the inpainting of incomplete rasterization, with only acceptable overheads. We also seek for a simple and lightweight alternative for environment modeling and an iterative method to alleviate the problem of poor geometry. By thorough evaluation on a wide range of datasets with different shooting conditions and camera trajectories, RPBG stably outperforms the baseline by a large margin, and exhibits its great robustness over state-of-the-art NeRF-based variants. Code available at https://github.com/QT-Zhu/RPBG.