GNeRF: GAN-based Neural Radiance Field without Posed Camera

TL;DR

GNeRF introduces a GAN-based framework that jointly optimizes camera poses and radiance fields from randomly initialized poses, enabling realistic novel view synthesis in complex scenes without accurate pose estimation.

Contribution

This work presents a novel two-phase end-to-end approach combining GANs with NeRF to handle unknown and randomly initialized camera poses in complex scenes.

Findings

Outperforms baselines in scenes with repeated patterns or low textures.

Effective in complex outside-in scenarios with unknown camera poses.

Overcomes local minima through hybrid iterative optimization.

Abstract

We introduce GNeRF, a framework to marry Generative Adversarial Networks (GAN) with Neural Radiance Field (NeRF) reconstruction for the complex scenarios with unknown and even randomly initialized camera poses. Recent NeRF-based advances have gained popularity for remarkable realistic novel view synthesis. However, most of them heavily rely on accurate camera poses estimation, while few recent methods can only optimize the unknown camera poses in roughly forward-facing scenes with relatively short camera trajectories and require rough camera poses initialization. Differently, our GNeRF only utilizes randomly initialized poses for complex outside-in scenarios. We propose a novel two-phases end-to-end framework. The first phase takes the use of GANs into the new realm for optimizing coarse camera poses and radiance fields jointly, while the second phase refines them with additional photometric loss. We overcome local minima using a hybrid and iterative optimization scheme. Extensive experiments on a variety of synthetic and natural scenes demonstrate the effectiveness of GNeRF. More impressively, our approach outperforms the baselines favorably in those scenes with repeated patterns or even low textures that are regarded as extremely challenging before.