CT-NeRF: Incremental Optimizing Neural Radiance Field and Poses with Complex Trajectory

TL;DR

CT-NeRF introduces an incremental optimization pipeline that jointly recovers camera poses and scene structure from RGB images, effectively handling complex trajectories with large rotations, surpassing prior methods in accuracy.

Contribution

The paper proposes a novel incremental reconstruction method with local-global bundle adjustment and geometric constraints, enabling NeRF to handle complex trajectories without depth or pose input.

Findings

Outperforms existing methods in novel view synthesis.

Achieves higher pose estimation accuracy.

Successfully handles scenes with complex trajectories.

Abstract

Neural radiance field (NeRF) has achieved impressive results in high-quality 3D scene reconstruction. However, NeRF heavily relies on precise camera poses. While recent works like BARF have introduced camera pose optimization within NeRF, their applicability is limited to simple trajectory scenes. Existing methods struggle while tackling complex trajectories involving large rotations. To address this limitation, we propose CT-NeRF, an incremental reconstruction optimization pipeline using only RGB images without pose and depth input. In this pipeline, we first propose a local-global bundle adjustment under a pose graph connecting neighboring frames to enforce the consistency between poses to escape the local minima caused by only pose consistency with the scene structure. Further, we instantiate the consistency between poses as a reprojected geometric image distance constraint resulting from pixel-level correspondences between input image pairs. Through the incremental reconstruction, CT-NeRF enables the recovery of both camera poses and scene structure and is capable of handling scenes with complex trajectories. We evaluate the performance of CT-NeRF on two real-world datasets, NeRFBuster and Free-Dataset, which feature complex trajectories. Results show CT-NeRF outperforms existing methods in novel view synthesis and pose estimation accuracy.