CAD-NeRF: Learning NeRFs from Uncalibrated Few-view Images by CAD Model Retrieval

TL;DR

CAD-NeRF enables 3D reconstruction from fewer than 10 images without known camera poses by leveraging a CAD model library for shape retrieval and joint optimization, advancing uncalibrated NeRF methods.

Contribution

Introduces a novel uncalibrated NeRF approach that uses CAD model retrieval for shape guidance and pose initialization from sparse images.

Findings

Successfully reconstructs 3D objects from less than 10 images.

Achieves accurate density and pose estimation without known camera parameters.

Demonstrates strong generalization on synthetic and real datasets.

Abstract

Reconstructing from multi-view images is a longstanding problem in 3D vision, where neural radiance fields (NeRFs) have shown great potential and get realistic rendered images of novel views. Currently, most NeRF methods either require accurate camera poses or a large number of input images, or even both. Reconstructing NeRF from few-view images without poses is challenging and highly ill-posed. To address this problem, we propose CAD-NeRF, a method reconstructed from less than 10 images without any known poses. Specifically, we build a mini library of several CAD models from ShapeNet and render them from many random views. Given sparse-view input images, we run a model and pose retrieval from the library, to get a model with similar shapes, serving as the density supervision and pose initializations. Here we propose a multi-view pose retrieval method to avoid pose conflicts among views, which is a new and unseen problem in uncalibrated NeRF methods. Then, the geometry of the object is trained by the CAD guidance. The deformation of the density field and camera poses are optimized jointly. Then texture and density are trained and fine-tuned as well. All training phases are in self-supervised manners. Comprehensive evaluations of synthetic and real images show that CAD-NeRF successfully learns accurate densities with a large deformation from retrieved CAD models, showing the generalization abilities.