MC-NeRF: Multi-Camera Neural Radiance Fields for Multi-Camera Image Acquisition Systems

TL;DR

MC-NeRF introduces a joint optimization approach for intrinsic and extrinsic camera parameters in multi-camera systems, enabling accurate 3D scene reconstruction without initial pose estimates, supported by a new dataset and calibration scheme.

Contribution

The paper presents MC-NeRF, a novel method for joint optimization of camera parameters in multi-camera NeRF systems, addressing previous limitations and supporting independent camera parameters per image.

Findings

Effective joint optimization of intrinsic and extrinsic parameters.

Successful 3D scene reconstruction without initial pose estimates.

Validation on a newly constructed multi-camera dataset.

Abstract

Neural Radiance Fields (NeRF) use multi-view images for 3D scene representation, demonstrating remarkable performance. As one of the primary sources of multi-view images, multi-camera systems encounter challenges such as varying intrinsic parameters and frequent pose changes. Most previous NeRF-based methods assume a unique camera and rarely consider multi-camera scenarios. Besides, some NeRF methods that can optimize intrinsic and extrinsic parameters still remain susceptible to suboptimal solutions when these parameters are poor initialized. In this paper, we propose MC-NeRF, a method that enables joint optimization of both intrinsic and extrinsic parameters alongside NeRF. The method also supports each image corresponding to independent camera parameters. First, we tackle coupling issue and the degenerate case that arise from the joint optimization between intrinsic and extrinsic parameters. Second, based on the proposed solutions, we introduce an efficient calibration image acquisition scheme for multi-camera systems, including the design of calibration object. Finally, we present an end-to-end network with training sequence that enables the estimation of intrinsic and extrinsic parameters, along with the rendering network. Furthermore, recognizing that most existing datasets are designed for a unique camera, we construct a real multi-camera image acquisition system and create a corresponding new dataset, which includes both simulated data and real-world captured images. Experiments confirm the effectiveness of our method when each image corresponds to different camera parameters. Specifically, we use multi-cameras, each with different intrinsic and extrinsic parameters in real-world system, to achieve 3D scene representation without providing initial poses.