Learning Radiance Fields from a Single Snapshot Compressive Image

TL;DR

This paper introduces SCINeRF and SCISplat, innovative methods that leverage neural radiance fields and 3D Gaussian splatting to reconstruct detailed 3D scenes from a single snapshot compressive image, enabling real-time multi-view rendering.

Contribution

It proposes integrating SCI with NeRF and 3D Gaussian splatting for improved 3D scene reconstruction from a single compressed image.

Findings

Outperforms state-of-the-art in image reconstruction and view synthesis

Enables real-time multi-view rendering from a single snapshot

Effectively captures complex 3D scene structures

Abstract

In this paper, we explore the potential of Snapshot Compressive Imaging (SCI) technique for recovering the underlying 3D scene structure from a single temporal compressed image. SCI is a cost-effective method that enables the recording of high-dimensional data, such as hyperspectral or temporal information, into a single image using low-cost 2D imaging sensors. To achieve this, a series of specially designed 2D masks are usually employed, reducing storage and transmission requirements and offering potential privacy protection. Inspired by this, we take one step further to recover the encoded 3D scene information leveraging powerful 3D scene representation capabilities of neural radiance fields (NeRF). Specifically, we propose SCINeRF, in which we formulate the physical imaging process of SCI as part of the training of NeRF, allowing us to exploit its impressive performance in capturing complex scene structures. In addition, we further integrate the popular 3D Gaussian Splatting (3DGS) framework and propose SCISplat to improve 3D scene reconstruction quality and training/rendering speed by explicitly optimizing point clouds into 3D Gaussian representations. To assess the effectiveness of our method, we conduct extensive evaluations using both synthetic data and real data captured by our SCI system. Experimental results demonstrate that our proposed approach surpasses the state-of-the-art methods in terms of image reconstruction and novel view synthesis. Moreover, our method also exhibits the ability to render high frame-rate multi-view consistent images in real time by leveraging SCI and the rendering capabilities of 3DGS. Codes will be available at: https://github.com/WU- CVGL/SCISplat.