SCIGS: 3D Gaussians Splatting from a Snapshot Compressive Image

TL;DR

SCIGS is a novel method that reconstructs 3D dynamic scenes from a single compressed image, improving multi-view consistency and artifact removal over existing techniques.

Contribution

It introduces a primitive-level transformation network for 3D scene reconstruction from compressed images, addressing limitations of prior deep learning and NeRF-based methods.

Findings

Outperforms state-of-the-art methods in dynamic scene reconstruction

Enhances multi-view 3D structural consistency

Effectively removes artifacts during transformation

Abstract

Snapshot Compressive Imaging (SCI) offers a possibility for capturing information in high-speed dynamic scenes, requiring efficient reconstruction method to recover scene information. Despite promising results, current deep learning-based and NeRF-based reconstruction methods face challenges: 1) deep learning-based reconstruction methods struggle to maintain 3D structural consistency within scenes, and 2) NeRF-based reconstruction methods still face limitations in handling dynamic scenes. To address these challenges, we propose SCIGS, a variant of 3DGS, and develop a primitive-level transformation network that utilizes camera pose stamps and Gaussian primitive coordinates as embedding vectors. This approach resolves the necessity of camera pose in vanilla 3DGS and enhances multi-view 3D structural consistency in dynamic scenes by utilizing transformed primitives. Additionally, a high-frequency filter is introduced to eliminate the artifacts generated during the transformation. The proposed SCIGS is the first to reconstruct a 3D explicit scene from a single compressed image, extending its application to dynamic 3D scenes. Experiments on both static and dynamic scenes demonstrate that SCIGS not only enhances SCI decoding but also outperforms current state-of-the-art methods in reconstructing dynamic 3D scenes from a single compressed image. The code will be made available upon publication.