RayFormer: Modeling Inter- and Intra-Ray Similarity for NeRF-Based Video Snapshot Compressive Imaging

TL;DR

RayFormer introduces a novel patch-level ray sampling and transformer-based approach to improve NeRF-based video snapshot compressive imaging, capturing structural similarities for superior reconstruction quality.

Contribution

The paper proposes a patch-level sampling strategy and a transformer model to better capture structural similarities in NeRF-based SCI, enhancing reconstruction performance.

Findings

Achieves state-of-the-art reconstruction results in simulated scenes.

Effectively models inter- and intra-ray structural similarities.

Incorporates total variation prior for spatial smoothness.

Abstract

Video snapshot compressive imaging (SCI) enables the reconstruction of dynamic scenes from a single snapshot measurement. Recently, NeRF-based methods have shown promising reconstruction performance. However, such methods typically adopt random ray sampling strategies and fail to capture content structural similarities, resulting in limited reconstruction quality. To address these issues, we first propose a patch-level ray sampling strategy to enable the modeling of content structure. Then, we propose an Inter- and Intra-Ray Transformer (RayFormer) to capture the structural similarities, modeling both inter-ray similarities among spatially neighboring points at the same depth and intra-ray correlations between adjacent points along the viewing ray. Finally, benefiting from the patch-level sampling strategy, the total variation prior is incorporated into the objective function to enhance spatial smoothness and suppress artifacts. Experiments in both simulated and real-world scenes demonstrate that the proposed method achieves state-of-the-art (SOTA) reconstruction performance.