SpikeGS: Learning 3D Gaussian Fields from Continuous Spike Stream

TL;DR

SpikeGS introduces a novel method to learn 3D Gaussian fields from continuous spike streams, enabling high-quality, real-time view synthesis with robustness in noisy, low-light conditions, surpassing existing approaches.

Contribution

The paper presents SpikeGS, a new approach that learns 3D Gaussian fields directly from spike streams, incorporating noise embedding and a differentiable rendering framework for improved robustness and efficiency.

Findings

Achieves high-quality, real-time rendering from spike streams.

Demonstrates robustness in noisy, low-light environments.

Outperforms existing methods in rendering quality and speed.

Abstract

A spike camera is a specialized high-speed visual sensor that offers advantages such as high temporal resolution and high dynamic range compared to conventional frame cameras. These features provide the camera with significant advantages in many computer vision tasks. However, the tasks of novel view synthesis based on spike cameras remain underdeveloped. Although there are existing methods for learning neural radiance fields from spike stream, they either lack robustness in extremely noisy, low-quality lighting conditions or suffer from high computational complexity due to the deep fully connected neural networks and ray marching rendering strategies used in neural radiance fields, making it difficult to recover fine texture details. In contrast, the latest advancements in 3DGS have achieved high-quality real-time rendering by optimizing the point cloud representation into Gaussian ellipsoids. Building on this, we introduce SpikeGS, the method to learn 3D Gaussian fields solely from spike stream. We designed a differentiable spike stream rendering framework based on 3DGS, incorporating noise embedding and spiking neurons. By leveraging the multi-view consistency of 3DGS and the tile-based multi-threaded parallel rendering mechanism, we achieved high-quality real-time rendering results. Additionally, we introduced a spike rendering loss function that generalizes under varying illumination conditions. Our method can reconstruct view synthesis results with fine texture details from a continuous spike stream captured by a moving spike camera, while demonstrating high robustness in extremely noisy low-light scenarios. Experimental results on both real and synthetic datasets demonstrate that our method surpasses existing approaches in terms of rendering quality and speed.